[Bug] b12x NVFP4 backend on sm_120 (RTX 5090) serves batch 1 only: not CUDA-graph capturable, and the worker dies with a tile-config ValueError at batch >= 2 (Qwen3.8-Flash-Next-NVFP4) #335

Summary

On sm_120 (RTX 5090) the b12x NVFP4 expert backend serves batch 1 only. It cannot be CUDA-graph captured, and the engine worker dies as soon as a second concurrent request arrives. --nvfp4-backend triton on the same box, same model, same memory envelope is completely healthy, including graph capture at bs=8 and eight concurrent streams.

I could not find either signature reported here or in flashinfer (force_tile_config, unjoined: 0 hits in both repos), and #324 — the closest report, same model and card class — does not pass --nvfp4-backend, so it runs the default triton path and never exercises b12x.

Environment

• FreeToken git 4b94bdc38a46a4dfe534e8793126160d56904c44 (freetoken version 0.1.2), installed with --no-build-isolation into a venv on pytorch/pytorch:2.11.0-cuda13.0-cudnn9-devel (sha256:6e8a7a6dedf900096f90190f66f988e7e658cda4f1e6cbc7c17e3a38980a4f89)
• RTX 5090 (sm_120, 32 GB), driver 590.48.01, CUDA 13.1 host max, torch 2.11.0+cu130, AMD EPYC 7K62, PCIe 4.0 x16, 501 GiB host RAM (rented vast.ai instance)
• Model RadixArk/Qwen3.8-Flash-Next-NVFP4 @ 7b719225242aacd3dbd3f9407468c2ee9a9d2594
• ft serve --max-seq-len-override 262144 --kv-reserve-tokens 262144 --moe-backend offload --moe-cache-auto --nvfp4-backend flashinfer --max-running-requests 8 --max-prefill-length 8192 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coder

Load envelope (healthy, both backends): Free memory before loading model: 30.80 GiB → --moe-cache-auto resolved moe_cache_size=2614 num_pages=4096 → Allocating 262144 tokens for KV cache, K + V = 6.19 GiB → Free memory after initialization: 2.85 GiB → NVFP4 expert backend: b12x.

Defect 1 — b12x cannot be CUDA-graph captured

With --cuda-graph-max-bs 8, capture runs largest-first and fails on the first size after 2m13s:

Capturing graphs: bs = 8   | avail_mem = 2.85 GiB:   0%|   | 0/4 [02:13<?, ?batch/s]
[core|rank=0] ERROR  Backend supervisor: AcceleratorError: CUDA error: capturing stream has unjoined work

  File "freetoken/engine/graph.py", line 179, in _capture_graphs
    self.buffer.logits[:bs] = model.forward()
  File "freetoken/models/qwen4_exp/model.py", line 184, in forward
  ...
  File "freetoken/layers/moe.py", line 453, in _expert_gemm
    return fused(
  File "freetoken/moe/nvfp4_backends.py", line 761, in b12x_fused_experts
    _launch_sm120_w4a16_moe(
  File "flashinfer/fused_moe/cute_dsl/blackwell_sm12x/moe_dispatch.py", line 2810, in _launch_sm120_w4a16_moe
    return run_w4a16_moe(
  File "flashinfer/fused_moe/cute_dsl/blackwell_sm12x/moe_w4a16_kernel.py", line 6666, in pack_topk_routes_by_expert
  File "flashinfer/fused_moe/cute_dsl/blackwell_sm12x/moe_w4a16_route_pack.py", line 444, in pack_topk_routes_by_expert
    _pack_topk_routes_prefix_kernel[(1,)](

cudaErrorStreamCaptureUnjoined means the route-packing prefix kernel launches work on a stream that is never joined back into the capturing stream, so it is not capture-safe.

This is not memory pressure. triton captures [1, 2, 4, 8] on the same box in 7 seconds using 0.13 GiB (2.87 → 2.74 GiB free), where b12x spent 2m13s on bs=8 alone and failed.

Workaround: --cuda-graph-max-bs 0 disables capture (_determine_cuda_graph_bs: if cuda_graph_max_bs < 1: return []) and the server then reaches API server is ready and answers correctly at batch 1.

Worth noting for defaults: with --cuda-graph-max-bs unset, _determine_cuda_graph_bs picks 160 on a 32 GB card (free_memory_gb <= 80), so the out-of-the-box configuration attempts capture at sizes up to 160 and hits this immediately.

Defect 2 — b12x kills the worker at batch >= 2

With capture disabled the server is healthy at batch 1 (verified: finish_reason: stop, correct output, 28 prompt / 10 completion tokens). Sending a second concurrent request hangs that request for 907s and then kills the backend:

ValueError: force_tile_config fc2 tile (tile_k=32, tile_n=512) does not fit problem N/K=2560/640 at moe_block_size=8

  File "flashinfer/fused_moe/cute_dsl/blackwell_sm12x/moe_w4a16_kernel.py", line 5507, in compile_w4a16_fused_moe
    raise ValueError(
  ... via moe_w4a16_kernel.py:5950 _w4a16_fused_moe_launch_flat
  ... via moe_w4a16_kernel.py:6150 _w4a16_fused_moe_launch_op
  ... via torch.ops.flashinfer.w4a16_fused_moe_launch (moe_w4a16_kernel.py:7144)
  ... via freetoken/moe/nvfp4_backends.py:761 b12x_fused_experts

[FrontendAPI] ERROR  Backend supervisor: backend worker freetoken-TP0-scheduler exited
[FrontendAPI] ERROR  Backend worker is gone and cannot be restarted; stopping the API server

Subsequent requests get 502. There is no valid tile configuration for this model's expert shape (N/K=2560/640) at moe_block_size=8, so the batched decode path is unreachable on this checkpoint.

What works

--nvfp4-backend triton, everything else identical:

• captures [1, 2, 4, 8] in 7s / 0.13 GiB
• moe_cache_size=2606, num_pages=4097, KV 262,208 tokens = 6.19 GiB, 2.88 GiB free
• 8 concurrent streams, 8/8 complete, zero failures

Concurrency sweep (2000-token prompts, 400-token outputs, thinking off, per-stream nonce so the radix cache cannot serve one stream from another):

Two smaller notes from the same session

1. CuTe kernels are shape-specialised, so the first request at each batch size pays its own JIT. The very first b12x request took 2m20s wall for 10 output tokens — the prefill line reports input throughput (token/s): 0.18 for a 28-token prompt. Any benchmark must warm each concurrency level or its first sample is a compile. (Related: Proposal: reproducible end-to-end benchmark / regression harness #281 asks for exactly this kind of harness.)
2. A short-output probe hides the concurrency cost. A 300-token-prompt / 23-token-output sweep on the triton config read 22.5 → 18.3 tok/s per stream (only −19% at 8×) and looked excellent; the 400-token run above shows −78%. With short outputs the run is TTFT-dominated and barely decodes.

Happy to test patches on this hardware — the box is a rented 5090 and can be brought back up.