Repository navigation
Unhandled CUDA OOM in prefill expert workspace kills the engine; server keeps accepting requests that never return (root cause behind #20-style headless state; likely also #72) #110
Description
Summary
When real free VRAM at request time is lower than the memory plan assumed at boot, the prefill MoE workspace allocation raises torch.OutOfMemoryError, the exception is not handled at the engine-loop level, and the engine process exits. The API frontend stays up and keeps accepting requests — which then hang forever with no error, no log line, and no recovery short of kill -9. I believe this is the mechanism behind #20 ("Backend worker is gone and cannot be restarted"), and very likely behind #72's prefill hangs on a desktop 16GB card (where compositor/OS VRAM shifts under the resolver's boot-time reading constantly; it also cleanly explains that issue's odd "gradual prompt escalation works, direct jump hangs" behavior via allocator block reuse).
Environment
- FreeToken v0.1.2 @ f0abe58, source install, torch 2.11.0+cu130, CUDA 13.0.1 (nvidia/cuda:13.0.1-devel-ubuntu24.04 container), driver 595.91.07
- RTX PRO 6000 Blackwell Workstation 96GB (sm_120), EPYC 128c / 503GB RAM
- Model:
nvidia/MiniMax-M2.5-NVFP4(equivalent failure also observed withdeepseek-ai/DeepSeek-V4-Flash-0731) ft bench bwprofile: CPU STREAM 135.7 GB/s, PCIe 28.0/28.7 GB/s, nvfp4 CPU-MoE 88.9 GB/s → hybrid 3.38×
The failing allocation
File ".../freetoken/moe/fused_nvfp4.py", line 336, in fused_experts_nvfp4
ic3 = torch.empty((M, top_k, H), device=dev, dtype=dt)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 156.00 MiB.
GPU 0 has a total capacity of 94.97 GiB of which 126.94 MiB is free.
Including non-PyTorch memory, this process has 86.06 GiB memory in use.
Process 34312 has 8.54 GiB memory in use.
The workspace is proportional to prompt tokens (M), so the prompt size at which this fires depends entirely on how much slack the plan left — which is why the failure presents as a config-dependent "hang cliff" (from ~180 tokens on a 16GB desktop card per #72, to 2k–4k tokens on 96GB under various plan-vs-reality gaps, to "never" on a clean dedicated GPU).
After the OOM the engine process is gone (confirmed via py-spy: the engine pid no longer exists while the tokenizer worker idles on zmq), the GPU sits at idle wattage, and the frontend accepts and queues requests indefinitely.
Deterministic reproduction
- Start clean (GPU at ~0 MiB used):
ft serve --model nvidia/MiniMax-M2.5-NVFP4 --moe-cache-auto→ resolver plans against ~94 GiB free (moe_cache_size=10114 num_pages=8209). - After the server is ready, consume ~8 GB from another process (I used
torch.empty(8*1024**3, dtype=torch.uint8, device="cuda:0")held in a sleep loop). This stands in for any organic consumer: a dying predecessor engine, another workload, a desktop environment. - Send a few growing prompts (~1k, 1.6k, 2.2k tokens — all fine), then a ~3.3k-token prompt.
- The 3.3k request never returns; the engine has died with the traceback above; every subsequent request also hangs forever.
Without step 2, the same sequence (and far larger prompts, verified to 7.1k tokens) completes at full speed.
Why this bites in practice
- Restarting
ft servewhile a previous instance is still dying reproduces the plan-vs-reality gap organically (the old engine's memory is still held during the new resolve or run). Engine workers also survivepkill -f "ft serve"(their cmdline ismultiprocessing.spawn ...), which makes this restart race easy to hit. - On consumer/desktop GPUs the OS takes a variable bite out of VRAM continuously — Prefill hangs on RTX 4060 Ti (SM89) for prompts beyond warmup length #72's tiny thresholds are exactly what this predicts.
Suggested fixes (in order of leverage)
- Catch
OutOfMemoryErrorper-request in the engine loop — fail that request with a 5xx and keep the engine alive. Smallest change, removes the catastrophic mode. - Frontend liveness on the backend worker — detect the dead engine and refuse requests with a clear error (would fix Backend worker is gone and cannot be restarted; stopping the API server #20 for every cause, not just this one).
- Pre-allocate the prefill workspace at
max_extend_tokensduring_warmup_prefill— moves any OOM to startup where it's visible, and structurally removes the mid-request death window. - Resolver safety margin (plan against boot free minus a configurable reserve), since the 0.9 ratio of a boot-time reading is the root exposure.
Offer
This came out of a day of systematic investigation on the hardware above; I'm posting the full debug trail (hypothesis table, experiment register, sweep data) as the first comment on this issue. The rig (96GB Blackwell workstation + 503GB EPYC) is available for validating patches — glad to run candidate fixes against the deterministic repro.
Companion issue filed for a distinct mechanism found in the same investigation: prompts longer than the KV pool wait in admission forever (engine healthy, no error) — see #111.
Activity
CygnusBill commentedon Aug 23, 2026
For anyone who wants the complete evidence chain (or to see which tempting wrong theories to skip): this is how we got from "prefill hangs somewhere around 2k tokens" to the OOM root cause, across ~9 experiment phases and ~24 server lifecycles in one day. Posting it because several intermediate theories looked very convincing and were falsified only by controlled cells — future investigators will hit the same traps.
Symptom as first observed
A ~4k-token prompt to a warmed
hybridserver hung >15 minutes (client timeout at 900s); the engine burned ~5 CPU-hours across ~20 threads during the window; signals ignored;kill -9only. Matches #72's description closely.Hypotheses run to ground
--disable-moe-prefill-overlapoffload(no CPU executor)num_pagesfused_nvfp4.py:336; engine pid dead under py-spy while tokenizer idlesKey experiment cells
--max-prefill-length 2048× model (V4-Flash control). Located config-dependent cliffs; every cliff later explained by either this issue's OOM (polluted-memory epochs) or the companion admission issue (prompt >num_pages).--max-prefill-length 2048passed everything (cost model: exactly N chunks × the full expert-stream floor — 9.0s at 2 chunks, 13.5s at 3, to the decimal).--moe-cache-size, warmed ladders, back-to-back sweeps) passed when re-run behind strict hygiene gates (GPU <500MiB before boot, port-bindable check). The only remaining reproduction was creating a genuine plan-vs-reality gap — which is the bug.10114/8209against 94GiB), hold 8GB from a second process after ready, warm with 1k/1.6k/2.2k prompts, send ~3.3k → engine dies with the OOM traceback; frontend hangs all subsequent requests. Engine pid captured before the hang and confirmed dead mid-request viapy-spy(needsdocker exec --privilegedin containers).Explaining #72's observations specifically
Measurement-integrity traps (so you don't repeat them)
moe_cache_size/num_pagesand moves the cliff — this both contaminated our early attributions and turned out to be the bug itself. Gate every run onnvidia-smi≈ 0 used.pkill -f "ft serve"(cmdline ismultiprocessing.spawn ...) and keep holding VRAM and the torch-distributed port (serialEADDRINUSEon restart). Kill by"ft serve|freetoken|multiprocessing".Raw sweep logs, all driver scripts, and the full write-up are preserved; happy to attach any of it here on request.
gdevenyi commentedon Sep 5, 2026
Reproduced on very different hardware, and at a different allocation site — which I think strengthens the case that the problem is the missing handler rather than any one workspace.
Environment: 2 x RTX 6000 Ada 48 GB (sm_89), TP=2,
nvidia/Qwen3.8-Flash-Next-NVFP4(qwen4_exp), offload MoE backend, pinned PLE. Flags at the time:--memory-ratio 0.94 --num-tokens 262144 --max-extend-tokens 32768 --max-running-requests 16 --tp-size 2. Free VRAM after init: ~2.7 GiB.What happened, ~20 s after
API server is ready to serve, on the first substantial prompt — both ranks, within the same second:So the fatal allocation was not the MoE prefill workspace (
fused_nvfp4.py:336) but the GatedDeltaNet WY-transform scratch in chunked prefill. Same shape of failure: a request-time buffer proportional to the prefill chunk, sized against headroom the boot-time plan never charged for, on a card with 47 of 47.37 GiB already committed by that plan.Two details that may be useful:
The "config-dependent cliff" is exactly as you describe, and it is sharp. The same build, same card, same
--memory-ratio 0.94, serving the same checkpoint, has been up continuously since with the defaultmax_extend_tokensand has never hit this — including a deliberate 100,000-token prompt benchmarked today (TTFT 40 s, decode 96.7 tok/s, no incident). Only raising--max-extend-lengthto 32768 makes it fatal. Nothing about the failure tells the operator that; it presents as "the server died on a long prompt".Our supervisor did notice, unlike your report: 10 s after the worker exited, the frontend logged
Backend worker is gone and cannot be restarted; stopping the API serverand shut down, rather than accepting requests that never return. That may be a TP>1 path difference (two ranks dying at once is harder to miss) or a version difference — worth knowing that the silent-hang variant is not universal, since it changes how the issue presents in the wild.Practical consequence, and why I care about the fix rather than the workaround: this is what caps
--max-extend-lengthfor us. We are not on the default because 32k chunks are wrong for this model — a 48 GB card at TP=2 has the bandwidth for them — but because an OOM there takes the server down instead of degrading. If the engine caught it and either retried with a halved chunk or refused the request, the knob would become tunable instead of a trap. Charging the prefill workspace in the boot plan (#303) helps the startup case but cannot cover this one, since the workspace scales with the request.🤖 Generated with Claude Code
https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
issue FlashML-org#110: v5 holdout-only execution authority (h109 admi…
Merge PR #32: Issue FlashML-org#110 v5 calibration and holdout producers
ogino commentedon Oct 6, 2026
Datapoint: prefill CUDA OOM kills the worker on RTX 5070 Ti 16 GB
Same failure class as this issue: an unhandled CUDA OOM during prefill exits the scheduler, and the frontend then stops because the worker cannot be restarted. The port closes. It is not a clean 400.
0.1.3+g581fdca31nvidia/Qwen3.8-Flash-Next-NVFP4--moe-strategy offload --nvfp4-backend triton --ple-backend disk --text-model-only --max-running-requests 1input throughput175.59 tok/s, HTTP 200). The next prefill died inqwen4_exp/gdn.pygdn_prefill_chunk_fla:--max-prefill-lengthbelow 5408, or shrinking--moe-cache-sizeto leave a prefill workspace. Short requests after a restart still serve.