Repository navigation
0.1.27 on 16 GB + vision: the CJK draft leaves 156 MiB VRAM free (below the 256 MiB stall threshold); setup's fixed reserve 700 no longer suffices #199
Description
Summary
On a 16 GB card with vision enabled, 0.1.27's CJK draft subset uses ~180 MiB more VRAM than 0.1.24's draft. That leaves 156 MiB free after loading, below the engine's own 256 MiB threshold (... MiB of VRAM free with everything loaded - LOW: requests may stall). setup still writes a fixed --vram-reserve-mib 700 (VISION[...]["reserve_mib"], setup.py:109-110), so every start on this setup sits in the "may stall" range. The engine itself suggests about 1,058.
Evidence (same machine, same config except the draft)
RTX 5070 Ti 16 GB (sm_120), Linux, IQ3_XXS native pack, --kv int8 --kv-resident 32768, --spec 4, --vision (CPU encoder), --vram-reserve-mib 700, --expert-cache auto, --max-context 131072.
| Engine / draft | Expert-cache slots | VRAM free after load |
|---|---|---|
0.1.24, --mtp .../rt-full |
5,612 | 336 MiB |
0.1.27, --mtp .../rt-cjk |
5,611 | 156 MiB (LOW) |
The only difference between the two run configs is the --mtp path. The expert cache kept the same number of slots, so the ~180 MiB went elsewhere, most likely to the larger draft vocabulary (40,525 → 106,299 ids, #137). With --max-context 262144 it is 154 MiB, so context length is not the cause; streamed KV keeps VRAM flat.
The fix costs almost nothing
Same prompts before and after, greedy, run straight after a restart. Each prompt is ~60K tokens read from zero, then 400 generated tokens.
--vram-reserve-mib |
Expert slots | VRAM free | Prompt tok/s | Output tok/s |
|---|---|---|---|---|
| 700 (setup default) | 5,468 | 154 MiB | 3,038 / 3,090 | 80.4 / 86.2 |
| 1,058 (engine's suggestion) | 5,254 | 510 MiB | 2,993 / 3,000 | 81.2 / 88.3 |
This is ~−2% on prompt reading, no change in output speed, and the warning is gone.
Suggestion
Make setup's reserve account for the draft subset it now requires, or compute the reserve from a trial load: aim for ≥256 MiB free plus a margin, which is what the engine's hint already does. A user who starts from the service or the web UI never sees that log line. The docs' troubleshooting row ("A request never finishes ... GPU ran out of VRAM") describes exactly this state.
Activity
jazir555 commentedon Sep 30, 2026
The larger draft head which regresses english text performance should be optional. It's an unnecessary performance regression for english users.
Niko1221 commentedon Sep 30, 2026
Confirmed, thanks for the careful A/B. The expert cache sizes itself before the draft layer allocates its head, so 0.1.27's larger CJK subset came out of the 700 MiB reserve. That affects every config, not only vision. 0.1.28 (in testing now) counts the draft head when it sizes the cache, so the reserve is kept again. It also adds a setup option to keep the smaller English-only subset (
--draft-vocab en), which @jazir555 asked for. Until then, add"--vram-reserve-mib", "1100"to theargsof yourstrata-<model>.json(or pass--vram-reserve-mib 1100to the engine).engine: the expert cache reserves the draft head's VRAM (0.1.27's CJK…
Niko1221 commentedon Sep 30, 2026
Fixed in 0.1.28: the expert cache now reserves the draft head's VRAM before it sizes itself, so the 700 MiB reserve stays free (on our 12 GB card: 210 -> 379 MiB free with everything loaded; the cache holds ~4% fewer experts).
--draft-vocab en(START-HERE.bat --setup --draft-vocab en) keeps the English/code subset if you'd rather have the VRAM. You can remove the--vram-reserve-mibworkaround after updating. Thanks @demon851113 and @jazir555!engine: the expert cache reserves the draft head's VRAM (0.1.27's CJK…
jazir555 commentedon Sep 30, 2026
Thanks @Niko1221!
Layer split: keep the draft head's room out of the last card's cache
engine: the expert cache reserves the draft head's VRAM (0.1.27's CJK…
engine: the expert cache reserves the draft head's VRAM (0.1.27's CJK…
engine: the expert cache reserves the draft head's VRAM (0.1.27's CJK…
generate: review pass on the startup VRAM plan (Niko1221#765)
layer split: the last stage books the draft head; the VRAM check read…
engine: the expert cache reserves the draft head's VRAM (0.1.27's CJK…
engine: the expert cache reserves the draft head's VRAM (0.1.27's CJK…