Repository navigation
[2x RTX 2080 Ti ] 262K context can OOM the weight arena / hard-freeze host; 204800 is stable #486
Description
[Linux][2x RTX 2080 Ti] 262K context can OOM the weight arena / hard-freeze host; 204800 is stable
Summary
On a Linux system with 2× RTX 2080 Ti 22 GB using a layer split, a 262144-token context can exhaust VRAM during startup / allocation and, in the worst case, make the whole machine stop responding. Reducing the context to 204800 makes the same setup stable.
The concrete allocation failure I observed was:
cudaMalloc(1538035200) weight arena failed
This was with IQ3_XXS + vision enabled. Because the failure can hard-freeze the host, I have not deliberately reproduced it again on 0.1.35 yet. I checked the 0.1.35 changes; #448 improves prompt-chunk diagnostics for split GPUs, but it does not appear to change this allocation path.
Environment
- OS: Ubuntu 24.04
- RAM: 126 GB
- CPU: AVX-2, no AVX-512
- GPU: 2× NVIDIA GeForce RTX 2080 Ti 22 GB
- Interconnect: NVLink (NV2)
- GPU topology: both GPUs on CPU affinity 0-23
- Model: Swift 1.5 / Qwen3.8 Flash-Next GSQ-RCO IQ3_XXS
- Vision: enabled
- Layer split: auto, CUDA0 then CUDA1
- KV: int8
- KV resident: 32768
- Speculation: 4
- PCIe fraction: 0.35
- Spec min p: 0.70
- Pool workers: 9
The machine normally runs near full GPU utilization during generation.
Reproduction shape
Problematic configuration:
--context 262144
--kv int8
--kv-resident 32768
--spec 4
--spec-min-p 0.70
--pcie-frac 0.35
--pool-workers 9
--vision
--layer-split auto
With IQ3_XXS, this reached a point where a ~1.43 GiB device allocation failed:
cudaMalloc(1538035200) weight arena failed
On at least one run, the system became effectively unresponsive rather than failing cleanly.
Stable comparison
The same machine is stable with:
--context 204800
--prefill 4096
--reserve 1800
--kv int8
--kv-resident 32768
--spec 4
--spec-min-p 0.70
--pcie-frac 0.35
--pool-workers 9
So the practical boundary on this machine is somewhere between ~205K and 262K with this model / vision / split configuration.
Why this seems worth guarding
At 262K the combined VRAM consumers include:
- dense/model weights
- expert cache
- streamed/resident KV
- prefill buffers
- MTP/speculative buffers
- vision allocations
- CUDA workspace / weight arena
The engine can apparently commit enough of these earlier allocations that a later large weight-arena allocation fails. On this hardware the failure can be severe enough to make the host unusable.
Request
Could Strata do one or more of the following?
- Budget the late weight-arena / workspace allocation before committing the expert cache / other large VRAM users, especially with vision + layer split.
- Keep a larger automatic safety margin when
--max-contextis very large. - If the requested context cannot fit safely, fail before loading / allocating everything, with a message suggesting a lower context such as 204800.
- If possible, make allocation failure recover cleanly instead of leaving the GPU / host in a state that can hard-freeze Linux.
- Log a compact per-GPU VRAM budget before the risky allocation (weights, expert cache, KV resident, prefill/workspace, vision, reserve), which would make this much easier to diagnose.
I can test a patch or provide a full startup log if there is a specific tracing flag you want enabled.
Note about 0.1.35
I saw the new #448 diagnostics around prompt-chunk borrowing and split-card cache constraints. My GPUs are symmetric 22 GB cards, and this failure is a later cudaMalloc(... weight arena ...) OOM rather than the small-secondary-GPU prompt-chunk case, so I don't think #448 addresses this particular failure.
Activity
HetagKoroev commentedon Oct 2, 2026
Same on Windows 11
wxaef commentedon Oct 2, 2026
{
"exe": "/home/wuxing/Strata-main/engine/strata",
"args": [
"--pack",
"/home/wuxing/Strata-data/packs/swift-iq2_xs",
"--native",
"/home/wuxing/models/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf",
"--ple-gguf",
"/home/wuxing/models/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf",
"--expert-profile",
"/home/wuxing/Strata-main/data/expert-profile.bin",
"--expert-cache",
"auto",
"--prefill",
"auto",
"--spec",
"4",
"--mtp",
"/home/wuxing/Strata-data/mtp/rt",
"--max-context",
"262144",
"--kv",
"int8",
"--kv-resident",
"32768",
"--vision",
"--vram-reserve-mib",
"700",
"--pcie-frac",
"0.35",
"--spec-min-p",
"0.70",
"--pool-workers",
"7"
],
"cwd": "/home/wuxing/Strata-main",
"tokenizer": "/home/wuxing/Strata-data/packs/swift-iq2_xs/tokenizer",
"model_name": "swift-1.5-iq2_xs",
"log": "/home/wuxing/Strata-main/strata-swift-iq2_xs.log",
"lib_dirs": [
"/usr/local/cuda-12.8/bin",
"/usr/local/cuda-12.8/lib64"
],
"port": 8000,
"gpu": [
0,
1
],
"gpus_asked": true,
"layer_split": "auto",
"host": "0.0.0.0",
"vision": {
"exe": "/home/wuxing/Strata-main/engine/strata-vision",
"mmproj": "/home/wuxing/models/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf",
"model": "/home/wuxing/models/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf",
"gpu": true,
"max_tokens": 1024
}
}
too
Niko1221 commentedon Oct 2, 2026
Thanks. The 1.43 GiB allocation that fails is the weight arena: the first large VRAM allocation, made before the KV cache and the expert cache, and its size doesn't depend on the context. So when it fails, something else held that VRAM at that moment, for example the previous engine still releasing during a restart, or the GPU image encoder. And the hard freeze at 262K is more likely host RAM (pinned memory for KV streaming) than VRAM. Could you post the full startup log of a failing 262K start (
strata-<model>.log), the RAM use at the freeze (free -gor Task Manager), andnvidia-smijust before starting? Then we can make the engine plan the budget first and say which context fits.#486: a weight arena that does not fit says how much VRAM was free
Niko1221 commentedon Oct 3, 2026
In 0.1.38 a weight arena that doesn't fit says how much VRAM was free on that card at that moment (and, on a layer split, how much the stage needed). That shows whether something else was holding the card.
Keeping this open for the bigger part: the engine should plan the per-GPU budget before it allocates (weights, KV, expert cache, prompt buffers, images, reserve), print it as one compact line, and stop early with the context that fits instead of failing late. The full
strata-<model>.logof a failing 262K start, plusfree -gat the freeze, is still what we need to see whether the freeze is VRAM or pinned host RAM.csyujianghong commentedon Oct 5, 2026
262144 starts fine on a 4-card pool (2×2080Ti 22G + 2×V100 32G, 110 GB VRAM): full startup log + RAM/VRAM before and after
Following Niko's request in this thread for a 262K startup log,
free -g, andnvidia-smijust before starting.Result: 262144 started clean in ~80 s. No
weight arena failed, no host freeze. So this is a negativedata point on the failure, not a reproduction. I attach everything asked for anyway, because it bounds the
freeze question.
Environment
AVX-2 expert-kernel path by design — the engine prints
this CPU has no AVX-512: the expert kernels run on AVX-2, which What are the requirements for the CPU instruction set? #34/Missing AVX-512 support on Intel Core Ultra 9 285K (Arrow Lake) – engine fails to start #142 confirm is the supported path for native packs.archs [70,75], CUDA 12.9--vision, MTP,--spec 4--max-context 262144 --kv int8 --kv-resident 0 --vram-reserve-mib 2048 --pcie-frac 0.36 --layer-split 20,40,46,gpu=[1,2,0,3]nvidia-smijust before startingAfter everything is loaded
Host RAM rose monotonically 7.2 GB → 59.5 GB with no spike and no swap (2 s sampling, 38 samples).
Why this matters for the freeze question
Same host RAM class as the report (126 GB), but 262K here only takes 59.5 GB of 126 GB, available 66 GB,
swap untouched. There is no sign of pinned-host-RAM exhaustion on this box, which argues the boundary in
this issue is a plain VRAM budget problem (44 GB vs 110 GB), not pinned memory — the opposite of the
host-RAM hypothesis, so please treat it as a counter-example rather than confirmation.
One thing worth noting for small-card splits
The 262K run starts, but the smallest stage already throttles the prompt path:
Full startup log (59 lines)
I did not attempt to reproduce the failure itself: reaching the 44 GB boundary from the original report would
mean shrinking this box's split by hand, and a hard freeze on this host would take down unrelated work. If the
startup log or the RAM numbers would be more useful in another shape, tell me which and I will collect it.
#486: a weight arena that does not fit says how much VRAM was free
Niko1221#486: a weight arena that does not fit says how much VRAM was…
Niko1221#486: a weight arena that does not fit says how much VRAM was…
Niko1221 commentedon Oct 7, 2026
0.1.40 prints the exact size that failed and the VRAM free on that card when the arena allocation fails. Could you run 262K again on 0.1.40 or newer and post that line? The host freeze is about system RAM, not VRAM; a bigger memory budget is #765.
csyujianghong commentedon Oct 9, 2026
Re-ran 262K on 0.1.40.2 as asked. It starts clean again on this box, so there is no failed-allocation line to post — the full fresh start-up log is below instead (serve stdout + engine ledger; environment at the top).
Since my Oct 5 post: the engine is stock 0.1.40.2 (was a local 0.1.38 build), and the two 2080 Ti now run a separate vLLM job (TP2, :8002), so the engine's GPUs are the 2×V100 pair (layer-split 24) with UD-Q4_K_XL instead of IQ3_S — still
--max-context 262144, single request (no--batch).Key lines:
expert cache auto: 27.20 GiB free, 1024 MiB reserved (+0 MiB for the draft head) -> 7043 slots·resident RAM mode: 21.08 GiB of experts in RAM (page-locked), 8975 in the GPU cache·751 MiB of VRAM free with everything loaded· every verify window passes (upload no error, sync no error).Present snapshot (hours after start, not at the start moment):
nvidia-smi— 2080Ti#0 19560/22528 (vLLM) · V100#1 31785/32768 · V100#2 31975/32768 · 2080Ti#3 20327/22528 (vLLM).free -g— total 123, used 43, buff/cache 74, available 80; swap used 0.Strata v0.1.40.2 startup log — 2×V100 / UD-Q4_K_XL / layer-split 24
Captured 2026-10-09 ~21:01 local (fresh restart of the running instance). Two views of the same
start: the serve layer's stdout, then the engine's own start-up ledger. Paths sanitized
(
/home/<user>); nothing else altered.Environment
--servemode--layer-split 24,--mmap-experts --resident-cpu-experts --resident-budget-gib 40,--vram-reserve-mib 1024,--max-context 262144 --kv int8 --kv-resident 0,--spec 4 --spec-min-p 0.70; single request at a time (no--batch)1) serve layer (stdout)
2) engine start-up ledger
Niko1221 commentedon Oct 13, 2026
Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.
If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.