Repository navigation
Layer split with the low-RAM resident mode: 2-GPU numbers on 16+16 GB / 47 GB RAM (IQ3_XXS and Q2_0) #384
Description
Follow-up to #250 (closed). The answer there was that the low-RAM resident mode runs on one GPU by design. #364 describes a related workaround. These are measurements from a 2-GPU / 47 GB RAM box to help decide whether the setup should offer the layer split here.
Setup
- 2× RTX 5060 Ti 16 GB, PCIe 5.0 x8 each (~26–28 GB/s H2D when the link is healthy)
- i9-12900KF (AVX2, no AVX-512), 47 GB RAM, Ubuntu, Strata 0.1.30 (30ec18e)
--max-context 131072 --kv int8 --spec 4 --mtp,--expert-cache auto- CPU package power capped at 45 W (temporary, stock box cooler). "Uncapped" runs hit 100 °C and throttle, so treat them as approximate.
Bench: two short prompts (27 tokens → 150–180 generated), then cold 27k and 46k prompts (128 generated). Numbers come from the strata serve: prompt … log lines.
Results
| Config | GPUs | Short decode | 27k prefill / decode | 46k prefill / decode |
|---|---|---|---|---|
IQ3_XXS, setup default (--resident-experts, low-RAM), CPU uncapped |
1 | 39–50 t/s | 1,573 t/s | 1,060 t/s |
| IQ3_XXS, setup default, CPU 45 W | 1 | 30–37 t/s | 966 t/s | — |
IQ3_XXS, hand-edited json ("gpu": [0,1], "layer_split": "auto", --resident-experts removed), CPU 45 W |
2 | 54–61 t/s | 1,090 / 58 t/s | 1,340 / 65 t/s |
Q2_0, setup --gpus 0,1 --low-ram off (layer split 0–24 / 25–47), CPU 45 W |
2 | 66 t/s | 1,338 / 62 t/s | 1,760 / 76 t/s |
| Q2_0, same, CPU uncapped | 2 | 68 t/s | 1,500 / 80 t/s | 1,746 / 70 t/s |
Caveat: this host has an unrelated PCIe problem where H2D drops to ~7 GB/s after a load transition. It happened during the 2-GPU runs: the engine probe logged CUDA1 PCIe probe 7.4 GB/s on the IQ3_XXS 2-GPU start, and a measurement after the Q2_0 runs showed 6.9 GB/s. So the 2-GPU numbers are probably a lower bound.
Observations
- On a 16 + 16 GB pair, the layer split roughly doubles decode. IQ3_XXS goes from 30–37 to 54–61 t/s at the same 45 W cap. With one GPU, the experts that miss the cache run on the CPU (AVX2), so decode and prefill follow the CPU power limit. With two GPUs, the second card's expert cache covers ~98% of the routed mass, which removes most of that dependence.
- The engine still refuses resident + split. With
--resident-expertsand a layer split it exits with--resident-cpu-experts does not support layer splits or remote expert caches. Without the flag it runs, but RAM sits at 46/46 GB used and 0 available during long prompts. That works for a bench, but it is fragile next to anything else on the box. - Q2_0 with a layer split is the fastest config here and stays within RAM (37 GB used), but it is the lower-quality quant.
Request
- Engine: allow
--resident-experts(or a variant of it) together with the layer split, so the split IQ3_XXS config can run with predictable RAM instead of filling page cache to 0 available. - Setup: for 2-GPU machines where the low-RAM rule (RAM < arena + 10 GB) applies, offer the 2-GPU layer-split variant instead of silently dropping to one GPU, as Low memory 32 Gb RAM and 2 x 3090 24Gb working with workaround (0.1.31) #364 also suggests. At minimum, print that the json edit is an option and what it costs in RAM.
Happy to rerun anything on this box (including 0.1.31) if useful.
Activity
Gpanazio commentedon Oct 1, 2026
Update: IQ3_XXS with the 2-GPU layer split (json edited, no
--resident-experts), CPU uncapped and PCIe healthy this time. The engine probed 28.0 / 27.4 GB/s and the host measured ~26 GB/s before and after the run.The split is ~1.3× faster on short decode, about even at 27k, and 1.6× faster on 46k prefill. RAM was again at 46/46 GB with 0 available during the 46k prompt. The CPU package hit 100 °C because of the stock cooler, so expect some throttling in these numbers.
Niko1221 commentedon Oct 1, 2026
Thank you for the careful numbers. In the next release:
--gpus 0,1is honoured), and explains the RAM cost (see Low memory 32 Gb RAM and 2 x 3090 24Gb working with workaround (0.1.31) #364).--resident-expertswith a layer split. It will warn that the resident RAM mode doesn't support a split yet and reads the experts the GPUs don't hold through the file cache, which is what your hand-edited config does.A real resident mode across a split (predictable RAM) is a larger engine change. It's on our list, and we'd welcome a rerun on your box when it lands. Keeping this open until the release.
tests: golden --yes / Enter configs on five PCs (#403 #406 #364 #384)
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (#411 #412), setup reco…
Niko1221 commentedon Oct 1, 2026
Released in 0.1.33: setup offers the layer split on low-RAM PCs (see #364), and the engine runs
--resident-expertswith a split as--mmap-expertswith a warning instead of refusing. One note: your hand-edited config (without--resident-expertsand without--mmap-experts) copied all experts into RAM, which is why it reached 0 free; the mapped mode setup writes now reads them through the file cache instead. A resident mode across a split stays on our list. Thanks!Merge origin/main (0.1.33): keep the resident RAM mode with a layer s…
Gpanazio commentedon Oct 2, 2026
Rerun on current main (engine 0.1.34, HEAD 1678de3 - includes the 0.1.33 changes), built from source on Linux (Release, sm_120, native experts). Same box: 2x RTX 5060 Ti 16 GB, 47 GB RAM, i9-12900KF. Differences from the first post: CPU not power-capped (new air cooler), and PCIe healthy the whole time (engine probe 27.9 / 27.3-27.5 GB/s, both links stayed Gen5 x8 before and after).
IQ3_XXS, 2-GPU layer split (auto, K=24: CUDA0 layers 0-23, CUDA1 24-47),
--max-context 131072 --kv int8 --spec 4 --mtp,--expert-cache auto. One run each, same json, only the experts flag differs.What changed for us
--resident-experts+ layer split now starts. It prints the warning, logsexperts via mmap, and serves normally. No more--resident-cpu-experts does not support layer splits.MemAvailableevery 2 s from load through the long prompts. Lowest value: ~42.0 GB with--resident-experts, ~42.9 GB with--mmap-experts. With the hand-edited json on 0.1.30 it was 0 available at the 46k prompt. (Caveat:MemAvailablecounts reclaimable file cache, so this shows the experts are no longer copied into anonymous RAM, not that file-cache pressure is gone.)Numbers (tokens/s)
--resident-experts+ split (runs as mmap)--mmap-experts+ splitCaveats
--resident-expertsarm ran first with a cold file cache (first request prefilled at 12 tok/s, the 21k prompt at 797 tok/s); the mmap arm ran second on a warm cache. Read the mmap row as the warm case. One run each, no repeats, so the two rows are not an A/B of the flags.iommu=pt, cooler), so the decode gains cannot be attributed to the engine version.Thanks for the quick change. Happy to rerun when the real resident mode across a split lands.
Gpanazio commentedon Oct 2, 2026
Follow-up: a cleaner A/B on the same box (2x RTX 5060 Ti 16 GB, 47 GB RAM, i9-12900KF), same prompts and same json, IQ3_XXS with the 2-GPU layer split, warm file cache. PCIe healthy throughout (engine probe 27.8 GB/s, both links Gen5 x8 before and after), CPU max 68 C, so no throttling.
Arms: (A) engine 0.1.30 with the json from my first post (no
--resident-experts, no--mmap-experts); (B) engine 0.1.34 (main, 1678de3) with that same json; (C) 0.1.34 with--mmap-experts. Order A, B, C, C, B, A to cancel ordering effects. Per arm: 2 server starts x 2 requests per size, so 4 per cell. Cells are medians (tokens/s); decode is shown with min-max because it is noisy.--mmap-expertsWhat I take from it:
--mmap-expertsis not faster than 0.1.30 on this box: prefill is about 4-5% lower (1,406 vs 1,473 at 21k, 1,516 vs 1,595 at 35.5k) and decode is within the noise. The earlier decode gains I saw were the host (iommu=pt, cooler), not the engine version.--mmap-experts, against about 0.5 GB with the 0.1.30 json. That is what the change was for, and it costs a few percent of prefill.session is up (engine 0.1.34)and the prompt-path lines, but /health never answered. The same json on 0.1.30 loads in about 40 s. I do not know the cause (RAM pressure is my guess, not verified), and my harness gave up at 600 s, so I cannot say it would never have come up. Mentioning it because it is the "no flag" path your last comment described.One log line to ignore:
cudaMalloc(1538035200) for the weight arena failedappears twice in every run, including the ones that work, so it looks benign.Single box, 4 samples per cell, so treat the 4-5% as indicative, not exact. I can send the raw logs for the B runs if useful.
tests: golden --yes / Enter configs on five PCs (#403 #406 #364 #384)
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (#411 #412), setup reco…
tests: golden --yes / Enter configs on five PCs (Niko1221#403 Niko122…
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (Niko1221#411 Niko1221#412
tests: golden --yes / Enter configs on five PCs (Niko1221#403 Niko122…
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (Niko1221#411 Niko1221#412