Repository navigation
Low memory 32 Gb RAM and 2 x 3090 24Gb working with workaround (0.1.31) #364
Description
I have this rig, and going through normal setup for IQ3_XSS Swift it forced it to 131k context and only utilising a single card. No obvious way around this, and talk of low-mem and layer partitioning don't work yet.
But digging around I edited the strata-swift-iq3_xxs.json. Changed "--max-context" to 262144, "gpu": [0,1] and added "layer_split": "auto"
The result is both cards utilised, 11,094 experts on GPU0 and 8,579 experts on GPU1, 19,673 of 24,576 experts resident in VRAM and 262144 context length. And it's running broadly 95-100 output tokens/second with acceptable prefill.
So
a) amazing tool, makes Qwen3.8 Flash Next genuinely usable, my next exploration is 27b int8 vs IQ3_xxs quality, but the performance difference is now slower, but good compared to 27b.
b) maybe the default setup could understand these constraints a bit better to avoid needing the manual tweaks.
I accept the weird combo of low System RAM and Dual 3090 might be a bit edge case, but RAM prices are a little horrific....
Activity
[-]Low memory 32 Gb RAM and 2 x 3090 24Gb working with workaround (0.1.31[/-][+]Low memory 32 Gb RAM and 2 x 3090 24Gb working with workaround (0.1.31)[/+]Niko1221 commentedon Oct 1, 2026
Thanks for this, and for the measurements. Your edit is exactly what setup should have offered. In the next release, setup on a low-RAM PC with two or more GPUs asks: one GPU with the experts kept in RAM (steady RAM use, the default), or all GPUs together with the experts read through the file cache (faster, as you and #384 measured, but RAM can fill up during long prompts). An explicit
--gpus 0,1is honoured, and the context is a recommendation, not a cap (#406). Keeping this open until the release.tests: golden --yes / Enter configs on five PCs (#403 #406 #364 #384)
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (#411 #412), setup reco…
Niko1221 commentedon Oct 1, 2026
Released in 0.1.33: on a low-RAM PC with several GPUs setup asks one GPU (default) or all of them (experts through the file cache),
--gpus 0,1is honoured, and the context is a recommendation. Thanks for showing the way!Merge origin/main (0.1.33): keep the resident RAM mode with a layer s…
tests: golden --yes / Enter configs on five PCs (#403 #406 #364 #384)
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (#411 #412), setup reco…
tests: golden --yes / Enter configs on five PCs (Niko1221#403 Niko122…
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (Niko1221#411 Niko1221#412
tests: golden --yes / Enter configs on five PCs (Niko1221#403 Niko122…
engine: --resident-experts with a layer split runs as --mmap-experts …
setup: recommend, never force - longer contexts, more GPUs, a bigger …
docs: the RAM rule recommends 128K, the low-RAM mode on several GPUs,…
Engine 0.1.33: a portable image encoder again (Niko1221#411 Niko1221#412