Repository navigation
[Bug] setup.sh can't setup dual nvidia GPUs #112
Description
Running on Ubuntu 26.04 + 64GB system memory + dual 5060 Ti 16GB
If I setup again with single GPU, then modify the json config with "gpu": [0, 1]. It would work.
Also calibration mode doesn't make use of two GPUs neither.
$ nvidia-smi
Mon Sep 28 23:06:32 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.91.07 Driver Version: 595.91.07 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5060 Ti Off | 00000000:04:00.0 Off | N/A |
| 0% 33C P8 2W / 180W | 13MiB / 16311MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 5060 Ti Off | 00000000:05:00.0 Off | N/A |
| 0% 39C P8 4W / 180W | 13MiB / 16311MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
$ ./setup.sh --gpus 0,1 --family qwen --model IQ3_XXS --kv int8 --vision no --context 131072 --experimental-speed-projection off
Strata - Qwen3.8-Flash-Next on a normal PC (NVIDIA GPU + system RAM + CPU)
=== Step 1: checking your PC ===
[ok] GPU: NVIDIA GeForce RTX 5060 Ti, 15.9 GB VRAM, compute capability 12.0, driver 595.91.07
2 NVIDIA GPUs: Strata uses GPU 0 - choose another with --gpu N:
0: NVIDIA GeForce RTX 5060 Ti, 16 GB
1: NVIDIA GeForce RTX 5060 Ti, 16 GB
[ok] RAM: 60 GB
[ok] CPU: Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz (AVX2)
=== Step 2: your choices ===
[ok] model: Qwen3.8-Flash-Next
1) Q2_0 2-bit, the fastest; download 66 GB, uses ~34 GB of RAM
2) IQ2_XS 2-bit i-quant, a little better quality, close in speed; download 68 GB, uses ~36 GB of RAM
3) IQ3_XXS 3-bit i-quant, better quality, slower (more CPU work per token); download 76 GB, uses ~43 GB of RAM
4) IQ3_S 3.5-bit i-quant, the best quality (matches the full model), the slowest; needs a 64 GB PC with little else running; download 84 GB, uses ~50 GB of RAM <- needs 62 GB RAM, you have 60
[ok] size: IQ3_XXS
[ok] context: 131072 tokens
[ok] KV cache: 8-bit
[ok] images: off
[ok] experimental speed projection: off
=== Step 3: Python packages ===
[ok] numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow, psutil already installed
=== Step 4: the Strata engine ===
[ok] llama.cpp 3cf0325 (gguf-py, ggml, mtmd)
[ok] ready-made engine already installed
[ok] engine: /home/steven/Projects/Strata/engine/strata
=== Step 5: downloading Qwen3.8-Flash-Next IQ3_XXS ===
[ok] Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf already downloaded
[ok] Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf already downloaded
[ok] model files present
=== Step 6: preparing the model for Strata ===
[ok] model prepared: /home/steven/Projects/Strata-data/packs/iq3_xxs
[ok] MTP draft layer: /home/steven/Projects/Strata-data/mtp/rt
=== Step 7: writing the start script ===
[ok] layer split across GPUs [0, 1] (auto)
[X] there is no GPU [0, 1]: 0 = NVIDIA GeForce RTX 5060 Ti, 1 = NVIDIA GeForce RTX 5060 Ti
Setup stopped. Fix the item above and run it again - everything already done is kept and skipped.
Activity
megakilo commentedon Sep 29, 2026
Another minor bug is that "monitor dashboard" doesn't show multiple GPUs
sal7g86 commentedon Sep 29, 2026
I have the same problem on linux mint:
this is my nvidia-smi output:
pdsmike commentedon Sep 29, 2026
Confirming. After git pull on Linux:
./setup.sh --setup --gpus 0,1 --layer-split auto
After running setup without params new engine is building correctly, but the same issue when running again ./setup.sh --setup --gpus 0,1 --layer-split auto
ymja commentedon Sep 29, 2026
Same issue with 5080+3090 combo in Win11.
All looks fine (both cards "detected") in the setup just as in the logs above but the script stops at "there is no GPU [0, 1]"
pdsmike commentedon Sep 29, 2026
Quick fix
insert these two lines after the ], closing lib_dirs in the_model.json (strata-swift-iq3_xxs.json)
"gpu": [0, 1],
"layer_split": "auto",
before "port" and use
./run-the_model.sh:
"lib_dirs": [
"/usr/local/cuda-13.3/bin",
"/usr/local/cuda-13.3/lib64"
],
"gpu": [0, 1],
"layer_split": "auto",
"port": 8080
}
Niko1221 commentedon Sep 29, 2026
Thanks @megakilo, @sal7g86 and @pdsmike - confirmed and fixed for the next release.
In 0.1.21,
--gpus 0,1stops at the end of setup ("there is no GPU [0, 1]"), on Linux and Windows alike: thecalibration lookup that runs last didn't expect a list of GPUs. Until the release, the workaround is the one
@megakilo found: set up with one GPU, then put
"gpu": [0, 1]and"layer_split": "auto"in the model'sstrata-*.jsonand start it again.In the next release you won't need any flags:
--gpus 0,1,--gpus allor--gpu 0,1also work at any start, and are remembered;--calibratemeasures the split as it runs;Tested on a Windows PC with an RTX 5080, RTX 2080 Ti and RTX 3090 (the 2080 Ti is refused with its reason). I don't
have a Linux PC with several GPUs; if you can try the release on yours, that would confirm the Linux side.
Monitor: every card of a layer split (issue #112)
Niko1221 commentedon Sep 29, 2026
Fixed in 0.1.22 (released today):
--gpus 0,1no longer stops at the end of setup,--gpusalso works on an installed model, setup asks which GPUs to use without any flag,--calibratemeasures the split, and the Monitor shows every card. Thank you @megakilo, @sal7g86 and @pdsmike. I tested it on Windows only (the Linux setup is the same code); if the Linux run doesn't work for you, please reopen with the output.sal7g86 commentedon Sep 29, 2026
@Niko1221, doesn't work :
on log:
I rerun without --setup
on log:
The Monitor's GPU readings on ROCm, and the disk rate without psutil
Monitor: every card of a layer split (issue Niko1221#112)
Monitor: every card of a layer split (issue Niko1221#112)
Monitor: every card of a layer split (issue Niko1221#112)
m-abs commentedon Oct 10, 2026
Hi @Niko1221
I think, I have this issue with a 5060 TI 16 VRAM + 3060 12 VRAM on v0.1.42
> ./update.sh Getting the newest Strata (git pull) ... Already up to date. Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU) [ok] numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow, psutil already installed [ok] qwen3.8-flash-next-q2_0: up to date [ok] swift-1.5-iq2_xs: up to date [ok] qwen3.8-flash-next-coder-iq1_m: up to date [ok] Strata is updated (engine 0.1.42). Start the model with ./setup.sh when you want it.