serve: let the image encoder run on its own GPU (vision.cuda_device) #408

Request: let the image encoder run on its own GPU, chosen in the config, separate from the cards the engine uses.

Why: I have three cards: two RTX 5060 Ti 16 GB that hold the IQ3_S layer split ("gpu": [0, 1]) and a spare RTX 3060 12 GB. With --vision gpu, strata-vision starts with the engine's environment (child_env(cfg), so CUDA_VISIBLE_DEVICES=0,1) and lands on card 0. It takes room the expert cache could use (the "few % slower" in the docs) while the 3060 sits idle. I couldn't find a setting for this; the env is shared at server.py where Vision(cfg["vision"], ..., env=env) is built.

What I run locally (0.1.30): an optional "cuda_device" in the config's vision section. When it is set, the encoder gets a copy of the env with CUDA_DEVICE_ORDER=PCI_BUS_ID and CUDA_VISIBLE_DEVICES=<n> (numbered like nvidia-smi, as in #51). When it is absent, nothing changes.

"vision": { "exe": "...\strata-vision.exe", "mmproj": "...", "model": "...",
            "gpu": true, "max_tokens": 1024, "cuda_device": 2 }

diff --git a/serve/server.py b/serve/server.py
index e03379e..e008b16 100644
--- a/serve/server.py
+++ b/serve/server.py
@@ -565,6 +565,19 @@ def child_env(cfg: dict) -> dict:
     return env
 
 
+def vision_env(cfg: dict, env: dict) -> dict:
+    """The image encoder's environment: the engine's, unless the config's vision section names its own
+    "cuda_device" (numbered like nvidia-smi) - then the encoder runs on that card alone, so a spare GPU can hold it
+    while the engine keeps every byte of its own cards."""
+    dev = (cfg.get("vision") or {}).get("cuda_device")
+    if dev is None:
+        return env
+    env = dict(env)
+    env["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+    env["CUDA_VISIBLE_DEVICES"] = str(dev)
+    return env
+
+
 class ByteTokenizer:
     """Tiny stand-in tokenizer for tests without the pack: one id per UTF-8 byte, specials as ids >= 256."""
     SPECIALS = ["<|im_start|>", "<|im_end|>", "<|endoftext|>", "<|vision_start|>", "<|image_pad|>", "<|vision_end|>"]
@@ -1917,7 +1930,7 @@ def main() -> int:
         if cfg.get("vision"):
             print("loading the vision encoder ...", flush=True)
             vision = Vision(cfg["vision"], log=open(cfg["log"], "a", encoding="utf-8") if cfg.get("log") else None,
-                            env=env)
+                            env=vision_env(cfg, env))
         print("loading the model (the first start takes a minute or two) ...", flush=True)
         if len(gpu_list(cfg)) > 1:
             print(f"[strata] layer split across GPUs {gpu_list(cfg)} ({cfg.get('layer_split') or 'auto'})", flush=True)

Plus a unit test next to GpuChoice.test_env: the encoder env gets "2", the engine env stays "0,1", and without cuda_device the engine's env is returned unchanged.

Measured (Windows 10, driver CUDA 13.0, engine 0.1.30, prebuilt strata-vision.exe with sm_86):

• nvidia-smi shows strata-vision.exe on the 3060 only. The 3060 went from about 0.9 to 2.3 GB, and the two 5060 Tis stayed at about 14.9 / 15.0 GB.
• /v1/status reports "vision": {"enabled": true, "available": true}.
• A 1328×1328 PNG through /v1/chat/completions (image_url data URL) was described correctly, with 1,073 prompt tokens and 11.5 s for the full request including 151 output tokens.

Open points for you:

• setup.py could offer this when it finds a card outside the engine's set. I only edited the config by hand. I appended --vision to args but left out --vram-reserve-mib, since 700 is already the engine default.
• The encoder's card has to be in vision_archs (86/89/120 in the prebuilt). This patch does not check that.

Happy to open a PR if this shape works for you.