[bytethecookie]

What

Optional support for sharing the GPU with other programs: Strata can unload the model when it is idle and load it again on the next request. Everything is off by default — without the new options the server behaves exactly as before.

Option	Config key	What it does
--idle-unload SECONDS	idle_unload_s	unload after that long without requests; the next request loads it again
--min-free-vram-mib N	min_free_vram_mib	load an unloaded model only when that much VRAM is free (waits up to 15 s), else 503 "the GPU is in use by another program"
--before-load CMD	before_load (string or list)	a command run before the model is loaded again, e.g. to unload another server's model

Plus POST /unload (409 while a request is running) and POST /load; /health gains "loaded", /v1/models lists the model with status unloaded (like llama.cpp's router, building on #122's endpoints), /props sets is_sleeping, and /metrics reports the state unloaded.

Why

Strata fills all free VRAM and holds it until the window is closed. On a PC that also games, runs ComfyUI or another model server, that means stopping and starting Strata by hand. With these options it can stay up permanently next to them.

How

• Unloading ends the engine process (and the image encoder, when images are on), so VRAM and pinned RAM go straight back. Loading reuses the existing restart() path from Generation stops after few hundred tokens in v0.1.8 #27 — the "engine died, start it again" check in run() now goes through one ensure_loaded() that also handles a deliberate unload.
• The image encoder is started again before the engine, as at a normal start, so a GPU encoder takes its VRAM before the engine sizes its expert cache. Its cache of encoded images is kept.
• Requests load the model before the answer starts, so a busy GPU is a clean JSON 503 rather than an error mid-stream.
• Free VRAM is read through the existing _Nvml class in telemetry.py; if it can't be read, nothing is refused.
• Unload only happens between requests (it takes the request FIFO non-blocking and checks busy/queued).

Measured

RTX 5060 Ti 16 GB, i5-12400F, 48 GB DDR4, Linux (Docker), Q2_0 with --mmap-experts, files in the OS cache:

• POST /unload returns after ~0.33 s with the VRAM free (15.7 GB -> 0.76 GB used)
• a text request to an unloaded model answers after 4.6 s
• a request with a picture (encoder on the CPU) answers after 14.7 s
• with 8 GB of VRAM held by another process and --min-free-vram-mib 11000: 503 after the 15 s wait, model stays unloaded

I run it this way next to llama.cpp's router and ComfyUI: --idle-unload 120 --min-free-vram-mib 11000 --before-load "<script that unloads llama-server's models>".

Tests

python -m unittest serve.test_server: the 39 existing tests pass unchanged, plus 9 new ones in SharingTheGpu (unload then reload on the next request, /load, refused while busy, idle unload, the VRAM guard refusing and then loading, unreadable VRAM never refusing, before_load running first, the encoder unloading and starting first, off by default).

Notes for review

• POST /unload and /load follow the API-key check like the other POST routes but not the _own_page check that /settings has, since other programs on the machine are the intended callers. The worst a cross-site request could do is unload an idle model.
• Not tested on Windows or with the GPU image encoder or multi-GPU — I only have the setup above. terminate() on Windows is a hard kill, which should be fine for a process between requests, but it is untested.
• Docs: a short section in docs/DETAILS.md. Setup does not ask about any of this; it seemed better left as an advanced option.

🤖 Generated with Claude Code

[Niko1221]

Useful and well tested. Two asks: set last_request_at in load() so the idle thread can't unload between load() and the request, and have unload() send QUIT (as close() does) before terminate. Please also note in DETAILS that the VRAM check only looks at the first GPU.

[bytethecookie]

Thanks for the review — all three are in (a57f1f8), and the branch is rebased onto current main (0.1.29):

1. load() counts as activity: it sets last_request_at (under status_lock) before anything else, so the idle thread's unload(idle_for=...) sees the request coming. New test test_load_counts_as_activity.
2. unload() sends QUIT first, as close() does, then waits; it only falls back to terminate(), then kill(), if the engine doesn't end. New test EngineUnloadQuits runs a small fake engine process and checks it ended through QUIT rather than SIGTERM (skipped on Windows, since the fake engine is a shebang script).
3. DETAILS: the min_free_vram_mib row now says that with several GPUs only the first one (the engine's main card) is checked.

One more change came out of the rebase: your new LiveRate test builds a StrataEngine with a stub proc, and my /metrics "unloaded" state called alive() (so proc.poll()). /metrics now reads the unloaded flag instead of probing the process, so a crashed engine still shows as before.

Checked on the real engine (0.1.29, RTX 5060 Ti 16 GB, Q2_0 low-RAM mode): unloading through QUIT takes 0.5–0.8 s (vs ~0.3 s with terminate()), frees the VRAM fully (15.7 → 0.76 GB used), never needed the fallback, and the next request reloads normally. I updated the measured figure in DETAILS. python -m unittest serve.test_server: 53 tests, all pass.

[Niko1221]

Released in 0.1.30: with three small follow-ups (a request stamps last_request_at in load() so the idle thread cannot unload between load and run, unload() sends QUIT before terminating, and the VRAM guard's docs name the first GPU). Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!