Repository navigation
Multi-GPU: a --layer-split stage holds only its own layers' weights, and every card's leftover VRAM spills into a helper expert tier (4-way IQ3_S: 9,269 -> 14,172 expert slots, prefill chunk 2,048 -> 4,096) - #390
Closed
gopinath87607 wants to merge 17 commits into
Closed
gopinath87607 wants to merge 17 commits into
gopinath87607 wants to merge 17 commits into
Conversation
A layer split gives every stage the whole model's dense weights, so a card running 9 of 48 layers still holds all 48. On the 4-GPU IQ3_S rig that is 3485 MiB per card holding nothing but cold layers - VRAM the expert cache should have. The session carve already solved this for session state: `session_bytes` is pure arithmetic over a layer range, so the split search prices a candidate placement before committing VRAM to it. This gives the weight arena the same treatment, on the compaction path the loader already had. With a `skip` set the loader re-places kept rows from offset 0; the range filter is one more predicate in that same pass, which is the point of putting it there rather than in a second function - one rule decides both the arena's size (`pool_bytes`) and its layout, so a slice cannot be priced by one and written by another. `tensor_layer` is shared with NativeDense (which is next) so the two can never disagree about which layer a tensor belongs to; a disagreement would not be a crash, it would be a stage quietly running another stage's layer. Model-global tensors have no ordinal, so a range never drops the head or the embedding - they are needed by stages the range did not give them to. `keep` is for the one tensor a stage needs although it does not run its layer: the split's router lookahead reads every layer's router from one stage's table. `layer_bytes` is the price model, and it is pure. `hi < 0` disables the carve and is byte-identical to no filter: the re-placement loop is not entered at all, so every tensor keeps the offset the pack gave it. No call site passes a range yet, so nothing changes for a running engine. weights_carve_test checks the two halves against each other over every range on a synthetic index with deliberately uneven, unaligned tensors, and requires the full range to reproduce a real pack's declared pool. Co-Authored-By: Claude Code <noreply@anthropic.com>
The other 2018 MiB of the 3485 MiB a split stage wastes. NativeDense uploads each eligible GGUF matrix into its own allocation and hangs it off the canonical WeightRef as `native_data`; `eligible()` matches `blk.<n>.<suffix>`, so the set is per-layer - a stage running 9 of 48 was uploading 39 layers' worth of projections it can never reach. `load` and `served_names` take the same `[layer_lo, layer_hi)` as `WeightTable::load`, and they MUST be given the same range. `served_names` populates the canonical loader's `skip` set, so a range applied in one place and not the other leaves a tensor with neither canonical bytes nor a native override - two null pointers where a weight belongs, which is not an error return but a kernel reading address 0. `served_layer_bytes` is the price model, the counterpart of `WeightTable::layer_bytes`: per-layer native byte totals from GGUF headers alone, the same eligibility `load` applies, so the split search can price a candidate range before committing VRAM to it. `hi < 0` is the whole model and is the previous behaviour exactly. Co-Authored-By: Claude Code <noreply@anthropic.com>
Every `--layer-split` stage used to load the WHOLE model's dense weights: the arena was sized by the whole-model `pool_bytes` and the native projections by an unrestricted `NativeDense::load`. A card running 9 of 48 layers held 39 layers' worth of matrices it can never reach - on the 4-GPU IQ3_S rig 3485.66 MiB per card (1466.78 pack arena + 2018.88 native), 10.2 GiB across the four. That VRAM is expert cache: decode here is bounded by the LOCAL cache hit rate, and CUDA3's 595 slots are what clamp the prefill chunk. The weights now follow the split search, exactly as the sessions already do. `st.lb`/`st.le` are set, and only then does each stage size, allocate and load its own range - the same "priced before it is allocated" rule `session_bytes` established, because `pool_bytes` and `layer_bytes` are pure arithmetic over a range. The stage loop keeps everything that touches no weights (preflight, streams, the control-vector tables, the vision position table, the PCIe probe) where it was. `predict`'s `capr` loop subtracts the range's weights beside its session, so what the search leaves for the expert cache is what the cache gets. Stage 0 keeps the weight term at zero for now: CUDA0's own load still happens before the search and its carve lands separately, so this commit changes nothing about CUDA0's accounting. STRATA_WEIGHT_SLICE=0 disables the carve everywhere, so one binary runs both arms of the A/B. Carved sizes on IQ3_S at split 10,19,34, against the predicted table: CUDA1 [10,19) 272.83 + 364.85 = 637.68 MiB (predicted 637.7); CUDA2 [19,34) 448.02 + 656.07 = 1104.09 (1104.1); CUDA3 [34,48) 419.66 + 588.55 = 1008.21 (1008.2). Co-Authored-By: Claude Code <noreply@anthropic.com>
The weight carve FREES VRAM, so an A/B against STRATA_WEIGHT_SLICE=0 would change the per-stage slot counts as well, and then a token difference would say nothing about the weights. Per-stage caches are sized from free VRAM with no way to pin them (unlike CUDA0's, which --expert-cache pins), so this is the one hook the A/B needs: stop after S slots and both arms hold the same pairs. Test-only, in the same spirit as STRATA_TEST_CACHE_FAIL. Co-Authored-By: Claude Code <noreply@anthropic.com>
The last 2.56 GiB of the hole, and the most valuable card: CUDA0's local
expert cache is what bounds decode, so its weights are the slice that
pays twice.
CUDA0's load moves down beside the stages' - `pool_bytes`, the arena,
`wt.load` and `native_dense.load` all now happen once `split_at[0]` is
known, and it runs [0, split_at[0]). The PLE's six `blk.1.*` weights
are wired there rather than in the PLE block, because that is where `wt`
is first valid; every one of them is inside CUDA0's range whatever the
split is, so nothing about the PLE changes.
THREE THINGS THIS HAD TO GET RIGHT, each one silent if it did not:
* THE ROUTERS STAY. CS-T's file-tier lookahead memcpy's every layer's
`ffn_gate_inp` out of CUDA0's table to warm the next layer's pages.
A dropped row keeps its metadata but has `data == nullptr`, so the
copy fails, `ok` goes false and the lookahead turns itself off
without saying so. All 48 are kept - 120 MiB against a prefetch.
* `--split-skip-if-fits` MEASURES CUDA0 BEFORE ITS WEIGHTS EXIST, so
the whole model's dense is added to what it holds back. Without
that term the carve makes the split look skippable by exactly the
VRAM it is about to spend.
* THE PRICE MODEL NOW COVERS THE CARVE-OFF ARM. `stage_room` runs
before the weights load in BOTH arms, so `weight_bytes` returns the
whole model when `STRATA_WEIGHT_SLICE=0` rather than zero - it was
zero, which would have let the search place the split against VRAM
that is not there. `--layer-split auto` with the kill switch set is
the only path that reached it.
`--split-device 0` needs no separate guard: `ver_same` replays a later
stage against THIS table, but that mode already clears `multi_gpu`, so
`hi0 < 0` and CUDA0 keeps the whole model.
A/B at split 10,19,34 with the stage caches pinned: four runs, one hash -
and the same hash as before this commit. CUDA0 3485.66 -> 868.30 MiB
(458.89 arena incl. the routers + 409.41 native, 62 of 300 matrices).
Co-Authored-By: Claude Code <noreply@anthropic.com>
IQ4_NL rows are 90 bytes (5 blocks of 18); Q8_0 rows are 170 (5 blocks of 34, `d` once per 32 values, no min term). The published IQ3_S/IQ2_S checkpoints carry the PLE table as IQ4_NL and the Q8_0 checkpoint carries it as Q8_0, so the row cache and the batch slab - both page-granular - have to take the stride from the tensor's own type rather than from a constant. `PleReader::open` takes the row size (default 90: every existing caller compiles and reads exactly as before), `PleTable::open` reads it off the tensor and logs it, and `ple_dequant_row` dispatches to the artifact reader's own IQ4_NL/Q8_0 block decoders, so the PLE row and `artifact/dequant.hpp` cannot drift apart on the block layout. A wrong stride still opens and decodes plausible garbage, so the row size is printed once at open, and the selftest now runs the whole reader - page arithmetic, straddling rows, dedup, bulk tickets, two tickets in flight, the injected-delay arm - twice, once per row size, plus a direct check of the Q8_0 block layout against known values. Co-Authored-By: Claude Code <noreply@anthropic.com>
…sion footprint The split is invisible in the UI: the metric cards show pooled totals, so a four-GPU run reads the same as one card holding 27 GiB. Each GPU now gets its own card, built from what the engine alone knows - its layer range, its KV and prompt buffers, its expert cache - paired with that card's VRAM from NVML. The engine reports its own allocation as seven parallel CSV fields on the INFO line (gpu_dev, gpu_layers, gpu_kv_mib, gpu_buf_mib, gpu_experts, gpu_expert_mib, gpu_draft_mib). They are single comma-separated tokens because the server's INFO parser splits on whitespace. A remote-expert GPU runs no stage, so its layer span is `-`; the MTP head is reported on the one card that holds it and nowhere else, since no totals line accounts for it. No VRAM figure comes from the engine: it knows only its own allocations, and NVML already reports used/free/total. For that last part NVML's `mem_free` was being read and thrown away, so telemetry now keeps it, and each card's name is asked for once rather than once a second. The image encoder is a separate process that reports no memory of its own - not to stdout, not to the log - so its footprint is measured as what that card's free VRAM lost while it started. `strata-vision` runs a 2048px warm-up before printing READY, so every buffer it will ever need is allocated by the time the reading is taken; a delta outside 64..8192 MiB is noise rather than this process and is discarded, keeping the last good number. Also folds in three unrelated fixes found on the way: an image source that 404s, refuses the connection or names an unknown host raised OSError, which the dispatcher does not map to a status, so the client's connection closed with no reply at all instead of a 400 - those now raise ValueError like every other bad image source; `free_vram_mib` is one helper taking an index rather than two copies of the same NVML call; and the vision encoder is pinned to the config's first GPU, which is the card the Monitor has to show it on. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
this pr is still on working will share all the test output if anythink i need to change it let me know it. |
…he cannot use A four-GPU run picked its split from a formula that reads only `layer_ms`. `capr` - the free VRAM that decides how many experts a stage can actually hold - never entered it, so the search could hand the most layers to the card with the least room and never notice. On this 3060/3060/5060/5060 box CUDA1 ended up with 1,841 MiB no cache could use while CUDA2 ran 15 layers it could hold a third of, and ~20 GiB sat stranded across the two 3060s. All C(L-3,3) placements are now scored - 15,180 at 48 layers. What makes that affordable: a stage's fill depends only on its own layer range and its own free VRAM (each ranked pair goes to the one stage that owns its layer, and each stage stops at its first pair that does not fit), so a placement's held mass is a sum over four independently memoised range walks - 1,121 walks instead of 15,180, measured at 14 ms against 740 ms for the same answer. Scoring the placements is not enough on its own, because the cost model's mass curve was wrong here. `mass[r]` was (r+1)^-1.2, fitted to the 5080+3090 sweep; it is still right on that rig (99.35% predicted at 20,000 held, 99.4% measured) and badly wrong on this one - it claims the top 5,805 pairs hold 94.9% of the routed mass where the rig measures 69.2% over four boots and 14.9M decode lookups. No exponent reconciles the two, because the curve is a property of the model's routing, and the profile file carries only a ranking, with no counts, so it cannot be read off the data. What both rigs do agree on is that the hit rate follows the held COUNT almost alone: 14,541 held measured 92.2%, 14,384 held measured 89.8%. The mass is therefore the increment of H(f) = 1 - (1 - f)^b over the held fraction, whose prefixes sum to exactly H(N/total). b = 3 is chosen to leave the validated regime where it was: at f = 0.8138 it gives 99.35%, the same as the old fit, so the 2-GPU placements are untouched, while at this rig's f = 0.59 it gives 93.2% against 92.2% measured. STRATA_SPLIT_COVER_B overrides b for a model whose routing is sharper. The second half is the VRAM no placement can reach. A stage's cache holds only its own layers' pairs (`GpuStage::profile` is filled from that stage's range alone, so `slot_of` never returns a foreign layer's slot), which means a card given few layers has room nothing can fill. With `--layer-split` every card the run already uses - CUDA0 included - now spills its leftover VRAM into a helper tier holding the globally hottest pairs no stage cache holds, and every stage's misses can hit it. The tiers are sized from `cudaMemGetInfo` after the weights, sessions, drafter and caches are all allocated, so they take what is really left: the session reserve, this card's verify window, the drafter's bind buffers, the prompt buffers when they are not borrowed, and `open`'s own 512 MiB come off first, and under 256 MiB no tier opens. They are disjoint by construction - one shared `claimed` bitmap plus CUDA0's cache as `primary` - so their order only decides whose PCIe round trip serves the hot pairs, and they are opened fastest-SM-clock first. The whole thing is automatic and has a kill switch: `--no-remote-auto`, or `STRATA_REMOTE_AUTO=0`. That required letting a helper tier live on a card that already runs a stage. `--expert-cache-deviceN` and the automatic tiers are never mixed: naming a device is a statement about where helper VRAM should go. A device-0 tier needs its context flags set before any context exists, since `cudaInitDevice` applies once per primary context and then fails with cudaErrorSetOnActiveProcess, so `preflight(0)` moved into the early window next to the existing CUDA1 one - before the PCIe probe and the weight arena - and device 0 takes `cudaDeviceMapHost` without `ScheduleSpin`, which belongs on a card sitting on the pool's critical path, not on the card the whole run is built on. If the mapped host allocation is refused (WDDM's pinned budget is largely spent by then) the staging falls back to plain portable pinned memory and copies, rather than losing the tier; how the activations actually cross is MEASURED from `cudaHostGetDevicePointer` and printed, never assumed. The tier count widens 3 -> 4 (`kRemoteMax`), and a spill tier that cannot open is dropped with a warning where an explicit one still fails the run. The Monitor merges a tier on a card that also runs a stage into that card's own cell instead of printing the device twice. `select_remote_pairs` is extracted from `open` so the sizing rule can be tested without a GPU, and takes a byte budget alongside the slot count. A budget SKIPS a pair that does not fit rather than stopping, because blob sizes differ per layer and one hot oversized blob must not hide the colder pairs behind it that still fit. tests/core/remote_budget_test.cpp checks 400 random walks against a reference implementation written the obvious way, plus the named properties, and skips the unequal-blob case where no native layout has loaded. `tools/make_profile.py` gains a note: `take()` skips a pair it has already ranked, and the shipped base ranks all 24,576, so with the default `--base` no trace can move a pair - the run prints "0 from the traces" and the output is the base again. The ordering in use is the shipped one, not this model's, which is why the cost model needed the measured curve above rather than another fit. Verified on the 4-way rig: the split moved from K=2,3,27 to K=14,29,39, held pairs from 5,805 to 15,190, and the caches now fill on every card (CUDA1 4,634 of its 7,680 pairs, CUDA2 2,877 of 5,120, CUDA3 2,008 of 4,608) instead of CUDA0 and CUDA1 being capped at their own 1,024 and 512 with ~20 GiB idle. The spill tiers found nothing to do and correctly declined (456 MiB free against a ~1.3 GiB reserve) - with a placement that no longer strands VRAM there is nothing left to lend. The first request measured a 90.6% decode hit rate and 50.2 tok/s against 38.1 tok/s for the imbalanced split with no tiers and 41.3 tok/s with them, on the same prompt shape; a run that short is not the last word on throughput. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
this pr is finished if anyone interested feel free to check it decode jumb 40 to 54 tps while prompt stay as it is |
…rve) 128 commits of 0.1.32-0.1.34 (AMD/HIP, MCP, setup, #340's split ring reserve, #369's per-layer admission, the pool affinity flag) onto the weight-carve work. Conflicts and how they were settled: * include/strata/kernels/ngram.hpp, src/kernels/ngram.cpp, src/ngram/ple_reader.{hpp,cpp}: upstream's row-size design is the base (the stride is the tensor's own GGML type, and PleReader::open takes it). Q8_0 is re-added on top as a fourth format - type 8, 170 B rows (5 blocks of 34, one fp16 `d` per 32 values) - because the Q8_0 checkpoint carries the PLE table as Q8_0 and upstream's whitelist refuses it. ple_row_bytes()/ple_dequant_row() are the dispatch; the Direct reader takes the variable row, so unlike Q5_0 there is no mmap-only guard. * include/strata/core/native_dense.hpp, src/core/native_dense.cpp: both sides kept - our served_layer_bytes() (the split search's price model) beside upstream's keep_unquantized_ple_key(). * serve/server.py: Service.free_vram_mib() and the Vision construction both delegate to the one shared serve.telemetry.free_vram_mib(index, amd) - AMD-aware, and the same helper the encoder's own footprint reading uses. The Vision call takes upstream's relative-path vcfg + vision_env() and keeps our device=, derived the way vision_env picks the card. The now-unused `import ctypes` is dropped. * src/program/generate.cpp: our structure (the carve, the ns == 4 placement enumeration, the coverage curve, the spill tiers) is kept; upstream's deltas in the same region are ported onto it - the SSD keep-alive lines, Verifier::set_commit_async, #340's split_ring_mib + Prefill::set_ring_override(96), and stage_room()'s `search` parameter with place_with_reserve. * src/ngram/ple_reader_test.cpp: auto-merged, then row_bytes threaded through the keep-alive blocks that came from upstream (check_rows/open calls). Built clean in build-v0131. 56 of 58 ctest targets pass serially; the two failures are environmental and pre-existing (ple_parity wants the Q2_0 shard 2, absent; expert_multi_test wants AVX512-VNNI/VBMI, which this CPU lacks). serve's 120 python tests pass (2 skipped). Under -j8 the CUDA parity tests fail on VRAM because the 4-way server holds the cards - serial run is the honest one. Not pushed: the rig builds and tests this first. Co-Authored-By: Claude Code <noreply@anthropic.com>
gopinath87607
changed the title
… empty result
The Anthropic dialect flattened a tool's result through _text_of(), which keeps the text
parts only, so an image a tool returned - Claude Code's Read on a picture, a screenshot
tool, a browser tool - was dropped with no error, no log line and no refusal. The model
received an empty tool result and answered anyway: a confident description of a picture it
had never been given. This is the shape Claude Code uses for every image its Read tool
returns, which is why images could work in one client and come back wrong in another.
Measured on the 4-way IQ3_S rig, one image, three request shapes, before the fix:
Anthropic, image inside tool_result 380 prompt tokens "B - blue shape on a white
background" WRONG
Anthropic, plain user image block 890 prompt tokens correct
OpenAI, image in the "tool" message 956 prompt tokens correct
The image contributes ~576 tokens, so 380 is the proof that it never arrived.
_parts_of() keeps it, and the pack's chat template already renders one image item per image
for a message of any role (chat_template.jinja:8), so the rows land inside <tool_response>
where the picture belongs. A message holding an image beside a tool_result was dropped the
same way and now keeps it too. A text-only result is unchanged: _parts_of() returns a
plain string when there is no image.
After: 959 prompt tokens and "The letter Q inside a green circle on a purple background."
tests: serve/test_server.py's ToolResultImages - the image survives and images_of() finds
it, a text-only result is unchanged, an image beside a tool result survives, and the
rendered prompt gets the encoder's rows. serve's 157 tests pass (7 skipped).
Co-Authored-By: Claude Code <noreply@anthropic.com>
The auto chunk scan prices a chunk with `bytes_needed` and lends it the slots those bytes want, so an over-count there is an under-sized loan - and this one under-sized every loan by a step. The count walked `carve`'s sequence from memory and got two of its buffers wrong: it took `xn` unconditionally, though `carve` takes it only under STRATA_GR_UNFUSED, and it never took `grs`. Net T*(D-HC)*4 bytes too many: 42 MB at a 1024-token chunk, 252 MB at 6144 (48 Q8_0 slots). The direction is safe - over-estimating, and `take` still bounds-checks - but the prompt path was told it had less room than it did. Co-Authored-By: Claude Code <noreply@anthropic.com>
…s own slot count The ring was 384 slots, a number measured on Q2_0 where a slot is one expert blob of 1,382,400 B - 506 MiB. A slot is a whole blob, so on a pack with bigger blobs the same 384 slots are a different amount of memory: 1,893 MiB on Q8_0, more than half of an 8 GB card's expert cache, and what kept that card at a 1024-token chunk when its own buffers would have fitted 6144. Measured on the 4-way rig, one env var and nothing else: chunk 1024 -> 6144, prefill 87 -> 402 tok/s. So RING_BYTES is the constant and slots are derived from the pack's max blob - 384 on Q2_0 (pinned) or 96 (not), exactly as before, and the pack it was tuned on does not move. This is also what lets the scan treat "the ring is full" as a plain byte comparison. Also exports the three accessors the scan needs: bytes_needed_no_ring (the chunk's own buffers, without the ring), ring_max_slots (the cap in slots) and ring_slots_for (what the ring resolves to for a chunk, after the override, the env var and the pinned-share rule - what `init` lays out and what the INFO line reports). Co-Authored-By: Claude Code <noreply@anthropic.com>
…the better buy `--prefill auto` walked a fixed list of chunk sizes - 32768, 16384, 8192, 6144, ... - and took the first that fitted, with the ring a fixed 384 slots taken off the top before the chunk was ever considered. Both halves of that were wrong on a rig like this one. The list is coarse exactly where it matters. `bytes_needed` is a sum of (T x positive constant) terms plus a max of such sums, so it rises monotonically with T, and so does every test the scan applies - which makes the largest chunk that fits a bisection on the 256-token grid the prompt path already works on. Seven probes against the list's ten, at one `bytes_needed` each. It matters because this rig affords ~8,700 tokens and was handed 8192; 8704 is not on the list. The order was wrong because chunk and ring spend the same borrowed VRAM, and a ring slot is worth far more than a chunk token. Measured here (2x RTX 3060 + 2x RTX 5060, CUDA3 lending at its 90% cap, 120K prompt): 8960 tokens with the 17-slot ring that leaves reads at 963 tok/s, 8192/130 at 1,008, 7168 with the ring full at its 199-slot byte budget at 1,000, 5632/199 at 915. A ring slot is worth ~0.53 tok/s and a chunk token ~0.05, so the 69 slots between a full ring and 8192's 130 are worth more than the 512 chunk tokens they cost - and once the ring IS full, a smaller chunk buys nothing. So: the largest chunk that still leaves the ring full, and only a rig where no chunk can afford one falls back to the old rule with a kRingMin floor. The room is counted in BYTES, not slots. A ring slot is max_blob, but a cache slot holds its own layer's blob - 2.15 MiB against a 2.54 MiB max_blob here. In slots the ring looked 18% cheaper than it is, a chunk whose ring fitted only after that discount was refused outright, and the scan stopped a step short. `bytes_from_slots` is the inverse of the existing `slots_from_bytes`, and `cache_slots_for` factors the latter out for the serve path's per-participant version. Both inverses clamp at k <= 0: the lend budget is `min(slots - 128, pct * slots / 100)` and goes negative on a cache under 128 slots, where `slots - k` would index slot_offsets() past its end. Verified on the rig: the scan picks 7680/199 against the old rule's 8960/17, 1018.8 and 1021.1 tok/s on two cold 120K prompts against 963.1 - +5.9%. Q8_0 picks 5632/101, its 506 MiB budget over a 5,222,400 B blob. Co-Authored-By: Claude Code <noreply@anthropic.com>
…e with it The auto scan now settles on a chunk and a ring for the run, and the engine reports both on its INFO line (`prefill_chunk`, `prefill_ring`). The Monitor's GPU panel gains a "Prompt chunk" row reading them, so the pair is visible from the web app instead of only in the boot log. `prefill_ring` is the RESOLVED ring, not the room the scan offered: ring_slots() clamps to [16, RING_MAX], a chunk under stream_all_min() gets STAGE, and STRATA_PREFILL_RING overrides both. What is shown is what `init` lays out. Co-Authored-By: Claude Code <noreply@anthropic.com>
One textual conflict, in src/program/generate.cpp, and only one: upstream's #448 ("name the small split card that caps the prompt chunk") refactored the serve path's chunk picker into fits_one/fits(c, cap, only)/pick(only) plus a warning that reports what CUDA0 alone would have chosen - the exact block this branch had replaced with the ring-first scan. Both sides rewrote the same lines. Resolved by keeping upstream's structure and putting this branch's rule inside it: `pick(only)` now runs the ring-first two-pass scan for the auto path and keeps upstream's halving loop for an explicit --prefill verbatim. `ring_room` gained the same `only` parameter as `fits`, so `pick(&pf_parts[0])` still means "CUDA0 by itself" and the #448 diagnostic is unchanged. `ring` is resolved from the chosen chunk before that diagnostic runs, because the diagnostic's own probes leave set_ring_override dirty and `if (chunk > 0)` puts the chosen ring back. Verified on the 4-way IQ3_S rig: the merged binary prints prompt chunk auto: 7680 tokens, a 199-slot ring - the same pair the branch chose before the merge, and the measured peak - and the #448 warning correctly names CUDA2 and CUDA3 as the cards that cap it: WARNING: prompt chunk 7680 tokens, not 16640: CUDA2 (NVIDIA GeForce RTX 5060) has 2812 expert-cache slots ... (#448) Both features are therefore live together. /metrics reports prefill_chunk=7680 prefill_ring=199, so the Monitor row still reads the pair. serve.test_server: 105 tests, OK (upstream added three). Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
@Niko1221 if you have time kindly check this pr prefill jumb to 1k token, decode also jumbed ,slots and chunk number auto fixed ,anthorpic image api fix ,monitor ui change, and reduced vram mem. |
The JSON behind the 4-GPU Q8_0 server: --pack/--native/--ple-gguf, --prefill auto:32768, --expert-cache auto, --layer-split auto over gpu [0,1,2,3], --max-context 262144, --kv int8 --kv-resident 32768, --spec 4, and fit_max_tokens, which clamps an oversized max_tokens to the room the prompt leaves rather than returning a 400 (serve/server.py:1210-1219). /strata-*.json is gitignored at the repo root, so this copy lives under configs/. The paths in it are this machine's (/home/gopi/Strata, /home/gopi/Strata-data, /mnt/nvme) and are not portable as they stand. Co-Authored-By: Claude Code <noreply@anthropic.com>
The same shape as configs/strata-q8_0-4way.json, differing only in the pack, the native/PLE shard names and the model name: --prefill auto:32768, --expert-cache auto, --layer-split auto over gpu [0,1,2,3], --max-context 262144, --kv int8 --kv-resident 32768, --spec 4, and fit_max_tokens. This is the config the 4-way rig's prefill numbers were measured on: a 7680-token chunk with a 199-slot ring, ~1020 tok/s on a 120K prompt. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
pushed the config file too |
Owner
|
Thanks! 0.1.38 adds a second-GPU helper tier (#531, |
Contributor
Author
just woke up doing it now. |
Closed
Contributor
Author
Thanks — that's a fair call on the second tier, and I'll split as you describe. (1) and (3) are already up as their own PRs — #580, #581 and #582 below. On (2) I'd rather check What the tier here actually is: with Before the carve that leftover was far larger — 3,144 MiB on CUDA0 and 1,739 MiB on CUDA1, while
So the question is which of these you'd prefer: (a) Extend (b) Treat them as disjoint: the peer is a second-GPU tierll tier is a Given the numbers above I'd now lean (b): with the carve merged, what the tier buys is ~1.5 GiB Either way, here is the split, all five against
#584 is one commit on top of #580 — it changes the split that gets chosen, so it could not ride with |
Contributor
Author
|
Split as you asked. #390 is superseded by these, all rebased onto
Two of those you didn't name, so the reasoning: the placement search was welded into the same commit On (1), the two things you asked for. The split is identical to before at a fixed cache — same The placement search is measured on top of that, and I've put the post-carve number in its body Nothing here rewrites the branch: this PR keeps its commits and its configs, so the links already in Two changes from what was in the file: PR numbers are now in the table (the branch column stayed, so he can still check out each branch), and the placement-search paragraph no longer claims it "targets the carve's branch" — it targets main, stacked on #580's commits, for the fork-base reason. |
Contributor
Author
|
pushed prefile to 2k+ please check this pr #761 |
Contributor
Author
|
Closing this. Every piece of it has either reached Where each piece went
The two that are worth your time. #880 is the one that pays on a normal config: the trim was refused under Worth saying plainly: not one of the PRs split out of this was merged. What reached v0.1.39 came in as your own commits.
Thanks for the review and for the split - the pieces are better apart than they were together. |
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Three things a
--layer-splitrun needed, plus the Monitor work that made the first one visible.The headline is the carve: every stage used to load the whole model's dense weights. Then the
VRAM those weights were wasting is given back — first to each stage's own expert cache, and then,
for whatever is still left over, to a helper tier that any stage's misses can hit.
1. A stage holds only its own layers' weights
Each
--layer-splitstage loaded the whole model's dense weights: on the 4-GPU IQ3_S rig,3,485.66 MiB per card — 1,466.78 pack arena + 2,018.88 native dense — about 10.2 GiB across four
cards, holding layers that stage will never run. A stage running 9 of 48 layers still held all 48.
Weights now get the treatment the session already got.
session_bytes(g, cells, k, lo, hi)is purearithmetic, so the split search prices a candidate layer range before allocating it; weights were
the one per-stage resource that never got that. The carve extends the same rule: a layer-range
predicate on the loader's existing skip/compact path, and the per-stage load moved down to after
st.lb/leare set. One rule produces both the priced size and the loaded layout, so they cannotdisagree.
Measured, 4-way IQ3_S (split 10,19,34 auto):
Per-card reclaim: CUDA0 2.56 GiB, CUDA1 2.78, CUDA2 2.33, CUDA3 2.42.
The same carve applies to the native GDN/QSA/shared-expert projections
(
NativeDense::load's range), which is 2,018.88 of those 3,485.66 MiB.Three things that fail silently
[lb, le), not[0, le).[0, le)leaves twothirds of the hole unclaimed while every printed number looks plausible — this got as far as a
commit before the per-range byte comparison caught it.
ffn_gate_inpout of CUDA0'stable to warm the next layer's pages. A dropped row is still in the metadata but has
data == nullptr, so the copy fails,okgoes false, and the lookahead turns itself offwithout saying so. All 48 are kept — 120 MiB is cheaper than losing the prefetch.
--split-skip-if-fitsmeasures CUDA0 before its weights exist, so the full dense cost isadded to what it holds back. Without that term the carve makes the check spend exactly the VRAM
it is about to free.
STRATA_WEIGHT_SLICE=0is the kill switch, so one binary runs both A/B arms.STRATA_STAGE_CACHE_SLOTS=Spins stage caches (test-only), because they are auto-sized from freeVRAM and freeing 2.5 GiB per stage would change the slot counts and mask the comparison.
2. Every card's leftover VRAM spills into a helper expert tier
The carve exposed a second problem it does not fix on its own: a stage's cache only ever holds its
own layers' pairs, so once a stage holds its whole range the cache cannot grow and the cost model
scores its spare VRAM at exactly zero. On the 4-way rig that stranded real memory — CUDA0 at
3,144 MiB and CUDA1 at 1,739 MiB while both 5060s were packed at 7,225 and 7,311.
With
--layer-split, every card the run already uses — CUDA0 included — now hands its leftover toa helper tier holding the globally hottest pairs no stage cache can hold. Stage misses hit it.
--no-remote-auto/STRATA_REMOTE_AUTO=0is the kill switch, soit is revertable and A/B-able from the config JSON without a rebuild.
--expert-cache-deviceNpath keeps its hard failure.Prefill, whose GPU MoE path measured 2.2× slower and corrupting.stage-relative
d_restable is untouched.untouched — it still picks what it picked.
The tier count goes 3 → 4 (
kRemoteMax) to cover four cards, and a helper may now live on device 0(
device < 1→device < 0), which is whypreflight(0)runs in the early window before anythingelse creates that context.
Selection is by byte budget, not slot count: a slot count is a poor proxy for a native pack
where
blob_bytes(l)varies per layer. The selection loop is extracted into a CUDA-free functionand unit-tested (
tests/core/remote_budget_test.cpp, run underCUDA_VISIBLE_DEVICES=-1) for thebudget bound,
claimed/primary-cache disjointness, rank order, and bit-for-bit agreement with theold
slotsmode.Not yet measured on the rig. The mechanism, the budget arithmetic and the disjointness are
covered by the unit test; the decode and hit-rate effect needs the 4-way run, which the cards were
busy for. Worth watching: the per-tier log line, and whether CUDA0's and CUDA1's stranded VRAM
shows up as tiers.
3. Score every four-way placement
The search tried every placement up to 3 GPUs and, beyond that, shared layers in proportion to
speed. Four cards got a heuristic instead of a search. It now enumerates all C(L−3, 3) = 15,180
placements and scores each. Memoised per-range fills (1,121 walks) bring it from 740 ms to 14 ms.
The mass model moved from
mass[r] = (r+1)^-1.2to the increment ofH(f) = 1 − (1 − f)^boverthe held fraction,
b = 3(STRATA_SPLIT_COVER_B). At f = 0.8138 (20,000 held) it gives 99.35%,identical to the old fit — so the validated 2-GPU regime is unchanged.
4. The Monitor's per-GPU cards
The split was invisible: one card holding 27 GiB read the same as four. Each GPU now gets its own
card, built from what only the engine knows (layer range, KV and prompt buffers, expert cache)
paired with that card's VRAM from NVML. The engine reports its allocation as seven parallel CSV
fields on the INFO line — single comma-separated tokens because the server splits on whitespace.
NVML's
mem_freewas being read and thrown away; each card's name is asked for once rather thanonce a second. A device that both runs a stage and hosts a helper merges into one card rather than
appearing twice.
The image encoder is a separate process reporting no memory of its own, so its footprint is
measured as the card's free VRAM lost while it started —
strata-visionruns a 2048px warm-upbefore printing READY, and a delta outside 64–8192 MiB is discarded as noise, keeping the last good
reading. The encoder is pinned to the card the config's
vision.cuda_devicenames (else theengine's first), which is the card the Monitor must show it on.
5. The PLE table's Q8_0 row
mainnow takes the PLE table's row size from the tensor's own GGML type (IQ4_NL 90 B, Q5_0110 B, FP8 E4M3 160 B). This branch adds the fourth format the published Q8_0 checkpoint carries:
Q8_0, 170 B — 5 blocks of 34, one fp16
dper 32 values, no min term. Same geometry, so therow cache, the batch slab and the 16-rows-make-2560 flatten are untouched; only the stride and the
block decoder differ.
ple_row_bytes()/ple_dequant_row()dispatch to the decoders inartifact/dequant.hpp, so the PLE row and the artifact reader cannot drift apart on the blocklayout. A row decoded at the wrong stride opens and decodes plausible garbage, so the row size and
format are logged once at open.
Unlike Q5_0, Q8_0 is not restricted to
--ple-io mmap: the Direct reader takes the row size.6. Three fixes found on the way
An image source that 404s, refuses the connection or names an unknown host raised
OSError, whichthe dispatcher does not map to a status — the client's connection closed with no reply at all
(
curl: (52) Empty reply from server) instead of a 400. Those now raiseValueErrorlike everyother bad image source.
free_vram_mibis one helper taking an index rather than two copies of thesame NVML call.
Testing
tests/core/weights_carve_test.cppcovers the range predicate and the priced-vs-loaded agreement.tests/core/remote_budget_test.cppcovers the helper tier's selection.There is no multi-GPU integration test in this repo, so the carve was verified by A/B: the engine
driven directly through
--serve(no tokenizer, so stdin takesGEN <n> <ids>and stdout emitsT <id>per token), with an explicit split and everything pinned — explicit--prefill,STRATA_LOOKAHEAD=0,STRATA_STAGE_CACHE_SLOTS— two binaries, one token hash, and the same hashas before the change. The
--split-device 0bit-exact A/B is only meaningful with--no-remote-auto; with tiers on, tokens may legitimately differ from the CPU pool's float order.Merged
mainmainis merged into this branch (128 commits, 0.1.32–0.1.34) and the conflicts resolved instead ofre-opened: the PLE row-size design is
main's with Q8_0 re-added on top, and the upstream changesin the split region (
Verifier::set_commit_async, #340's streamed-ring reserve,stage_room'ssearch/reserve distinction) are ported onto this branch's structure rather than replacing it.
Verified after the merge: builds clean, 56 of 58 ctest targets pass serially (the two failures are
environmental —
ple_paritywants a model shard that is not in the tree,expert_multi_testwantsAVX512-VNNI/VBMI), and
serve's 120 Python tests pass.🤖 Generated with Claude Code