generate: one startup VRAM plan before the expert cache is committed (#765) by j-luwierski · Pull Request #796 · Niko1221/Strata

j-luwierski · 2026-10-04T16:46:22Z

Closes #765

Problem

At very large contexts, individually reasonable VRAM decisions combined into an
unworkable whole. Startup sized the expert cache from cudaMemGetInfo and a
one-line prefill estimate (160 + chunk * 680 / 1024, ~7x high at a
24576-token chunk), and only found out at the first prompt whether the chunk it
had committed to could still be served - the reported 524K failure: an int8 KV
of ~3.7 GiB, a cache that shrank around it, and a --prefill 24576 whose
buffers no longer fit anywhere, so the engine died between the model load and
READY.

Enky's measurements on a 5080 (Windows/WDDM) sharpened this: an allocation that
does not fit does not necessarily fail - under default WDDM the driver pages it
into shared system memory (+9.25 GiB of Shared Usage, prefill collapsed to
~105 tok/s), so "cudaMalloc succeeded" proves nothing about the configuration
being safe.

What this does

One startup VRAM plan (strata/prefill/vram_plan.{hpp,cpp}): the prompt
path is priced with the exact Prefill::bytes_needed() BEFORE the expert
cache is committed; expert residency is the elastic consumer and gives way
first; an impossible configuration is refused at startup with its budget and
the knobs that make room, instead of dying at the first prompt.
The chunk/loan policy exists once (plan_lend_chunks): the Prefill chunk ring #583
bisection with its ring what-ifs, 0.1.39's list, a fixed chunk's halving -
shared by the startup plan, the WDDM post-touch revalidation and the runtime
loans.
The accepted plan is the runtime's contract (EffectivePrefillPlan +
check_runtime_prefill_use()): generate and single-GPU serve consume the
accepted chunk/loan/ring directly and refuse - before any prompt allocation -
on a mode or size departure. The runtime may run less (a shorter prompt),
never more.
WDDM post-touch correction: every opened cache is touched, read and
revalidated; a shrink re-prices the prompt path against the final physical
cache (a borrowed plan that lost its loan is repriced as owned); the retry
limit bounds shrinking, never validation. The loan's request-time
bookkeeping (PfPart) is built from the accepted plan, so lend/refill keep
the cache residency contract.
One borrowing capability (prefill_borrow_available): the residency map
is built whenever the prompt path may borrow, token graph or not, and
startup and runtime read the same flag.
Enky's findings: --prefill auto owned fallbacks are logged (planned vs
unexpected); the serve pre-check refuses when no owned chunk through 512
fits instead of handing the allocation to WDDM (with a Windows sysmem
warning); owned fallbacks reduce on the 256-token grid; budget hints name
--kv-resident as a lever where the model supports KV streaming.

Measured (RTX 4070 Ti SUPER 16 GB, Linux)

arm	before	after
32K, `--prefill auto` (IQ3_XXS, int8 KV)	8192 borrowed / 2489 slots / 1946 tok/s	same plan, 1906–1933 tok/s
524K, int8 KV, `--prefill 24576`	died between model load and READY	17664 owned, 2139 tok/s prefill, decode 22.8 tok/s
524K, no KV residency, `--prefill 24576` (the WDDM-paged config)	unsafe 24576 handed to the driver	2048 owned, selected at startup, 1104 tok/s
serve, two 8K requests back to back	-	each lend marks 2489 experts, each refill restores 2489, no stale residency

Tests

vram_plan_test (new, no GPU): the planner's budget cases, the shared
chunk/loan policy, the runtime contract, Enky's fallback reductions (E1–E5)
and the runtime-mode cases (R19–R34) - all pass.
ctest: same results as main (the 9 pre-existing environment failures are
unchanged); prefill_fused_iq_test passes.
Existing behavior preserved: STRATA_RING_BYTES, STRATA_PREFILL_RING,
STRATA_PREFILL_LEND_PCT, the 128-slot loan floor, --expert-cache-per-layer,
--vram-elastic, the reserve adaptation on small cards (RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "")) #496), and multi-GPU
stage planning (untouched by design - the planner does not model stage caches
yet).

Not covered

Windows/WDDM validation with "Prefer No Sysmem Fallback" (no Windows machine
available); the planner decides from arithmetic before any allocation, so
the fallback setting has nothing to change - but the issue's 16-GiB
Blackwell arm deserves a re-run on the original rig.
Layer-split stage caches are still reserved with the split's own estimate
(sessions for later stages do not exist at planning time); their per-stage
serve logic is unchanged.

…iko1221#765) Startup sized the expert cache from cudaMemGetInfo and a one-line prefill estimate (160 + chunk * 680 / 1024, ~7x high at 24576 tokens), and only found out at the first prompt whether the chunk it had committed to could still be served - the 524K int8-KV failure of Niko1221#760: a --prefill 24576 the cache cannot lend fell back to its own buffers after the cache had taken everything, and the engine died between the model load and READY. - vram_plan.{hpp,cpp}: the plan (mandatory vs elastic) and the chunk/loan policy as ONE copy (plan_lend_chunks: the Niko1221#583 bisection with its ring what-ifs, 0.1.39's list, a fixed chunk's halving), pure over injected byte costs, shared by the startup plan and the runtime loans - the expert cache is sized by the plan: --prefill auto resolves against the planned cache before it is committed; an explicit chunk the cache cannot lend gets its own buffers booked exactly (reduced on the 256-token grid when even they do not fit, applied to --prefill); --spec keeps one cache slot; an impossible configuration is refused at startup with its budget and the knobs that make room - prefill: bytes_needed prices an explicit ring budget (the planner's what-if); the carve takes the streamed ring in one allocation - 384 separate 2.7 MiB cudaMallocs each rounded up to a 2 MiB page, ~0.5 GiB of VRAM the count never saw; the do-not-fit error carries the budget (free, allocated, counted, the failing buffer's size) - tests/core/vram_plan_test.cpp: the planner's cases A-F plus the policy's list, halving and bisection, without a GPU Measured on an RTX 4070 Ti SUPER (16 GB): at 32K with --prefill auto the plan matches the old engine exactly (8,192-token chunk, 2,487 borrowed slots, 5,578-slot cache, 1,950 tok/s prefill); the 524K int8-KV boot with --prefill 24576 now runs end to end (17,664-token chunk on its own buffers, 2,139 tok/s prefill, the cache keeps --spec's slot).

One memory invariant, enforced in one place over the layout that will actually be opened: mandatory Strata allocations + an owned prefill booked once (a borrowed one lives inside the cache) + the final expert-cache bytes <= the VRAM the plan saw. actual_cache_bytes() prices a sized (native pack + profile) layout at its exact offsets - the old uniform view priced the lend tail at max_blob a slot and approved loans the opened cache could not serve. - an explicit cache resolves --prefill auto with the same plan_lend_chunks the auto cache uses, against its exact sized layout built before the lend decision; a non-list boundary (8704) comes out of the bisection - the WDDM post-touch correction holds the cache to the plan's complete post-cache figure (reserve + draft head + an owned prefill), not just --vram-reserve-mib; post_cache_required_bytes() is the one definition - the carve's ring allocation fails cleanly (no pointer arithmetic on a null base; STRATA_TEST_RING_FAIL=1 exercises it), and bytes_needed counts the ring as the one allocation the carve takes - runtime_reserve_bytes stays 0 on one GPU by documented policy: the hit-path scratch, the verify windows and the decode graphs are funded by --vram-reserve-mib (Niko1221#199), which the correction now protects - vram_plan_test: every successful case asserts the final invariant through actual_cache_bytes/post_cache_required_bytes; new cases for the overflowing explicit cache, the explicit cache's auto fallback (4096, 8704), the sized cache's exact-suffix lend and the post-touch figure Measured on the RTX 4070 Ti SUPER (16 GB): the 32K boot is unchanged (8192-token chunk, 2489 borrowed slots, 5539-slot cache, 1914 tok/s prefill); the 524K int8-KV boot with --prefill 24576 --spec 4 plans a 17664-token owned chunk and finishes end to end with 9124 MiB free after the cache touch - exactly the plan's post-cache requirement.

…iko1221#765 review) The startup plan's lend decision did not survive the correction's own cache shrink: a borrowed 24576 could become a runtime-owned prompt whose buffers the accepted post-cache requirement never carried - the exact late-allocation failure Niko1221#765 closes. And the loop's retry limit broke out BEFORE touching the cache, so the last opened cache was accepted without any validation. - revalidate_prefill_after_cache(): the effective prompt path against the touched cache - a fixed chunk must lend EXACTLY (the runtime rejects a halved chunk with owned buffers at the original size), auto re-runs plan_lend_chunks and falls back to the runtime's owned 1024; effective_post_cache_required() carries an owned prompt exactly once - the correction loop touches, reads and revalidates EVERY opened cache; the retry limit bounds the shrinks only: the last allowed cache is accepted on its merits or fails with the full budget (free, required, reserve, draft head, prompt path, cache) instead of slipping through - the accepted plan is what the runtime runs: a prompt path validated as borrowed always lends at the prompt step, and a drift refuses startup rather than allocating buffers outside the budget - diagnostics: post-touch borrowed->owned and auto-chunk changes are said; STRATA_TEST_POSTTOUCH_SHORTFALL=N drives the whole correction deterministically, without a real desktop eating the card - vram_plan_test: R8-R15 - fixed stays borrowed, fixed flips owned at its original size, auto falls to a smaller loan, auto loses the loan to a priced 1024 fallback, the shrink decision (~700+ MiB), the retry limit failing and accepting, sized offsets over exact suffixes Measured on the RTX 4070 Ti SUPER (16 GB): 32K unchanged (8192-token chunk, 2489 borrowed slots, 1916 tok/s); with a 3000 MiB shortfall the correction shrinks the cache to 3204 slots / 5.23 GiB, keeps the prompt borrowed and the runtime lends exactly that; with 8500 MiB it fails after the six allowed shrinks with the full budget; the 524K arm runs end to end (17664 owned, 2136 tok/s prefill, 9126 MiB free after the touch against a 9124 MiB requirement).

…iko1221#765 review) Startup could accept a borrowed prefill while the runtime's borrowing infrastructure did not exist: the residency map (the loan's bookkeeping) was built only inside the token-graph setup, so --no-token-graph or --no-capture silently turned a borrowed plan into owned buffers at Prefill::init() - allocated outside the accepted VRAM budget. - ONE definition of the capability, prefill_borrow_available (an expert profile and not --no-prefill-borrow), read by the startup planner, the split's reserve arithmetic, the serve loan and the prompt path alike - the residency map is built whenever the prompt path may borrow, token graph or not: only the graph's hit wiring stays behind its own gate (the device copy keeps the paths that read one in sync, and the arena's release/prefetch now also runs in the borrowing-only case) - the mode guards sit outside every infrastructure condition, right before Prefill::init(): an accepted borrowed plan with no loan refuses startup, and so does an owned plan with a loan set up - the accepted EffectivePrefillPlan is the contract: its chunk is pinned into o.prefill_chunk when the correction accepts the cache - vram_plan_test: R16-R18 - the flag on lends, the flag off plans owned at the same chunk, the disabled flag keeps both sides owned Measured on the RTX 4070 Ti SUPER (16 GB): the normal path is unchanged (8192-token chunk, 2489 borrowed slots, 1933 tok/s prefill); under --no-token-graph and --no-pool the prompt path still borrows (2489 slots, ~1913 tok/s) where it silently went owned before (--no-capture alone keeps its pre-existing refusal: the unfused token loop has no CPU pool hook); the 524K arm runs end to end (17920 owned, 2167 tok/s prefill, decode 23.1 tok/s).

…1221#765, Enky's measurements) Enky measured on a 5080 (524K, IQ3_S, int8 KV, main 6f32ec0): an explicit --prefill 24576 whose buffers did not fit did not fail - under default WDDM the driver paged ~+9.25 GiB into shared system memory and prefill ran at ~105 tok/s. cudaMalloc succeeding says nothing about whether the allocation is in dedicated VRAM, so a keep-the-chunk-and-try strategy is not safe. - the serve pre-check's step-down had no else: when even 512 tokens did not fit, the requested chunk was kept and handed to the driver. It now refuses with the budget (requested, smallest checked, free, the 512-token need, headroom) and, on Windows, a plain warning that WDDM may otherwise page the buffers; the init-failure step-down stays as the secondary net for real allocation failures - --prefill auto's owned fallbacks are said, planned and runtime alike (generate distinguishes 'as the accepted VRAM plan planned' from 'expected a loan'; serve names its two no-loan cases) - the planner's owned 1024 fallback now reduces on the 256-token grid when even it does not fit, like an explicit chunk always did (largest_owned_fit walks from the fallback's bound, not from a request auto never made) - the budget hints name the KV-streaming lever where the model has one: 'or stream more KV from RAM (--kv-resident, e.g. 32768)' - the biggest single lever in Enky's table (his A vs AN: 8.26 vs 1.27 GiB free) - vram_plan_test: E1-E5 - the fallback is explicit and priced, the reduction is said at planning, nothing-fits refuses, the budget alone decides (an allocator that would succeed is not consulted), and the pricing is bytes_needed + kOwnedPageMarginBytes Measured on the RTX 4070 Ti SUPER (16 GB, Linux): the Enky CN arm (524K, int8 KV, no --kv-resident, --prefill 24576) selects a 2048-token owned chunk and runs at 1104 tok/s - the unsafe 24576 is never attempted, so there is nothing for WDDM to page; the same chunk prices at 397 MiB without KV residency and 910 MiB with it, and the plan-fail hint names --kv-resident. 32K unchanged (8192 borrowed, 1906 tok/s).

…Niko1221#765 review) A validated owned auto fallback below 1024 could be raised back to 1024: the planner priced and accepted 768, and the runtime's hard-coded request_chunk(n, 1024) then allocated 1024 - outside the accepted budget, exactly the silent growth Niko1221#765 closes. Single-GPU serve also ran its own chunk/loan scan with no contract check against the accepted plan. - check_runtime_prefill_use(): THE contract as one pure function - the runtime may run less (a shorter prompt), never more and never another mode; generate and single-GPU serve both refuse on it right before their prefill init, and R19-R28 test it without CUDA - a validated borrowed plan is not re-scanned: generate and serve consume the accepted chunk, loan and ring budget directly and only shrink for a shorter prompt (one decision, not two expected to agree) - an accepted owned fallback's ceiling is the accepted chunk: the runtime's request_chunk runs against it, never a fresh 1024; the explicit-cache branch reduces its owned fallback like every other - serve's auto fallback logs print the actual accepted chunk; the STRATA_TEST_SERVE_DROP_LOAN=1 hook proves the serve guard refuses a borrowed plan without its loan instead of relying on inspection - multi-GPU keeps its stage-aware serve logic; the Enky c == 0 refusal and every earlier fix stay Measured on the RTX 4070 Ti SUPER (16 GB): the forced below-1024 case (explicit 100-slot cache, tuned reserve) plans owned 768 and runs 768 - 12 chunks, never 1024 (873 tok/s prefill); serve with a dropped loan refuses before sp.init(); normal serve applies the accepted 8192/2489 loan and reaches READY; 32K unchanged (1912 tok/s); the 524K arms run end to end (1792 owned without KV residency at 973 tok/s, 17664 owned with it at 2139 tok/s - the planner tracks the actual budget, never grows after acceptance).

Niko1221#765 review) af2cdd4's serve_follows_accepted skipped the whole loan-scan block - and with it the CUDA0 PfPart, which is not just decision input but the request-time bookkeeping: the per-request lend marks the lent slots' experts not-resident through it and the refill restores them. An accepted borrowed single-GPU serve could reach READY with borrow pointing into the cache tail and an empty pf_parts, so a request would overwrite the tail while host_res/d_res still claimed experts there. - the accepted borrowed loan builds CUDA0's PfPart from the accepted plan (first = slots - lend_slots) and derives the loan pointer and bytes through the same part_bytes helper the scan path uses; no scan is re-run - accepted owned keeps pf_parts empty and borrow null (lend(): nothing to lend) - the runtime contract now also caps the loan's BYTES: a shorter chunk under a different ring rule (the accepted budget vs the 0.1.39 ring for chunks at or below its small_max) can price more loan bytes than the accepted one; EffectivePrefillPlan carries lend_bytes, the callers pass borrow_bytes - STRATA_TRACE's request-loan/refill traces name the first slot, the evicted count and what is left lent; the accepted loan's line carries its first slot Measured on the RTX 4070 Ti SUPER (16 GB): serve, two real 8192-token requests back to back - each lend marks 2489 experts not-resident and each refill restores 2489 with 0 left lent, so the second request starts from restored residency; the dropped-loan hook still refuses before sp.init; 32K generate unchanged (8192 borrowed, 1916 tok/s).

blange48 · 2026-10-04T19:52:27Z

Feedback on a multi-GPU split (rig: 4 x RTX 5080 16 GB (PCIe Gen3), Linux, IQ3_S, 262K context, int8 K/V, layer split 12,24,36 with --trim-stage-weights, --batch 8 --batch-groups 4, vision on (GPU encoder)): built on top of 0.1.39 and tested. Not deployed.

Good: nothing changed on our split, and the outputs stay exact:
- tools/batch_test.py with 8 and 3 slots and tools/parking_test.py are identical (greedy, --pcie-frac 0);
- the server starts at 262K as before.
Neutral: the expert cache came out the same as without the patch (5,332 experts, 9.40 GiB), and so did the throughput:
- 1 / 2 / 4 / 8 clients: 119 / 150 / 185 / 365 tok/s, within our run-to-run noise;
- a 45K-token prompt: 12.65 s against 12.72 s.
That matches your description, which targets the single-GPU serve path. I mention it in case the split path was meant to get the plan too.

T-Crypt · 2026-10-04T21:42:52Z

Tested on a single RTX 4090 (sm_89, Linux, 32 GB RAM), Flash-Next IQ2_XS, --expert-cache auto --prefill auto --spec 4 --mtp --max-context 262144 --kv q4_0 --resident-experts, on v0.1.39 + #783 with this PR merged on top (merged cleanly). vram_plan_test passes. I alternated against the same build without this PR, two rounds:

Startup accepted the plan. Expert cache is the same size (11,818 slots, 15.84 GiB). The prompt path is now lent 2,806 slots (3.77 GiB) instead of 3,040 (4.08 GiB) under the old estimate.
No speed change: prompt reading 4,046 / 3,900 t/s and decode 121.5 / 139 t/s at 119K / 238K, both identical within noise. Recall correct in every run.

So on this config it frees about 0.3 GiB of loan without costing anything.

gopinath87607 · 2026-10-05T03:51:44Z

guys can you check this pr and share your thought

#880

j-luwierski · 2026-10-05T06:15:28Z

Thanks a lot @blange48 for testing this on a multi-GPU setup! I'd be very happy to extend the planner to properly cover the multi-GPU split path as well. Unfortunately I only have a single GPU available, so I can't really validate that path myself. Your test results are very helpful here.

j-luwierski · 2026-10-05T06:21:17Z

Hi @gopinath87607 Thanks! I’d be happy to take a look at your changes when I get back home.

As I mentioned earlier though, I only have a single GPU available, so I don’t have a proper multi-GPU setup to validate the behavior end-to-end. I can still review the changes and see how they interact with my VRAM planning work.

Zauberio · 2026-10-05T07:08:21Z

Tested on Windows — the platform class your problem statement cites (WDDM paging), single GPU. Rig: RTX 5070 Ti 16 GB (driver 617.14), Windows 11 Pro build 26300, 62 GiB RAM. Model: Qwen3.8-Flash-Next IQ3_S, 46.84 GiB across 2 shards, --max-context 262144 --kv q4_0 --kv-resident 32768 --vram-reserve-mib 800 --prefill auto:32768 --spec 4, served via serve/server.py.

Build: PR head 65e1581 built from source on the box (CUDA 13.3.33, MSVC), strata.exe sha256 1fb43743…5f80e7f9537cc83ae73fdab56d7b5a6. Swapped into the live engine path against stock 0.1.39 (d4c3a51f…), fresh boot per leg, greedy (temp 0) probes with a unique marker per run so the prompt cache never hits, box-side timing.

Startup plan accepted, and it matches what the runtime actually does. Stock 0.1.39 on this rig negotiates prefill auto: requested ceiling 32768, selected 20224 due to the VRAM budget, then post-touch shrinks to 19712 and lends 3715 cache slots (7.07 GiB). The PR's plan reports the same numbers up front (post-touch cache shrink reduced prefill auto from 20224 to 19712 tokens, lends the prompt path 3715 cache slots (7.07 GiB) for 19712-token chunks) — the plan and the realized loan agree, which is the whole point.

The refusal path fires, cleanly. Using your STRATA_TEST_SERVE_DROP_LOAN=1 hook: the engine logs dropping the accepted loan, then refusing rather than departing from the accepted budget, and startup stops there — no OOM death between load and READY, no WDDM paging attempt, no 105 tok/s zombie. Reproduced twice (two separate boots). Red control: the same env var on stock 0.1.39 is inert — it boots normally and probes steady (3,325–3,343 tok/s), so the refusal is genuinely this PR's behavior, not the rig.

Steady state unchanged. pr796 boot: 22.6K-token probe 3,130 tok/s first run, 40.6K-token probe 3,359 tok/s; stock legs in the same hour: 3,330–3,382 tok/s across both sizes. Within the ±5% cross-boot band we measure on this box — no prefill regression from planning before the cache commit.

No multi-GPU here, so nothing to add on @blange48's split path. Nice work — the plan/realized-loan agreement at startup is exactly what was missing on this WDDM rig.

paulhothersall · 2026-10-05T08:03:48Z

On a mixed Tesla P40 + RTX 3070 pair this PR changes nothing I can measure at a normal context (same split, slots, chunk and speed as v0.1.39), and on a single P40 at 98K context the plan accepts the same configuration. One launch per row; I did not test the large-context cases the PR is aimed at.

Setup: v0.1.39 and v0.1.39 + this PR at 65e1581, CUDA 12.9 (-DCMAKE_CUDA_ARCHITECTURES="61;86"), P40 24 GB (cc 6.1) + RTX 3070 8 GB (cc 8.6), no P2P, 64 GB RAM, Qwen3.8-Flash-Next IQ3_XXS, --layer-split auto, context 36,864, 3 runs per size. Prefill = prompt tokens / time to first token.

case	v0.1.39	+ #796
3070 first, auto	K=43, slots 1,536 / 2,560, chunk 4096, prefill 462 / 617 tok/s (4K / 32K; earlier run)	K=43, 1,536 / 2,560, chunk 4096, 460 / 617
P40 first, auto	K=16, slots 8,192 / 736, chunk 1024, prefill 192 / 220	K=16, 8,192 / 736, chunk 1024, 193 / 220
P40 alone, `--max-context 98304`, 4K prompt	9,537 slots, chunk 8192, 12.9 s to first token	9,537 slots, chunk 8192, 12.9 s; the log reports "the accepted VRAM plan lends the prompt path 2436 cache slots (4.00 GiB) for 8192-token chunks"

Decode stayed inside the usual noise (30-41 tok/s) in all rows. Not tested: contexts above 98K (the 262K / 524K cases in the description), q4 or int8 KV, Windows / WDDM, three or more GPUs, the vram_plan_test target, a second launch.

paulhothersall · 2026-10-06T02:33:49Z

On an 8 GB RTX 3070 alone with IQ3_XXS and --max-context 65536, v0.1.39 reports ready and then dies with an out-of-memory error on the first prompt; with this PR the same command is refused at startup, with the budget and the knobs that make room.

Setup: v0.1.39 and v0.1.39 + this PR at 65e1581, CUDA 12.9, RTX 3070 8 GB alone, Qwen3.8-Flash-Next IQ3_XXS, --layer-split not used, --max-context 65536, default flags.

v0.1.39: strata serve: 66 MiB of VRAM free with everything loaded - LOW: requests may stall; add --vram-reserve-mib 1146 ... (or lower --max-context), the server says ready, and the first request fails with ggml-cuda (strata mmq): out of memory: cudaMalloc.

This PR: the load is refused with user reserve: 700 MiB, short by: 1371 MiB and "To make room: a smaller --max-context (the session takes 1770 MiB at 65536 tokens), --kv q4_0, a smaller --prefill, a smaller --vram-reserve-mib, or stream more KV from RAM (--kv-resident, ...)". That is the behaviour the description asks for, on a card class (8 GB) your own single-GPU testing may not cover. One launch each. Not tested: whether the suggested knobs then load and run on this card, Windows / WDDM, other cards.

…ricing (STRATA_OWNED_PRICE=exact), OOM diagnostics (#796 part A) - carve: with owned buffers the streamed ring is one cudaMalloc instead of up to 1,024 pieces each rounded to 2 MiB (~0.5 GiB). A borrowed region keeps its per-slot layout and price, so the loans' slot counts do not move. - Prefill::bytes_needed_owned prices the real allocation (2 MiB pages, the ring in one piece). The cache sizing uses it only with STRATA_OWNED_PRICE=exact; the default is still 0.1.39's 160 + chunk * 680 / 1024 rule. - the "device buffers do not fit" error names the free VRAM, the bytes taken and the buffer that failed. No refusal was added: parts B (borrow_residency) and C (the advisory planner) stay out. Co-authored-by: Jakub Luwierski <j.luwierski@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Niko1221 · 2026-10-06T06:45:30Z

Part A is in 0.1.40: the prompt path's own buffers take the ring in one allocation, the 'do not fit' error now says what was free and what failed, and Prefill::bytes_needed_owned prices the real allocation (STRATA_OWNED_PRICE=exact). Loans keep their 0.1.39 layout and price, so slot counts do not move. Parts B and C (borrow_residency, the advisory planner) are not in this release; please send them as separate PRs where every refusal is a warning.

…ricing (STRATA_OWNED_PRICE=exact), OOM diagnostics (#796 part A) - carve: with owned buffers the streamed ring is one cudaMalloc instead of up to 1,024 pieces each rounded to 2 MiB (~0.5 GiB). A borrowed region keeps its per-slot layout and price, so the loans' slot counts do not move. - Prefill::bytes_needed_owned prices the real allocation (2 MiB pages, the ring in one piece). The cache sizing uses it only with STRATA_OWNED_PRICE=exact; the default is still 0.1.39's 160 + chunk * 680 / 1024 rule. - the "device buffers do not fit" error names the free VRAM, the bytes taken and the buffer that failed. No refusal was added: parts B (borrow_residency) and C (the advisory planner) stay out. Co-authored-by: Jakub Luwierski <j.luwierski@gmail.com>

j-luwierski · 2026-10-06T11:37:57Z

Working on it.

… token graph (Niko1221#796 part B) The residency table (host_res/d_res) was built only inside the token-graph setup, so the prompt loan - whose lend marks the lent slots' experts not-resident and whose refill restores them - lost its bookkeeping under --no-token-graph or --no-capture: borrowing silently fell back to owned buffers the cache sizing had not reserved room for. - the table is now built whenever the prompt path can borrow (an expert profile, not --no-prefill-borrow, and a --prefill chunk), token graph or not; the graph's hit wiring (the device hit count, the TokenHits fields, drive.d.host_res) stays behind the graph's own gate, and the arena release/prefetch also runs in the borrowing-only case - the decision is one pure predicate, strata::prefill::borrow_residency, with its mode matrix unit-tested (tests/core/borrow_residency_test.cpp); the decode pool's view (drive.d.host_res) is unchanged - STRATA_TRACE names the loan (slots, first slot, marked non-resident) and the refill (restored, 0 left lent) so consecutive borrowed requests can be checked from the log Measured on the RTX 4070 Ti SUPER (16 GB), IQ3_XXS native pack, 12K prompt, --prefill auto: the normal captured path is unchanged (8192-token chunk, 2489 borrowed slots, 2198.6 tok/s prefill); under --no-token-graph the prompt path now borrows the same 2489 slots at 2195.5 tok/s where v0.1.40 silently went owned; --no-prefill-borrow and the no-profile runs still own their buffers and build no residency state; serve, two 8192-token requests back to back: each loan marks 2489 non-resident and each refill restores 2489 with 0 left lent. --no-capture + borrowing cannot start on a native pack (it needs --spec, a pre-existing requirement), so that mode is code-verified only.

… sees, and only warns (Niko1221#796 part C) Part A's exact owned pricing (0.1.40) made the prompt path priceable before the expert cache is committed; the planner here SAYS that arithmetic before the cache takes what is left of the VRAM - and warns instead of refusing, which is the one design change from the Niko1221#796 draft Niko asked for: no startup refusal, no runtime contract, no second allocator. - advise_startup_vram (new strata/prefill/vram_plan.{hpp,cpp}, pure, injected costs) prices free VRAM, the reserve, the draft head, the prompt path (owned at the exact bytes_needed_owned price beside the sizing rule's reservation, or the loan the shared plan_lend_chunks predictor derives), the predicted cache (the sizing walk, sized slots included) and the Niko1221#496 reserve adaptation the runtime is expected to make. A shortfall, an over-large --expert-cache or an owned fallback is a warning with the knobs that make room; nothing is set, shrunk or stopped - after the cache opens: the free read is compared with the requirement (the auto sizing's own post-touch loop already wrote the slots), the prompt path is re-derived against the cache that actually exists, and a moved prediction (borrowed -> owned, a different auto chunk) is one warning. STRATA_POSTTOUCH=1 writes an explicit cache's slots before the read - the deep WDDM check, off unless asked (env_enabled: unset/0 off, non-zero on) - at the generate loan and the serve loan scan: check_runtime_prefill_use reports what the runtime resolved against the prediction - one warning line, never a refusal - the planner prices at ITS OWN measured pinned share through the new share-pure Prefill pricing (bytes_needed_at_share / ring_slots_at_share / ring_default_slots_at / ring_cap_for_at: the same rules with the share in place of the global), so it neither sets nor waits for Prefill::set_pinned_share - the runtime keeps its upstream call site below the cache sizing, and with STRATA_ADVISORY_PLAN=0 the run reaches that site exactly as pristine main does; kAutoLendPct/auto_ceiling are hoisted above the sizing so planner and scans price with the same numbers, and the ring what-if is additive (bytes_needed_no_ring + budget slots), touching no Prefill global - env flags go through one parser (env_enabled): STRATA_ADVISORY_PLAN unset/non-zero on, =0 off (previously ANY value disabled it); STRATA_POSTTOUCH unset/0 off, non-zero on - vram_plan_test: the old draft's useful coverage (the 13824/8704 bisections, sized offsets, the reserve as the user's knob, R8-R15 re-derivations, R19+ mismatch rules) with every !p.ok refusal assertion rewritten as has_warning() + usable numbers, plus the env parser No runtime decision moves: 3-way parity on the RTX 4070 Ti SUPER (IQ3_XXS, 12K prompt, --prefill auto) - pristine main vs STRATA_ADVISORY_PLAN=0 vs advisory on, with the arena fully pinned AND with STRATA_ARENA_PIN_GIB=4 (pinned share ~0.1, where a hoisted set_pinned_share would have shifted the runtime's ring pricing): the pinned<0.9 runs are decision-identical across all three binaries, and in the pinned case the 8192-token chunk and the 2489-slot loan are identical everywhere (cache slot counts jitter +-1% between ANY two runs on this desktop - main vs itself spans the same spread). Prefill 2190-2194 tok/s pinned, 1410-1427 tok/s unpinned - run noise. ctest: the same 3 pre-existing machine failures as pristine main; vram_plan_test passes. Multi-GPU untouched: the planner and every comparison run single-GPU only.

…ricing (STRATA_OWNED_PRICE=exact), OOM diagnostics (Niko1221#796 part A) - carve: with owned buffers the streamed ring is one cudaMalloc instead of up to 1,024 pieces each rounded to 2 MiB (~0.5 GiB). A borrowed region keeps its per-slot layout and price, so the loans' slot counts do not move. - Prefill::bytes_needed_owned prices the real allocation (2 MiB pages, the ring in one piece). The cache sizing uses it only with STRATA_OWNED_PRICE=exact; the default is still 0.1.39's 160 + chunk * 680 / 1024 rule. - the "device buffers do not fit" error names the free VRAM, the bytes taken and the buffer that failed. No refusal was added: parts B (borrow_residency) and C (the advisory planner) stay out. Co-authored-by: Jakub Luwierski <j.luwierski@gmail.com>

…Niko1221#796）リングを 1 回で確保する部分と確保量の見積もり（STRATA_OWNED_PRICE=exact）は、 CUDA の 2 MiB ページの丸めが前提なので、Xe での VRAM を測ってから決める。

… token graph (Niko1221#796 part B) The residency table (host_res/d_res) was built only inside the token-graph setup, so the prompt loan - whose lend marks the lent slots' experts not-resident and whose refill restores them - lost its bookkeeping under --no-token-graph or --no-capture: borrowing silently fell back to owned buffers the cache sizing had not reserved room for. - the table is now built whenever the prompt path can borrow (an expert profile, not --no-prefill-borrow, and a --prefill chunk), token graph or not; the graph's hit wiring (the device hit count, the TokenHits fields, drive.d.host_res) stays behind the graph's own gate, and the arena release/prefetch also runs in the borrowing-only case - the decision is one pure predicate, strata::prefill::borrow_residency, with its mode matrix unit-tested (tests/core/borrow_residency_test.cpp); the decode pool's view (drive.d.host_res) is unchanged - STRATA_TRACE names the loan (slots, first slot, marked non-resident) and the refill (restored, 0 left lent) so consecutive borrowed requests can be checked from the log Measured on the RTX 4070 Ti SUPER (16 GB), IQ3_XXS native pack, 12K prompt, --prefill auto: the normal captured path is unchanged (8192-token chunk, 2489 borrowed slots, 2198.6 tok/s prefill); under --no-token-graph the prompt path now borrows the same 2489 slots at 2195.5 tok/s where v0.1.40 silently went owned; --no-prefill-borrow and the no-profile runs still own their buffers and build no residency state; serve, two 8192-token requests back to back: each loan marks 2489 non-resident and each refill restores 2489 with 0 left lent. --no-capture + borrowing cannot start on a native pack (it needs --spec, a pre-existing requirement), so that mode is code-verified only.

… sees, and only warns (Niko1221#796 part C) Part A's exact owned pricing (0.1.40) made the prompt path priceable before the expert cache is committed; the planner here SAYS that arithmetic before the cache takes what is left of the VRAM - and warns instead of refusing, which is the one design change from the Niko1221#796 draft Niko asked for: no startup refusal, no runtime contract, no second allocator. - advise_startup_vram (new strata/prefill/vram_plan.{hpp,cpp}, pure, injected costs) prices free VRAM, the reserve, the draft head, the prompt path (owned at the exact bytes_needed_owned price beside the sizing rule's reservation, or the loan the shared plan_lend_chunks predictor derives), the predicted cache (the sizing walk, sized slots included) and the Niko1221#496 reserve adaptation the runtime is expected to make. A shortfall, an over-large --expert-cache or an owned fallback is a warning with the knobs that make room; nothing is set, shrunk or stopped - after the cache opens: the free read is compared with the requirement (the auto sizing's own post-touch loop already wrote the slots), the prompt path is re-derived against the cache that actually exists, and a moved prediction (borrowed -> owned, a different auto chunk) is one warning. STRATA_POSTTOUCH=1 writes an explicit cache's slots before the read - the deep WDDM check, off unless asked (env_enabled: unset/0 off, non-zero on) - at the generate loan and the serve loan scan: check_runtime_prefill_use reports what the runtime resolved against the prediction - one warning line, never a refusal - the planner prices at ITS OWN measured pinned share through the new share-pure Prefill pricing (bytes_needed_at_share / ring_slots_at_share / ring_default_slots_at / ring_cap_for_at: the same rules with the share in place of the global), so it neither sets nor waits for Prefill::set_pinned_share - the runtime keeps its upstream call site below the cache sizing, and with STRATA_ADVISORY_PLAN=0 the run reaches that site exactly as pristine main does; kAutoLendPct/auto_ceiling are hoisted above the sizing so planner and scans price with the same numbers, and the ring what-if is additive (bytes_needed_no_ring + budget slots), touching no Prefill global - env flags go through one parser (env_enabled): STRATA_ADVISORY_PLAN unset/non-zero on, =0 off (previously ANY value disabled it); STRATA_POSTTOUCH unset/0 off, non-zero on - vram_plan_test: the old draft's useful coverage (the 13824/8704 bisections, sized offsets, the reserve as the user's knob, R8-R15 re-derivations, R19+ mismatch rules) with every !p.ok refusal assertion rewritten as has_warning() + usable numbers, plus the env parser No runtime decision moves: 3-way parity on the RTX 4070 Ti SUPER (IQ3_XXS, 12K prompt, --prefill auto) - pristine main vs STRATA_ADVISORY_PLAN=0 vs advisory on, with the arena fully pinned AND with STRATA_ARENA_PIN_GIB=4 (pinned share ~0.1, where a hoisted set_pinned_share would have shifted the runtime's ring pricing): the pinned<0.9 runs are decision-identical across all three binaries, and in the pinned case the 8192-token chunk and the 2489-slot loan are identical everywhere (cache slot counts jitter +-1% between ANY two runs on this desktop - main vs itself spans the same spread). Prefill 2190-2194 tok/s pinned, 1410-1427 tok/s unpinned - run noise. ctest: the same 3 pre-existing machine failures as pristine main; vram_plan_test passes. Multi-GPU untouched: the planner and every comparison run single-GPU only.

j-luwierski added 7 commits October 4, 2026 16:21

j-luwierski mentioned this pull request Oct 4, 2026

Make VRAM budgeting aware of KV, prefill staging and expert residency #765

Closed

jmnargi mentioned this pull request Oct 4, 2026

feat: one startup VRAM plan before the expert cache is committed jmnargi/Strata-V100#29

Merged

jomcgi mentioned this pull request Oct 4, 2026

inference-oom: elastic expert VRAM that tracks the KV footprint jomcgi-org/homelab#6841

Open

j-luwierski mentioned this pull request Oct 5, 2026

layer split: the stage weight trim works under auto too 4X gain in the prefill in 4 way ring #880

Closed

hiru0118 mentioned this pull request Oct 6, 2026

8 GB card: 0.1.39's default prefill is ~40% slower than 0.1.31 (prompt chunk 512 -> 256 when borrowing cache slots) #1009

Closed

Niko1221 closed this Oct 6, 2026

sergqwer mentioned this pull request Oct 6, 2026

prefill: bytes_needed sizes the MoE buffers for the layout init picks (with or without an expert source) #1042

Closed

This was referenced Oct 6, 2026

generate: the prompt loan keeps its residency bookkeeping without the token graph (#796 part B) #1131

Closed

prefill: the advisory VRAM plan - the startup arithmetic says what it sees, and only warns (#796 part C) #1133

Closed

sergqwer mentioned this pull request Oct 7, 2026

prefill: bytes_needed sizes the MoE buffers for the layout init picks (with or without an expert source) #1283

Closed

Niko1221 mentioned this pull request Oct 7, 2026

Proposal: single-GPU prefill buffer planning and RAM budgeting for conversation caching (RTX 5090 measurements) #658

Closed

This was referenced Oct 20, 2026

[Engine] preserve prompt loan residency without token graph #3391

Draft

[Engine] add opt-in advisory VRAM plan #3392

Draft

Navigation Menu

Repository navigation

generate: one startup VRAM plan before the expert cache is committed (#765) - #796

generate: one startup VRAM plan before the expert cache is committed (#765)#796
j-luwierski wants to merge 7 commits into
Niko1221:mainfrom
j-luwierski:fix/765-startup-vram-plan

j-luwierski commented Oct 4, 2026

Uh oh!

blange48 commented Oct 4, 2026

Uh oh!

T-Crypt commented Oct 4, 2026

Uh oh!

gopinath87607 commented Oct 5, 2026

Uh oh!

j-luwierski commented Oct 5, 2026

Uh oh!

j-luwierski commented Oct 5, 2026

Uh oh!

Zauberio commented Oct 5, 2026 •

edited

Loading

Uh oh!

paulhothersall commented Oct 5, 2026

Uh oh!

paulhothersall commented Oct 6, 2026

Uh oh!

Niko1221 commented Oct 6, 2026

Uh oh!

j-luwierski commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

j-luwierski commented Oct 4, 2026

Problem

What this does

Measured (RTX 4070 Ti SUPER 16 GB, Linux)

Tests

Not covered

Uh oh!

blange48 commented Oct 4, 2026

Uh oh!

T-Crypt commented Oct 4, 2026

Uh oh!

gopinath87607 commented Oct 5, 2026

Uh oh!

j-luwierski commented Oct 5, 2026

Uh oh!

j-luwierski commented Oct 5, 2026

Uh oh!

Zauberio commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paulhothersall commented Oct 5, 2026

Uh oh!

paulhothersall commented Oct 6, 2026

Uh oh!

Niko1221 commented Oct 6, 2026

Uh oh!

j-luwierski commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Zauberio commented Oct 5, 2026 •

edited

Loading