Repository navigation
Conversation
|
Another Windows data point from a local port of #1323 + #1324 onto v0.1.40.2: Ryzen 9 7940HS, RTX 4070 Laptop 8 GB, 64 GB RAM, IQ3_S, 64K context. I kept the total expert RAM budget at 36 GiB: 32 GiB fixed + 4 GiB LRU versus 36 GiB fixed in the control. The elasticity worked in a bounded pressure test. With I haven't established a speed win on this machine. Fresh prompt-read medians, two samples per size:
So the RAM headroom behavior is useful here, particularly for agents that also compile, while the smaller prompt reads need weighing against it. The port retained the release's stager-wait and stream-ahead policies; this is not a test of the original PR binary. Actual free VRAM stayed at least 353 MiB. |
…erts.bin view once reads are unbuffered NTFS serves unbuffered reads of a mapped file one at a time (WD SN580: 2 MiB reads at QD32 2.35 GB/s mapped vs 3.57 GB/s unmapped). With the file tier unbuffered (--resident-budget-gib or STRATA_UNBUFFERED_LOAD=1) the mapped view is only a fallback after startup, so it is closed (STRATA_KEEP_MAPPING=1 keeps it). The prompt path's stager claims runs of consecutive jobs and reads them with one read_direct batch (STRATA_STAGER_BATCH, default 8 for file-backed blobs). RTX 5070 Ti, 32 GB RAM, IQ3_XXS, 47.6K prompt, --resident-budget-gib 18 --prefill auto:32768: 1260 -> 1370 tok/s (batch 8) -> 1669 tok/s (view closed). Ported from Niko1221#833 onto the rewritten main. Two fixes on the way: unmapped_ is declared on every platform (mapped_blob reads it outside the Windows block, so Linux did not compile), and copy_blobs asks resident_blob, which follows the exchange rotation (STRATA_EXCHANGE_ROTATE). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oss decode windows (experiment) With unbuffered reads nothing caches a file-tier expert once its stage buffer is reused, three layers later, so an expert routed again in the next decode window is read from the drive again. Measured on bestia (IQ3_S, 2x RTX 5060 Ti, --resident-budget-gib 8, unbuffered): a 600-token answer read 8.5-14 GB. The pool now grows to the given size before it reuses a buffer, least recently used first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of reading the drive again With STRATA_STAGE_KEEP_MIB the pool keeps experts the decode read from the files; the prompt path's stager now copies those from memory. It neither admits nor refreshes entries, so a prompt's sweep over every cold expert does not push out what the decode reuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of the experts the decode reads The RAM budget was filled once from the global expert profile. Replaying a traced session (IQ3_S, 2x RTX 5060 Ti, six answers on different topics) against other policies of the same size, an LRU fed by the decode's file reads cut the decode's drive reads by ~45%; measured, 10 GiB by profile + 4 GiB LRU decoded ~12% faster than 14 GiB by profile. The LRU is the unbuffered reads' stage pool; the serve log reports its hits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the GPU caches (measurement) One line per blob - time, tier (RAM copy or files), decode or prompt path, layer, expert, bytes - so the RAM tier's choice of experts can be compared offline against other policies on a real workload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…b (decode, prompt path, fills) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to Windows instead of keeping them The LRU part of the RAM budget was ordinary memory: under pressure Windows paged it out, and reading it back was slower than the drive. Buffers unused for about two decode windows are now offered (OfferVirtualMemory, low priority): the OS may discard them for other programs, never page them, and a hit reclaims the buffer first, reading the expert again when its contents are gone. So the LRU can be large while the PC is idle and shrink by itself when other programs need the RAM. The serve log reports offers, intact and discarded reclaims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Offering each idle LRU buffer to Windows (STRATA_LRU_OFFER) cost half the decode speed: every offer and reclaim moves hundreds of pages. Instead a watcher checks the available RAM twice a second: below the floor it frees the coldest LRU buffers (back to the OS at once, nothing paged) and lowers the pool's cap; with RAM free again it raises the cap a GiB at a time up to --resident-lru-gib. Measured, a 12 + 18 GiB budget with a plain LRU decoded at 64.0 tok/s against 59.3 for 12 + 6, prompts 958 / 1514 against 768 / 1412 tok/s (8K / 32K). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Squeezed below what one decode window claims at once, the pool allocated and freed its buffers every window: decode fell to 43 tok/s under memory pressure, reading up to 52 GB per 600-token answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…astic LRU replaces it Offering idle LRU buffers to Windows halved decode speed (every offer and reclaim moves hundreds of pages); the elastic LRU gives RAM back when other programs need it without a call per access. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, 2x RTX 5060 Ti, 64 GB) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every LRU buffer is committed memory. Watching only the available RAM, the LRU could fill the commit charge up to its limit (112-120 of 121 GiB while serving, with 8-9 GB of RAM free), and then another program's allocation fails even with RAM free. The watcher now takes the smaller of the available RAM and the available commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
20e55bc to
4b28dc1
Compare
|
Thanks for porting and testing it, and for the pressure test. On elasticity: agreed, it is a gradual response, not a guarantee. The watcher polls every 0.5 s and gives back the On speed: the prompt-read cost you measured is expected and matches what I saw. The prompt path copies from the LRU Where RAM allows, the LRU can also go on top of the fixed budget instead of being carved out of it; then the prompt I have pushed two updates to the branch:
One Windows finding that may matter on your machine too: with a system-managed page file, the commit limit shrinks AI-generated. This reply, the rebase and the measurements in it were made by an AI (Claude Opus 5.5, via Claude Code) on the submitter's behalf. |
|
Agreed: my table measures prompt processing, so it does not settle the decode benefit. The short, sampled high-reasoning replies in that run are not a substitute for the sustained multi-topic decode comparison you describe. I would keep the conclusion as “elasticity verified under gradual pressure; decode speed still unqualified on this laptop.” I checked the startup evidence too: the requested 32 GiB fixed + 4 GiB LRU split was retained, with 67.5 GiB of commit available immediately before the fixed allocation. So startup budget clamping was not the cause of this particular prompt result. I have not benchmarked the new commit-aware watcher at |
What the release brought for this box (Niko1221#1709 Windows HIP VRAM floor, the HIP stream-capture fix, the Linux-only PCIe telemetry), the merge table with what was kept and why Niko1221#1439/Niko1221#1841/Niko1221#1656 went, the measurements (0.1.42+local +5.6% at 16K; the four new PRs +4.4% on top), and the ranked shortlist for the next cycle (Niko1221#1297 bisect, Niko1221#1909, Niko1221#1324, Niko1221#1167, Niko1221#1298/Niko1221#1151) with the gfx1100 reading list.
The auto-merge wove in the preamble (stage_keep_ / lru_cap_ / stage_live_), so the conflict was the allocator: upstream's pinned cudaHostAlloc path is strictly better than the PR's VirtualAlloc, so it stays, with the LRU's stage_live_ accounting added and the PR's freed-slot reuse deliberately NOT taken (that is the half that could hand out a slot the watcher freed; it only matters after a shrink). Off by default: stage_keep_ is 0 and the watcher only starts with the opt-in.
…ropped at the merge boundary)
… consults the staged tier)
…10.9% Six PRs merged individually (Niko1221#1297 Niko1221#1909 Niko1221#1324 Niko1221#1167 Niko1221#1298 Niko1221#1151), none of which changes the default path (the arms are opt-in and Niko1221#1297 needs a tuning table). Reading the issue tracker produced the real win: issue Niko1221#1549 documents ring_default_slots() sizing the staging ring at 384 (1024 for a fused ring) when g_pinned_share >= 0.9, and our STRATA_PREFILL_RING=96 override - left from tuning against a smaller budget - was costing 16.5% at 4K and 10.9% at 16K. The override is now gone, so the engine's own rule applies. Measured, with the LRU left off (it buys prefill but eats the resident budget: 11.9 -> 9.1 GiB).
…p-server.bat fixed A (in-tree levers): - Niko1221#1297 kernel tuning: RAN. The tuner kept the DEFAULT shape in all nine cases (3 expert shapes + 6 mmvq shapes) and wrote a table with 0 changes - nothing to gain on gfx1100. The tuner also needs its binary copied next to the engine on Windows (autotune.py looks for a literal 'tune_decode'). - Niko1221#1151 SIMT scorer: ADOPTED (STRATA_SELECT_SIMT=1) - +3.9% at 16K and +4.5% at 4K on top of the WMMA select, measured at matched budgets after the leak below was cleared. - Niko1221#1324 LRU: off. 2 GiB is a wash (16K -0.8%, budget -1.4 GiB); 6 GiB costs 14% and crushes the budget to 4.9 GiB. - Niko1221#1167 rocBLAS table: inert on gfx1100 - prompt_f16() is gfx103x-only ('on gfx103x it is ~2x faster'). - A5 hipBLASLt rows: deferred (our 45-row table already beats upstream's 34; the tuner needs a GPU-exclusive run). B: Niko1221#1838/Niko1221#1803/Niko1221#1876 applied and NEUTRAL (-2..+2% at both sizes, so kept); its conversational test shows no measurable difference either. C: Niko1221#1913 does not reproduce here - per-entry host cost is flat at 0.77-0.86 ms (their box jumps 3-5x). Niko1221#1903's placement axis answered: --expert-cache-per-layer costs 3%. Issue Niko1221#1910 does not apply: the interleaved 4K pairs put 0.1.42 +2.1% AHEAD of 0.1.41 (923.6/923.5/962.2 vs 958.7/957.4/953.2). Niko1221#1906 answered: reuse works, up to 96% of a turn's prompt reused (1286/1341). Bug found and fixed: stop-server.bat killed a fixed list of old binary names, so newer A/B engines leaked - one held 11.4 GiB and crushed the next measurement's RAM budget to 0.0 GiB. It now kills any strata*.exe (matching tasklist) plus tune_decode; that leak is what made the first group-B/SIMT/per-layer readings unusable.
|
Thanks. We have no Windows machine to test the file tier on, so we cannot gate it. If you can post decode medians on a second Windows PC we will look again. |
|
I cannot supply the second-PC confirmation from the hardware used for my results above: those measurements all come from the same Ryzen 9 7940HS / RTX 4070 Laptop Windows machine. Repeating on this laptop would not satisfy your independent second-machine gate. The claim from my existing runs remains limited to elasticity under gradual pressure; sustained decode benefit is still unqualified. I have no new decode medians to add and will keep the second-Windows-PC validation explicitly outstanding. |
4 participants
AI-written. This PR was prepared by an AI (Claude Opus 5.5, via Claude Code) at the submitter's request: design, code and every measurement were done by the AI on the submitter's machine, and the submitter reviewed it before opening.
Depends on #1323 (the #833 port is this branch's first commit); review that one first.
Update 2026-10-07: rebased onto 0.1.40.2 (
e8ca9af) together with #1323. New commit: on Windows the elastic watcher keeps commit available as well as RAM (every LRU buffer is committed memory; here the commit charge reached 112-120 of 121 GiB while serving with 8-9 GB of RAM free). The release's I/O prefetch flag (stage_pf_) is cleared when the LRU frees a buffer. Measured on 0.1.40.2 with the production config: decode 72.9 tok/s against 71.4 for the 0.1.40.1-based build, prompts unchanged.What
--resident-budget-gibfills the RAM once from the global expert profile, so a conversation on another topic keeps reading the drive for experts that are cold in the profile but hot for it.--resident-lru-gib L: L GiB of the budget hold the experts the decode reads from the files, least recently used out first (the unbuffered reads' stage pool). The prompt path copies from it but never adds to it or refreshes it: a prompt chunk sweeps every cold expert once and would push out what the decode reuses.STRATA_LRU_KEEP_FREE_GIB=G: the LRU is elastic. A watcher frees the coldest buffers when the available RAM (on Windows also the available commit) falls below G (back to the OS at once, never paged) and grows the pool back a GiB at a time when RAM is free again. It never shrinks below 1 GiB.STRATA_TIER_TRACE=<file>: one line per expert served outside the GPU caches. It is how the policy was chosen: replaying a traced session against same-size policies, the decode-fed LRU cut the decode's drive reads by ~45%.The serve log reports the LRU's hits and, when elastic, what it holds and what it gave back. Without the flag and the variable nothing changes.
Measured
Full tables and method in
bench/results/2026-10-07-windows-ram-tier-lru/. IQ3_S, 2x RTX 5060 Ti, i7-12700K, 64 GB:Tried and not in this PR: offering idle buffers to Windows (
OfferVirtualMemory) halved decode speed; a routing lookahead reading the predicted blobs tripled the bytes read.Not tested
The elastic watcher on Linux (it uses
MemAvailable; only Windows was measured). HIP.🤖 Generated with Claude Code