Repository navigation
verify: a window graph that finds no VRAM frees older batch slot layouts' graphs instead of failing (#997) - #1185
Closed
ischencheng wants to merge 1 commit into
Closed
ischencheng wants to merge 1 commit into
ischencheng wants to merge 1 commit into
Conversation
…uts' graphs instead of failing (Niko1221#997) With --batch, each set of slots that decodes together gets its own captured window and commit graph pair, and without --batch-mtp's limit they are kept for good. On an L40S a pair takes 20-30 MiB, 8 slots have 255 possible sets, and the expert cache sizes itself to leave only --vram-reserve-mib free. The next cudaGraphInstantiate then fails ("verify: batch instantiate: out of memory", or "verify: instantiate" for a one-request window captured on first use) and the engine exits. An instantiate that is out of memory now frees the least recently used batch layouts' graphs (never the one being captured) and tries again; a freed layout is captured again when it comes back. While the graphs fit, nothing is freed. --batch-mtp's limit uses the same eviction. A failed batch commit instantiate now names its error instead of "batch commit capture: no error". Measured on an L40S (46068 MiB), IQ2_XS, "parallel": 8, 12 clients with random short prompts for 7 minutes, with 12000 MiB held by another process so the cache leaves 440 MiB free: main restarted the engine 8 times and answered 45 of 142 requests; this change answered 623 of 623 with no restart. With nothing held (1334 MiB free) both answered every request without evicting, and fixed batches of 8 gave the same 24 replies on main and here.
fragtion
added a commit
to fragtion/StrataPool
that referenced
this pull request
Oct 6, 2026
…1#1101, Niko1221#1166, Niko1221#1163 (open, on 0.1.40.1) - Niko1221#1102 / Niko1221#1201: the batch host paths skip the per-layer service per window (ar_on), not per process: with a prompt loan or swap in flight an all-resident stage takes the doorbell graph and stalled (the watchdog's "no progress" in several chats). - Niko1221#1185: a window graph that finds no VRAM frees older batch slot layouts' graphs instead of failing. - Niko1221#1101: prefill Stager threads sleep instead of yield-spinning (they competed with the CPU pool). - Niko1221#1166: --host-core sibling / STRATA_HOST_CORE=sibling: the host thread on the first core's other hardware thread, interrupts left alone (hybrid Intel CPUs, both of our PCs). Opt-in. - Niko1221#1163: a conversation parked again keeps at most an eighth of each K/V buffer in reserve (was up to 16 MiB per buffer, 0.4-0.5 GB per conversation). Skipped: Niko1221#1107 (conflicts with the staged walk's stream-ahead; no measurement given), Niko1221#1181 (decode unchanged in its own measurement). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD
Niko1221
pushed a commit
that referenced
this pull request
Oct 7, 2026
…uts' graphs instead of failing (#1185) Every layout of active slots keeps its own graph pair, and with the expert cache sized to the reserve the next window graph can find no VRAM left; the engine then exited (#997, `parallel` 4 to 8). cudaGraphInstantiate that runs out of memory now evicts the least recently used batch layouts, other than the one being captured, until it fits; they are captured again when needed. The start prints how many were freed. Nothing changes while everything fits.
Owner
|
Thank you, ischencheng. The graph eviction for batch layouts is in; it also needed cudaErrorMemoryAllocation mapped for HIP. It is in 0.1.40.2 with your authorship kept. |
fragtion
added a commit
to fragtion/StrataPool
that referenced
this pull request
Oct 7, 2026
Upstream now carries our picks of Niko1221#851 (AVX2 Q8_K quantizer), Niko1221#789 (expert uploads beside the shared expert), Niko1221#813 (--ple-io ram on Windows), Niko1221#1185 (window graphs free older batch layouts), Niko1221#1201/Niko1221#1118 (doorbell per window), Niko1221#1101/Niko1221#1057 (Stager sleep: now the default on Linux only, STRATA_STAGER_SLEEP=1 forces it) and Niko1221#1063 (the shared draft head's type); their versions are kept. Conflicts resolved: - prefill: upstream's own CPU share (STRATA_PREFILL_CPU_SHARE, opt-in) sits beside CPU assist. Its setter is set_cpu_share_pool; when it is asked for, it replaces CPU assist for the chunks it covers, otherwise CPU assist runs as before (on by default). The debug NaN check skips both kinds of CPU rows. - conversation cache: a pool-borrowed entry and upstream's pinned shared prefix (pin=N) are both kept from eviction; the borrow (Niko1221#1164) and pin=N's sibling path do not park at once. - generate: pool batch groups launch only up to their last active slot (Niko1221#1249's G.S), also on the workers; --adapt-async's poll between verdicts runs beside the pool's verified-path-first loop. - server: upstream's chunked bodies (Niko1221#893) and the pool's read-once body are one helper; do_POST's BadBody wrapper around the pool's routed-request check. - native dense: the pool's served_bytes beside upstream's weight_bytes_for (Niko1221#1238). - expert source: the commit-limit retry (Niko1221#1064) beside upstream's paced page-locking (Niko1221#1250). - setup: no ready-made engine (StrataPool compiles its own); REPO names StrataPool for the SHA-256 lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD
0xrjman
pushed a commit
to 0xrjman/Strata
that referenced
this pull request
Oct 7, 2026
…uts' graphs instead of failing (Niko1221#1185) Every layout of active slots keeps its own graph pair, and with the expert cache sized to the reserve the next window graph can find no VRAM left; the engine then exited (Niko1221#997, `parallel` 4 to 8). cudaGraphInstantiate that runs out of memory now evicts the least recently used batch layouts, other than the one being captured, until it fits; they are captured again when needed. The start prints how many were freed. Nothing changes while everything fits.
MistVVK
added a commit
to MistVVK/XeStrata
that referenced
this pull request
Oct 8, 2026
upstream の次を移す: - 536db01: エキスパートのプロファイルのヘッダーの版が 1 でなければ、 両方の版とファイルを名指しして断る（expert_profile_version_test を CTest に足す、GPU を使わない） - ccfdc2e: 密な重みの構成の確認を、分割した GGUF の最初のシャードでだけ行う - e8b53d5: --expert-cache を明示したのにカードが埋まっているとき、 そう言い、どうすればよいかを書く（Niko1221#781、Niko1221#831） - e155cf8: 大きなプロンプトの塊が貸すキャッシュのスロットの エキスパートが RAM に入りきらないとき、起動時に警告する（Niko1221#669、Niko1221#765）。 XeStrata ではここに来る時点で常駐の準備は済んでいるので、 upstream の resident_ok の条件は要らない - 41e21f5: --spec と --spec-min-p を --help に書く 入れないもの: 6f7f680（Niko1221#871）と 53ab5f7 は XeStrata では起きない （窓ごとにホストの常駐表を確かめてグラフを選ぶ、コピーは完了を待つ）。 ebd9c81 と 832b2e4 は CUDA の API の話、73d39830 と b540e99 は XeStrata が外した融合 MoE の経路の話。3bcfd8f3（Niko1221#1185）は並列の要求の まとまりで移す。 CTest は llvm7・icpx で 55 件通る。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
"parallel": N, every set of slots that decodes together gets its own window and commit graph pair (exec_bm_/commit_bm_, keyed by the row layout), and they stay for the life of the engine unless--batch-mtpsets its limit of 64. On an L40S one pair takes 20-30 MiB (more rows, more MiB), and 8 slots have 255 possible sets. The expert cache sizes itself to leave only--vram-reserve-mibfree, so after a dozen or a few dozen layouts the nextcudaGraphInstantiatefails and the engine exits withverify: batch instantiate: out of memory. That's what Jackwwg83 found in #997 (379 MiB free after load, fails when 7-8 slots decode together,--vram-reserve-mib 1700avoids it). The one-request windows captured on first use (2-5 tokens) hit the same wall asverify: instantiate: out of memory.Now an instantiate that runs out of VRAM frees the least recently used batch layouts' graphs (never the layout being captured) and tries again. A freed layout is captured again when it comes back. While everything fits nothing is freed, so the default path doesn't change. The
--batch-mtplimit goes through the same eviction helper now. I also fixed the batch commit graph's message, which saidverify: batch commit capture: no errorwhen its instantiate failed.Measured on an L40S (46068 MiB, sm_89, Linux), IQ2_XS, setup's config plus
"parallel": 8, 12 clients sending random short prompts with randommax_tokensfor 7 minutes. For the low-free-VRAM case in the issue, another process held 12000 MiB before the engine started, so the cache sized itself to ~440 MiB free:So the plain 48 GB setup was a few more layouts away from the same crash. Three fixed waves of 8 concurrent requests at temperature 0 gave the same 24 replies byte for byte on main (two runs) and on this PR. The per-layout MiB come from a debug-only log line (free VRAM after each capture) that isn't in this PR.
Not tested: Windows, HIP (
cudaErrorMemoryAllocationmaps tohipErrorOutOfMemorythere),--batchon a layer split, and--batch-mtp(its eviction only moved into the helper). It also doesn't touch #1102, which applies cleanly on top.Fixes the batch instantiate crash from Jackwwg83's comment in #997. The original report there (RTX 3090, 360 MiB free) may be the same thing, but I can't tell without its ERR line.