Repository navigation
conversation cache: a re-parked conversation reserves at most 1/8 of each K/V buffer instead of up to 16 MiB, so fewer get evicted - #1163
Conversation
…th of each K/V buffer in reserve (was up to 16 MiB per buffer, 0.4-0.5 GB per conversation) A conversation that was restored and is parked again keeps its unchanged K/V pages and appends the new ones (92a3d5f). The appended segment reserved room for later turns, growing geometrically up to 16 MiB - per buffer, and a parked image has one per K/V layer and part (k, v, their scales, the indexer rows, the draft layer). With Qwen3.8-Flash-Next (IQ3_S, int8 K/V) a 32K-token main agent parked again took 1.49 GB instead of 0.97 GB, a 17K-token subagent 1.16 GB instead of 0.74 GB. The cache counts that capacity against its budget, so with 2816 MiB the reserve decides which conversations are evicted. The reserve is now at most an eighth of what the buffer held before the append (at least 64 KiB, and at most 16 MiB as before). Small turns still fill the reserve instead of adding a transfer each: the 4096-append test keeps its segment bound, and 256 turns of 1 MiB onto a 16 MiB buffer take 28 segments (17 before). The docs said parking falls back to a full capture when the reserve would evict another conversation; that check went in 6cd29d9. With the reserve bounded the two captures differ by at most an eighth of the K/V, and the docs now say what the reserve is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKZxC8sUotEzdWXws2FTUC
…1#1101, Niko1221#1166, Niko1221#1163 (open, on 0.1.40.1) - Niko1221#1102 / Niko1221#1201: the batch host paths skip the per-layer service per window (ar_on), not per process: with a prompt loan or swap in flight an all-resident stage takes the doorbell graph and stalled (the watchdog's "no progress" in several chats). - Niko1221#1185: a window graph that finds no VRAM frees older batch slot layouts' graphs instead of failing. - Niko1221#1101: prefill Stager threads sleep instead of yield-spinning (they competed with the CPU pool). - Niko1221#1166: --host-core sibling / STRATA_HOST_CORE=sibling: the host thread on the first core's other hardware thread, interrupts left alone (hybrid Intel CPUs, both of our PCs). Opt-in. - Niko1221#1163: a conversation parked again keeps at most an eighth of each K/V buffer in reserve (was up to 16 MiB per buffer, 0.4-0.5 GB per conversation). Skipped: Niko1221#1107 (conflicts with the staged walk's stream-ahead; no measurement given), Niko1221#1181 (decode unchanged in its own measurement). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD
|
Thank you, this is in for 0.1.45 with your authorship. The unit tests pass on CUDA sm_86 and sm_60. At 12K tokens the snapshots are the same size as before (the reserve only exceeds 1/8 once a K/V buffer is over 16 MiB), so the saving shows from about 30K tokens. |
|
Thanks for the pull request. Your change is already in If part of your work did not make it in and you want it considered, open a new pull request on top of |
…h "parallel", the fork's changes and the measured costs docs/BATCHING.md "This fork" and docs/NVFP4.md "Several dialogs at once": the switch cost of dialogs that take turns without and with --conversation-cache-mib (8K / 32K / 100K), why "parallel" was unusable on the 262K config (14.6 GiB of slots), --batch-max-context in the options table, the control-vector window fix, the conversation-cache PRs taken (Niko1221#1503, Niko1221#1163, Niko1221#1164, Niko1221#1959, Xiaomi2008's STRATA_PARK_REWIND_MAX), the checks, one dialog's decode per config, and the recommended setting (the conversation cache; "parallel": 2 with 64K slots when requests really overlap). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
…gs, the rope warning, the final build's checks (81049f34)
What's new and "Upstream 0.1.43.1" add the EOS-literal fix ("stop_token_text"), the parallel-dialog fixes (upstream
PRs Niko1221#1503 Niko1221#1163 Niko1221#1164 Niko1221#1959; link to "Several dialogs at once") and the --max-context warning; the checks name the
release build 81049f34: 2K / 32K logits and tokens = 0.1.42-nvfp4.2's engine, the serve KL texts byte-identical,
ctest 116/119, serve 849, tools 821.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
A conversation that was restored and is parked again keeps its unchanged K/V pages and appends the new ones (92a3d5f, from #189). The appended segment also reserves room for later turns, growing geometrically up to 16 MiB, and it does that in every K/V buffer of the image: k, v, their scales and the indexer rows of each QSA layer, and the draft layer's. With Qwen3.8-Flash-Next (IQ3_S, int8 K/V) that came to 0.4 to 0.5 GiB of unused capacity per conversation parked again, about half the image on top. The cache counts capacity against
--conversation-cache-mib, so the reserve decides which conversations get evicted.This bounds the reserve by an eighth of what the buffer held before the append (at least 64 KiB, and at most 16 MiB as before). Turns still fill the reserve instead of adding a restore transfer each: the 4096-append test keeps its bound, and 256 turns of 1 MiB onto a 16 MiB buffer take 28 segments instead of 17.
The docs said parking falls back to a full capture when the reserve would evict another conversation. 6cd29d9 removed that check, since a fresh capture never evicts less - which did not hold with this reserve: below, the images parked again are about 1.5x the fresh size. With the reserve bounded, the two differ by at most an eighth of the K/V, and the docs now say what the reserve is instead.
Measured
On 0.1.40.1, this commit against v0.1.40.1. RTX 4080 SUPER 32 GB, Ryzen 7 5800X3D, 96 GB DDR4, Windows 11, IQ3_S, 262K context,
--conversation-cache-mib 2816 --conversation-cache-slots 3, a fixed expert placement (--adapt-every 1000000 --expert-cache 8031), greedy, 64-token answers. A main agent P (30,000 tokens) and two subagents S1 and S2 with the same system prompt (15,000 each) take turns for two rounds, and every return adds 2,300 tokens: P1 S1 P2 S2 S1b S2b P3 S1c S2c P4.Parked images from the engine log, against (1 + checkpoints) x 0.11 GiB + 14.9 KiB per token:
What that does to the turns (prompt tokens read, time to first token):
(S2c in v0.1.40.1 loses S2 another way: S1c, reading from the shared system prompt, moves S2's checkpoint into the session and overwrites S2. That is the next PR: #1164)
With the cache on and the fixed placement, the main agent's four answers equal a run without subagents, in both runs.
conversation_cache_testpasses on Windows (its new checks fail on v0.1.40.1 without the change), as doconversation_snapshot_test(with CUDA) andconversation_file_test.cc @jeremiahritchey - the reserve came with your incremental capture; I kept its shape and only bounded it.