Skip to content

conversation cache: a re-parked conversation reserves at most 1/8 of each K/V buffer instead of up to 16 MiB, so fewer get evicted - #1163

Closed
BlueKingMuch wants to merge 1 commit into
Niko1221:mainfrom
BlueKingMuch:cache-reserve
Closed

BlueKingMuch wants to merge 1 commit into
Niko1221:mainfrom
BlueKingMuch:cache-reserve

Conversation

@BlueKingMuch

@BlueKingMuch BlueKingMuch commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

A conversation that was restored and is parked again keeps its unchanged K/V pages and appends the new ones (92a3d5f, from #189). The appended segment also reserves room for later turns, growing geometrically up to 16 MiB, and it does that in every K/V buffer of the image: k, v, their scales and the indexer rows of each QSA layer, and the draft layer's. With Qwen3.8-Flash-Next (IQ3_S, int8 K/V) that came to 0.4 to 0.5 GiB of unused capacity per conversation parked again, about half the image on top. The cache counts capacity against --conversation-cache-mib, so the reserve decides which conversations get evicted.

This bounds the reserve by an eighth of what the buffer held before the append (at least 64 KiB, and at most 16 MiB as before). Turns still fill the reserve instead of adding a restore transfer each: the 4096-append test keeps its bound, and 256 turns of 1 MiB onto a 16 MiB buffer take 28 segments instead of 17.

The docs said parking falls back to a full capture when the reserve would evict another conversation. 6cd29d9 removed that check, since a fresh capture never evicts less - which did not hold with this reserve: below, the images parked again are about 1.5x the fresh size. With the reserve bounded, the two differ by at most an eighth of the K/V, and the docs now say what the reserve is instead.

Measured

On 0.1.40.1, this commit against v0.1.40.1. RTX 4080 SUPER 32 GB, Ryzen 7 5800X3D, 96 GB DDR4, Windows 11, IQ3_S, 262K context, --conversation-cache-mib 2816 --conversation-cache-slots 3, a fixed expert placement (--adapt-every 1000000 --expert-cache 8031), greedy, 64-token answers. A main agent P (30,000 tokens) and two subagents S1 and S2 with the same system prompt (15,000 each) take turns for two rounds, and every return adds 2,300 tokens: P1 S1 P2 S2 S1b S2b P3 S1c S2c P4.

Parked images from the engine log, against (1 + checkpoints) x 0.11 GiB + 14.9 KiB per token:

conversation parked again v0.1.40.1 this PR computed
P at S2 (32,429 tokens, 3 checkpoints) 1.38 GiB 0.92 GiB 0.90 GiB
S2 at P3 (17,429 tokens, 3 checkpoints) 1.08 GiB 0.69 GiB 0.69 GiB
P at S1c (34,791 tokens, 4 checkpoints) 1.49 GiB 1.09 GiB 1.04 GiB

What that does to the turns (prompt tokens read, time to first token):

v0.1.40.1 this PR
S1c, subagent 1 in the second round 17,242, 5.5 s (S1 evicted at P3) 2,300, 2.9 s
S2c, subagent 2 in the second round 17,242, 5.4 s 2,369, 2.9 s
all ten requests 116,147 read, 49.9 s 86,332 read, 44.6 s
evicted 3 1

(S2c in v0.1.40.1 loses S2 another way: S1c, reading from the shared system prompt, moves S2's checkpoint into the session and overwrites S2. That is the next PR: #1164)

With the cache on and the fixed placement, the main agent's four answers equal a run without subagents, in both runs. conversation_cache_test passes on Windows (its new checks fail on v0.1.40.1 without the change), as do conversation_snapshot_test (with CUDA) and conversation_file_test.

cc @jeremiahritchey - the reserve came with your incremental capture; I kept its shape and only bounded it.

…th of each K/V buffer in reserve (was up to 16 MiB per buffer, 0.4-0.5 GB per conversation)

A conversation that was restored and is parked again keeps its unchanged K/V
pages and appends the new ones (92a3d5f). The appended segment reserved room
for later turns, growing geometrically up to 16 MiB - per buffer, and a parked
image has one per K/V layer and part (k, v, their scales, the indexer rows, the
draft layer). With Qwen3.8-Flash-Next (IQ3_S, int8 K/V) a 32K-token main agent
parked again took 1.49 GB instead of 0.97 GB, a 17K-token subagent 1.16 GB
instead of 0.74 GB. The cache counts that capacity against its budget, so with
2816 MiB the reserve decides which conversations are evicted.

The reserve is now at most an eighth of what the buffer held before the append
(at least 64 KiB, and at most 16 MiB as before). Small turns still fill the
reserve instead of adding a transfer each: the 4096-append test keeps its
segment bound, and 256 turns of 1 MiB onto a 16 MiB buffer take 28 segments
(17 before).

The docs said parking falls back to a full capture when the reserve would evict
another conversation; that check went in 6cd29d9. With the reserve bounded the
two captures differ by at most an eighth of the K/V, and the docs now say what
the reserve is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKZxC8sUotEzdWXws2FTUC
@BlueKingMuch
BlueKingMuch marked this pull request as ready for review October 6, 2026 13:42
fragtion added a commit to fragtion/StrataPool that referenced this pull request Oct 6, 2026
…1#1101, Niko1221#1166, Niko1221#1163 (open, on 0.1.40.1)

- Niko1221#1102 / Niko1221#1201: the batch host paths skip the per-layer service per window (ar_on), not per process:
  with a prompt loan or swap in flight an all-resident stage takes the doorbell graph and stalled
  (the watchdog's "no progress" in several chats).
- Niko1221#1185: a window graph that finds no VRAM frees older batch slot layouts' graphs instead of failing.
- Niko1221#1101: prefill Stager threads sleep instead of yield-spinning (they competed with the CPU pool).
- Niko1221#1166: --host-core sibling / STRATA_HOST_CORE=sibling: the host thread on the first core's other
  hardware thread, interrupts left alone (hybrid Intel CPUs, both of our PCs).  Opt-in.
- Niko1221#1163: a conversation parked again keeps at most an eighth of each K/V buffer in reserve (was up
  to 16 MiB per buffer, 0.4-0.5 GB per conversation).
Skipped: Niko1221#1107 (conflicts with the staged walk's stream-ahead; no measurement given), Niko1221#1181 (decode
unchanged in its own measurement).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD
@Niko1221

Copy link
Copy Markdown
Owner

Thank you, this is in for 0.1.45 with your authorship. The unit tests pass on CUDA sm_86 and sm_60. At 12K tokens the snapshots are the same size as before (the reserve only exceeds 1/8 once a K/V buffer is over 16 MiB), so the saving shows from about 30K tokens.

@Niko1221

Copy link
Copy Markdown
Owner

Thanks for the pull request.

Your change is already in main for 0.2.0-beta, with your credit, so there is nothing left for you to do here. We're closing it because we're resetting the tracker for 0.2.0-beta: there are more than 1,300 open issues and pull requests, and we're closing everything that isn't being worked on right now.

If part of your work did not make it in and you want it considered, open a new pull request on top of main that follows CONTRIBUTING.md, or comment /reopen after fixing the title area prefix and the template.

@Niko1221 Niko1221 closed this Oct 19, 2026
DarthCY-K pushed a commit to DarthCY-K/Strata that referenced this pull request Oct 20, 2026
…h "parallel", the fork's changes and the measured costs

docs/BATCHING.md "This fork" and docs/NVFP4.md "Several dialogs at once": the switch cost of dialogs that take turns
without and with --conversation-cache-mib (8K / 32K / 100K), why "parallel" was unusable on the 262K config (14.6 GiB
of slots), --batch-max-context in the options table, the control-vector window fix, the conversation-cache PRs taken
(Niko1221#1503, Niko1221#1163, Niko1221#1164, Niko1221#1959, Xiaomi2008's STRATA_PARK_REWIND_MAX), the checks, one dialog's decode per config, and
the recommended setting (the conversation cache; "parallel": 2 with 64K slots when requests really overlap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
DarthCY-K pushed a commit to DarthCY-K/Strata that referenced this pull request Oct 20, 2026
…gs, the rope warning, the final build's checks (81049f34)

What's new and "Upstream 0.1.43.1" add the EOS-literal fix ("stop_token_text"), the parallel-dialog fixes (upstream
PRs Niko1221#1503 Niko1221#1163 Niko1221#1164 Niko1221#1959; link to "Several dialogs at once") and the --max-context warning; the checks name the
release build 81049f34: 2K / 32K logits and tokens = 0.1.42-nvfp4.2's engine, the serve KL texts byte-identical,
ctest 116/119, serve 849, tools 821.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants