[xrip]

What

Opt-in --prompt-cache-spread: when the conversation checkpoints are over --prompt-cache, the one that leaves is
chosen so that the kept ones stay spread back from the conversation's end, instead of the least recently used. Without
the flag nothing changes.

Why

The engine keeps 6 checkpoints and rotates them by least recent use. Every turn adds one at its start, so after a few
short turns all of them sit within the last 1-2K tokens (plus the pinned system-prompt root). That is fine while a
client only appends. Agent clients do not always: they trim an old tool output, drop a step, or rewrite an earlier turn
a few thousand tokens back. The new prompt then matches none of the recent checkpoints, the resume falls back to the
system prompt, and the whole conversation is read again.

Found with the pi coding agent (and the pi-blackhole extensions) on a long agent session, from the engine log:

conversation held	new prompt	same as held for	resumed at	read again
210,952	210,528	204,349 (6K before the end)	9,525	201,003 tokens, 100 s
206,572	205,253	163,508 (42K before the end)	0	205,253 tokens, 101 s

In both, a checkpoint a few thousand tokens before the edit would have saved most of the read; least recent use had
rotated those out.

How

• conv_cache::spread_victim (include/strata/program/conv_cache.hpp): over the chain (ascending), the victim is the
  checkpoint whose removal leaves the smallest gap for its distance from the newest one,
  (len[i+1] - len[i-1]) / (len[newest] - len[i-1]). Dense recent points go first, so the kept ones thin out
  geometrically towards the root. Never the root, the newest or a pinned (strata_prefix) one; the
  --prompt-cache-tail checkpoint first, as in the existing policy; with no candidate, the existing policy.
• generate.cpp: the flag, and the choice between the two policies at the one place a checkpoint is dropped.
• The resume is unchanged: a checkpoint is used only when the prompt starts with exactly its tokens and pictures.
• Same number of checkpoints, no more RAM.

In a simulation of 200 turns of 1,000 tokens after a 9,525-token system prompt (conv_cache_test):

6 checkpoints, kept (tokens before the end)	edit 6K back	edit 42K back
least recent use: 0, 1K, 2K, 3K, 4K and the root	reads 200K	reads 200K
spread: 0, 1K, 3K, 8K, 43K, 200K	reads 8K	reads 43K

Measured

2x CMP 50HX (Turing sm_75, --layer-split auto), 2x Xeon E5-2670 v2, 64 GB RAM, Qwen3.8-Flash-Next IQ3_XXS, CUDA 13.3,
GCC 14. An 82,593-token conversation (a 65K-token document, then 20 short turns sent 2 at a time, each request
continuing the previous one), then the same conversation with a turn ~6 steps back rewritten and a question; greedy,
thinking off:

	resumed at	time
--prompt-cache-spread	73,610	10.2 s
--prompt-cache 0 (read in full)	0	40.0 s

With the expert placement fixed (--adapt-swaps 0 --pcie-frac 0.17) the answer after the resume was the same text as
the full read. Without fixing it, two full reads already differ in wording (the placement moves with timing), so that
comparison needs the fixed placement.

Tests: conv_cache_test passes, with the new cases (the simulation above against least recent use; every candidate
pinned and one slot fall back to the existing policy). Not run on Windows or HIP.

Relation to #1937

Found while measuring the disk tier of #1937 on the same agent sessions, with a diagnostic that logs how many tokens a
request read again although a held conversation had them. It is independent of #1937: any long conversation whose
client edits its history benefits, with or without parked conversations.

How to try

Add "--prompt-cache-spread" to the config's args.

🤖 Generated with Claude Code

[Niko1221]

Thank you, this is in for 0.1.45 as an opt-in with your authorship (docs included). The conv_cache_test cases pass on CUDA sm_86, sm_60 and HIP. Our engine run did not reach the case your flag changes: with appended turns and an edit 6 turns back both arms resumed at the root, so we have no engine measurement of our own; your 10.2 s against 40 s stands as your number.

[xrip]

@Niko1221 Thank you for taking it in, docs and all!

[Niko1221]

Thanks for the pull request.

Your change is already in main for 0.2.0-beta, with your credit, so there is nothing left for you to do here. We're closing it because we're resetting the tracker for 0.2.0-beta: there are more than 1,300 open issues and pull requests, and we're closing everything that isn't being worked on right now.

If part of your work did not make it in and you want it considered, open a new pull request on top of main that follows CONTRIBUTING.md, or comment /reopen after fixing the title area prefix and the template.