[routhjim]

Stacked on #1271 (spill directory) and #1269 (streaming restore). The branch is #1271 (at 4af468f, which brings in v0.1.41) with #1269 (at d258f18) merged in. Only the last eight commits are new:

• dd146f4: the mode;
• b189048: tests and docs;
• ff30368: the cold-page-cache numbers in the docs;
• 2987ac4, a040357, 9eac670: persistent prefix entries (below), with tests and docs.
• 83ad1c5, 155dd84: from @ed-hch's CUDA test: a refused disk read keeps its file, and the monitor card shows the disk tier.

I will rebase once those two land, or fold this into #1271 if its author prefers.

#1271 keeps the conversations the RAM cache evicts on disk. A conversation reaches disk only through a RAM eviction (make_room), so the RAM budget still decides what is kept. On a machine where the expert cache takes nearly all memory (Strix Halo: 72 GiB of experts in 128 GB), the conversation cache needs that budget most and can afford it least.

--conversation-cache-disk-only, with --conversation-cache-spill-dir, keeps the conversation cache on disk only:

• When a request switches to another conversation, the outgoing one is written to the directory the way a session SAVE writes it. The running state and the deepest checkpoint are copied; the K/V streams from its pools into the file (conversation_snapshot_sources + the streaming session_file_write). No host image is made.
• When a conversation comes back, it is read the way [Engine] restore a session file without holding its K/V in RAM #1269's RESTORE reads it. The read pass checks the whole file before anything on the GPU changes; a bad file is dropped, and the prompt is read as usual. Then the apply pass writes the pools 16 MiB at a time, with RESTORE's fail-stop after the first block.
• --conversation-cache-mib is not used. --conversation-cache-disk-mib is the only limit, so no conversation is refused for its size. A read or save still checks free RAM for its transient buffers (one checkpoint's running state and two 16 MiB blocks) against --conversation-cache-min-free-mib. When that refuses, the file is kept for a later turn.
• v0.1.41's RAM-admission eviction (serve: a full conversation cache drops the snapshot at the physical-RAM gate instead of evicting to pass it (270k tokens re-read, 96 s, 32 times) #1347) is on the RAM parking path; disk-only mode never reaches it.
• The new file is written before this conversation's older copies are dropped. drop_superseded gains a keep path, because its rule alone would drop the new copy too. A failed write loses nothing.
• The agent's next turn of the live conversation writes nothing. It resumes at the newest turn checkpoint, and the outgoing tail is only the reply the client sends again.
• A clean shutdown saves the live conversation, flushed. Switch saves skip the flushes: a file torn by a crash fails the payload hash when it is read and is dropped.
• Single GPU, without --batch or --peer-device; otherwise the flag says so and is off.

Where it helps. A coding agent run as several sessions on one local engine: Claude Code through an Anthropic-compatible proxy, or a manager such as Happier that starts sessions from a phone. Each session is a long conversation (20-120K tokens), and they alternate. With one GPU and no RAM to spare, every switch read the whole conversation again. One real Claude Code session of 121K tokens took 181 s to read and 2.0 s to restore from disk. A new session started from a 13K-token preamble that is the same every time, and read all of it.

Prefix entries. When a conversation is saved, its shared opening is also saved on its own as strata-conv-prefix-N, once per distinct opening. The opening is the deepest "strata_prefix" checkpoint on the live path when the request pinned one. Otherwise it is the root by the engine's rule: the first turn boundary at least --prompt-cache-root tokens in. A new conversation that opens the same way resumes from it.

• Prefix entries are never removed as superseded. The disk budget evicts every ordinary conversation before one of them. A restart keeps them as prefixes, and has_prefix dedupes them.
• The sidecar format is unchanged; the kind is in the file name.
• In disk-only mode a request that resumes at a pinned prefix is another conversation, so the one it leaves is saved. The RAM cache's skip ("N queries cost one prefix") would drop it.

Measured with real Claude Code sessions. The client marks "strata_prefix" after Claude Code's leading <system-reminder> blocks, so the system prompt, tools and reminders form the shared part.

first turn of a new session	prompt read	reused
without prefix entries	15.7 s	4,350 tokens
with them	4.7 s	13,434 of 15,924
with them, after an engine restart	5.1 s	13,434 (the 420 MiB entry restored in 326 ms)

The rest of each first turn is the user's own message and a per-session block, so it has to be read.

Why this matters beyond the budget. With #1271 as it is, a conversation larger than the free RAM budget is neither parked nor spilled:

skip parking (snapshot 1292 MiB exceeds available budget)

If its disk copy was restored a turn earlier, that copy was already removed as superseded, so the conversation is gone, and its next turn reads 15,954 tokens from the start (18.9 s, in the run below). The long conversations are the ones that gain the most from a disk tier. Disk-only mode never asks the RAM budget.

Measured on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151, 128 GB, Linux 7.2, ROCm 7.14.1, the iGPU alone).

• Build: per docs/STRIX_HALO.md.
• Model and flags: Qwen3.8-Flash-Next with K-quant experts, --mmap-experts --expert-cache auto --spec 4 --mtp mtp/rt --lookup-chain 3 --kv int8 --max-context 65536. Spill directory on the internal NVMe.
• Workload: three agent-style conversations taking turns (A1 B1 C1 A2 ...). Each is a 9-12K-token first prompt, then turns that each add ~1.1K tokens of source text. Greedy, 64 output tokens. 3 rounds, an engine restart, then 2 more rounds.
• Reuse and prompt time come from the /metrics totals around each request.

	prompt read, later turns (12)	turns resumed
no conversation cache	11.4-22.3 s, nothing reused	0 of 12
#1271, --conversation-cache-mib 2048 --conversation-cache-slots 1	2.4-3.0 s, 88-94% reused	11 of 12 (the one above lost)
this PR, --conversation-cache-disk-only	2.3-3.6 s, 88-94% reused	12 of 12

• A switch wrote 360-443 MiB in 181-196 ms. A return read it back in 291-351 ms, of which the read pass was ~150 ms.
• Cold disk: the same run again, with every spill file pushed out of the page cache before each request (fsync, then posix_fadvise(DONTNEED); fincore showed 0 bytes resident each time). Restores took 275-361 ms (read pass 128-162 ms), and all 12 later turns resumed with the same 88-94% reuse.
• The files are smaller than serve: --conversation-cache-spill-dir keeps evicted conversations on disk across restarts #1271's parked images (1.0-1.4 GB) because only the deepest checkpoint is saved, as SAVE already does. Three conversations took 1.3 GB on disk.
• Process memory (RssAnon, sampled every 0.1 s) peaked 0.6 GiB above a run without the cache during a switch, and ended 0.15 GiB above it. No conversation stays in RAM.
• Greedy replies matched the run without the cache for 13 of 15 turns. The other two (the first turn after the restart, and the turn after it) are identical to serve: --conversation-cache-spill-dir keeps evicted conversations on disk across restarts #1271's replies for the same turns: a restore and a full read of the prompt round differently.

Tests. conversation_spill_test: 32 new checks, 73 pass. test_monitor: 2 new tests, 12 pass.

• A streamed spill writes byte for byte the file the image spill writes, matches and reads back with the same K/V.
• It refuses a host K/V image or an empty conversation.
• A failing source leaves no file and no index entry.
• drop_superseded spares the keep copy, while the rule alone would drop it.
• Prefix entries: named as prefixes, kept by drop_superseded, evicted last, deduped, and still prefixes after a reopen.

The engine paths were run end to end as above.

Not tested:

• CUDA (no NVIDIA card here) and Windows;
• --kv-grow: the restore calls kvg_ensure as the RAM restore does, but I did not run it with elastic K/V.

Measured with the help of Claude (Anthropic).

🤖 Generated with Claude Code

https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp

[CC-David-CC]

"Not tested: CUDA (no NVIDIA card here) "

put in my testing queue

[CC-David-CC]

routhjim, ANBAL534, pspranger-throw damn guys. leave some for me.

Very nice

[ed-hch]

It works on my CUDA machine. "--conversation-cache-disk-mib",
"32768",
"--conversation-cache-disk-only",

One thing to notice is after remove --conversation-cache-mib flag. the conversation cache section in front end monitor page gone. I ask my agent on Strata fix it. But since I'm just a vibe coder, I'm to ashamed to put is on any PR. I'm more than sure all the contributor can work something better.

Below is the AI summery:
Monitor: the Conversation cache card follows --conversation-cache-disk-only

Problem
The card keyed on conversation_cache_mib > 0. With the RAM cache off (disk-only, #1480) it hid entirely — or showed its RAM bars with zeros — and its note told users to add --conversation-cache-mib, which is the opposite of what disk-only means.

Changes (3 files, kept as two local commits on our dev branch — happy to PR them separately if useful upstream)

Engine (generate.cpp): the INFO status line now reports conversation_cache_disk_only=0/1, so the server knows the mode without re-parsing the run's args.
Server (ConvCacheLog): learns #1480's log vocabulary — spill dir ready (…), disk-saved N tokens (B MiB, …) … disk=B MiB in N files, disk-restored N tokens (…), and refusals (disk hit/save skipped, discard unusable). Counts per event from the current run's log window, same mechanism as the RAM counters: conversations saved/restored, accumulated tokens saved, accumulated MiB written, the directory's current totals, refused count, last event.
Web (app.js): with disk_only, the card swaps the Parked/Slots + RAM bars for: Saved to disk · Restored from disk · Tokens saved to disk · Written to disk (GB) · Spill directory (files · GB) · Refused · Last switch, plus the reused-prompt summary. Rows always render — 0 / Nil instead of hiding — and the note describes the disk tier (or warns that the spill dir failed to open, with the three flags it needs).
Measured (RTX 4000 Ada, Qwen3.8-Flash-Next IQ3_S, ctx 196,608, kv int8, one working day): 49 of 52 requests reused part of their prompt; 10 conversations saved to disk and 8 restored from disk; 838,824 tokens saved, 14.1 GB written; the spill directory holds 5 files · 6.4 GB; 0 refused; last switch saved 57,050 tokens. Per event, saves run 1.8–2.2 GB in 1.3–3.2 s for 105–142K-token conversations; a return restores in ~1.7 s and the next request reuses ~99% of its prompt. Spill files from a previous run survive a restart and still restore. Refused counts the RAM-window refusals ("not enough RAM for the read pass") that the log otherwise buries — 0 here after dropping ctx to 196,608 and the floor to 256 MiB; useful as a tuning signal.

Tests: existing test_monitor suite passes; disk-line counters verified against a replay of a real session log.

[routhjim]

@ed-hch thank you for running it on CUDA, and for the numbers: a working day of 105-142K-token conversations is more than I could test here.

You found two gaps. I put both into this PR (credit to you in the commits):

The monitor card. The engine's INFO line now carries conversation_cache_disk_only=0/1. The server reads the disk-only log lines (saved, restored, tokens saved, MiB written, the directory's files and size, shared prefixes, refusals with the last reason, discarded files). In disk-only mode the card shows those, with a bar for the spill directory against --conversation-cache-disk-mib, instead of the RAM bars, and its note describes the disk tier (or says the directory did not open). The approach follows your summary; the code is new.

The read-pass refusals. Two problems were behind them:

• The RAM preflight of the read pass was sized for --prompt-cache checkpoints (6 by default), capped at the file size. A disk-only save writes one checkpoint, so it asked for more than the file needs: for these ~420 MiB files the old formula comes to ~450 MiB; the new one asked 260 MiB. With your 2 GB files at 196K context the gap should be larger. The read now admits one checkpoint first. A file the RAM cache spilled (the same directory without the flag) can hold more, and is read once more under the engine's own limit before it counts as bad.
• A refusal deleted the file. Any failed read pass went to "discard unusable disk conversation", including "not enough RAM". Now only a file that fails validation (or is gone) is removed; a RAM, storage or I/O refusal keeps it for a later turn. The refusal also says how much it asked for and the floor.

Checked here on Strix Halo:

• Disk-only rounds: the card showed 8 saves, 6 restores and 3 prefixes.
• Restart with a 200 GB floor: this forces every read to be refused. Three restores were refused, and all six files were kept.
• Normal restart: the three conversations were restored from those files.

I can't test CUDA here, so a rerun of your setup would be welcome when you have time.

Measured with the help of Claude (Anthropic).

[ed-hch]

@ed-hch thank you for running it on CUDA, and for the numbers: a working day of 105-142K-token conversations is more than I could test here.

You found two gaps. I put both into this PR (credit to you in the commits):

The monitor card. The engine's INFO line now carries conversation_cache_disk_only=0/1. The server reads the disk-only log lines (saved, restored, tokens saved, MiB written, the directory's files and size, shared prefixes, refusals with the last reason, discarded files). In disk-only mode the card shows those, with a bar for the spill directory against --conversation-cache-disk-mib, instead of the RAM bars, and its note describes the disk tier (or says the directory did not open). The approach follows your summary; the code is new.

The read-pass refusals. Two problems were behind them:

* The RAM preflight of the read pass was sized for `--prompt-cache` checkpoints (6 by default), capped at the file size. A disk-only save writes one checkpoint, so it asked for more than the file needs: for these ~420 MiB files the old formula comes to ~450 MiB; the new one asked 260 MiB. With your 2 GB files at 196K context the gap should be larger. The read now admits one checkpoint first. A file the RAM cache spilled (the same directory without the flag) can hold more, and is read once more under the engine's own limit before it counts as bad.

* A refusal deleted the file. Any failed read pass went to "discard unusable disk conversation", including "not enough RAM". Now only a file that fails validation (or is gone) is removed; a RAM, storage or I/O refusal keeps it for a later turn. The refusal also says how much it asked for and the floor.

Checked here on Strix Halo:

* **Disk-only rounds:** the card showed 8 saves, 6 restores and 3 prefixes.

* **Restart with a 200 GB floor:** this forces every read to be refused. Three restores were refused, and all six files were kept.

* **Normal restart:** the three conversations were restored from those files.

I can't test CUDA here, so a rerun of your setup would be welcome when you have time.

Measured with the help of Claude (Anthropic).

Thank you for making it perfect but I afraid I couldn't test until next week. Hope other can provide more data point.

edit: just one more thing: should a note be added to remind this feature will cause extra write to SSD which might accelerate its ageing?

[shahrokhzargarpour]

Hi! We're on the same layer (our PR #2510), same SCSM spill sidecar; ours is v2 (stage count for --layer-split, still reads v1).

Your --conversation-cache-disk-only (no RAM budget) is a mode worth keeping - our spill tier keeps a RAM mirror, not the reverse.

Proposal: converge the spill file on the v2 (backward-compatible) and keep the disk-only mode as a policy flag on top of it. Could we coordinate on that, crediting you and #1271 for the v1 base and the disk-only idea?

#2510

[CC-David-CC]

i love this community

[shahrokhzargarpour]

The disk-only mode is the part of this stack ours does not have, and by your own numbers, routhjim, it is the
cleaner one: 8 saves, 6 restores and 3 shared prefixes in a disk-only round on a Strix Halo, and after a restart the
three conversations were restored from those files; cold restores at 275-361 ms with the page cache evicted
(posix_fadvise(DONTNEED)). The streamed write is the part I keep coming back to — the park goes straight from the
live session's pools through SessionKvSource, so it never materialises a host image — and the persistent prefix
entries (strata-conv-prefix-N) that are never dropped as superseded and are evicted last are a simpler answer to
the shared-opening problem than ours.

Heads-up, so nobody is surprised when the maintainer picks one: our branches add the same three new paths
(include/strata/core/conversation_spill.hpp, src/core/conversation_spill.cpp,
src/core/conversation_spill_test.cpp), added on both sides, so whichever merges, the other rebases. It is what
the diff shows, nothing more.

One piece of ours that may be useful to you: we keep a separate, lower RAM floor for anything already on disk
(--conversation-cache-park-min-free-mib, 512 MiB against the parking floor's 2560), applied to the disk hit, the
segment hit, the system-prompt-cache hit, the streamed restore and the API session save. It came out of a Windows
read-back refusing its own write, and it sits on the same read path that is gated by the large floor here.

Either route serves the goal; the merge order is the maintainer's call.

[Niko1221]

Thank you. See #1090; we have no Strix Halo box now to measure this.

[G0-0000]

Frequent 1–3K-token side conversations can fill the disk budget and evict valuable long sessions. In park_current_body, the disk_only branch returns before the RAM path’s conversations.wants((int64_t) live.size()) guard; the shutdown save bypasses it too. The minimal guards from our commit are:

if (disk_only) {
    if (!disk_continuing && !conversations.wants((int64_t) live.size()))
        return true;
    if (!disk_continuing) disk_save_live(false, "switch");
    return true;
}

if (conversation_spill.enabled() && disk_only) {
    if (conversations.wants((int64_t) live.size()))
        disk_save_live(true, "shutdown");
}

No read-side gate and no gate on disk_save_root; --conversation-cache-n-min is read-side and separate. The default min_tokens=0 preserves behavior.

At threshold 10000 (the production value was later changed to 6000), the 2053-token session was skipped (disk files 7→7), while the 11653-token session was saved (7→8; 394 MiB in 593.9 ms). CPU checks passed: 78 spill, 4772 cache, 223 file.

Could this be incorporated into #1480? I can send a small PR against your branch if preferred. This fork includes adaptations, so I’m not implying this commit cleanly cherry-picks onto upstream.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.