[konijiwa110]

Reopening #960 here. It got auto-closed when main was force-pushed, and its branch was on the old history.

Same change, cherry-picked onto the new main (82f46a8). It conflicted only with the session files that landed in 0.1.40, in three places: conversation_file.cpp next to prefix_store.cpp in CMakeLists, --session-min-free-mib next to the new flags in the argument parser, and the sentence in DETAILS.md about what survives a restart. I kept both sides everywhere. Nothing else changed.

What

The system-prompt checkpoint from #62/#65 lives in the one K/V arena. It is gone as soon as a request with a
different start comes in (a title request, a subagent, a second client), and after every restart. Parking keeps it
inside the parked conversation, but lending it takes that conversation out whole, so the conversation reads its
history again on its next turn. With agent clients this means many new chats still start from token 0, and their
system prompt plus tool list is often 20-35K tokens.

--prefix-cache-dir DIR keeps that checkpoint in a file:

• Save. A request that read its system prompt has answered, so its DONE is already out. Its root checkpoint's
  running state and the K/V up to that point, draft layer included, are streamed from the session into DIR. This
  happens once per distinct system prompt.
• Restore. A later prompt that starts with a saved prefix restores it, then reads only the rest. It does this
  when the live session or a batch slot cannot serve more of the prompt. A parked conversation of the same length
  loses to the file, so it stays parked. The file stays on disk.
• Disk and RAM limits.
  • --prefix-cache-disk-mib N (default 51200) caps the directory, removing the least recently used files first.
  • --prefix-cache-ram-mib N (default 2048, 0 = disk only) also keeps recent snapshots in RAM, when
    --conversation-cache-min-free-mib allows it.
  • Without RAM, save and restore stream through one 64 MB buffer, so low memory does not block either one.
• Stale files. Each file records the engine build, the pack, --native, --mtp, --kv and STRATA_BF16_TC.
  If any of these differs (a rebuilt engine counts), the file is deleted at start-up.
• Validation. Every restore goes through the same checks as a parked snapshot before anything is written.
  • A restore that fails before writing keeps the session as it was.
  • A restore that fails after it has started writing reads the whole prompt instead. It does not exit.

This works on one GPU without --batch only, and stays off unless the directory is set.

Measured

All numbers are from this branch alone: main (82f46a8) plus this commit, nothing else on top.

Swift 1.5 IQ3_XXS, 256K context, --kv-resident 32768, --prefix-cache-ram-mib 0, --adapt-every 100000 (fixed
experts, so the answers can be compared). Each prompt is a 30K-token system prompt plus a short user message, with
greedy decoding and 64 output tokens.

RTX 3080 12 GB, i7-13700KF, 64 GB RAM, Windows 11, Samsung 980 PRO. --kv int8.

request	reused	prefix restore	request time
first chat, full read (30,046 tokens)	0	-	27.53 s
same prompt after an engine restart	30,028	396 ms (disk)	2.06 s
new chat after a request with another start	30,028	258 ms (disk)	2.31 s

• The answer after the restart (content and reasoning) is byte-identical to the full read's.
• Saving the 30,028-token prefix took 484 ms after DONE. The file is 577 MB.
• With --prefix-cache-ram-mib 2048, the first restore reads the file into RAM (402 ms) and the next one comes from
  RAM (129 ms).

RTX 2080 Ti 22 GB, i5-12490F, 62 GB RAM, Ubuntu, NVMe. --kv k8v4.

request	reused	prefix restore	request time
first chat, full read (30,046 tokens)	0	-	27.92 s
same prompt after an engine restart	30,028	357 ms (disk)	2.43 s
new chat after a request with another start	30,028	181 ms (disk)	1.80 s

• Here too the answer after the restart is byte-identical to the full read's.
• Saving took 413 ms. A K8V4 file is about 118 MB of running state plus 12.4 KB per token, so this one is 490 MB.
• With the RAM tier on: 436 ms for the first restore (file into RAM), 88 ms from RAM after that.

I also checked whether the model still knows the restored prompt. I put a code word in the middle of a 13K-token
system prompt. After a restore from disk, a newly worded question still got it right.

Verification

• prefix_store_test (new, GPU) runs FP16, INT8 and Q4_0, each with resident and streamed K/V, plus K8V4. It saves
  a prefix from a session that holds more than the prefix, then restores it three ways: streamed from the file, read
  into RAM, and from RAM. After each restore, the K/V, indexer and running state are byte-identical to the state at
  the prefix. It also checks four cases:

  • the index survives a restart;
  • a file with another identity is deleted;
  • a disk budget for one file evicts the older file;
  • a truncated file is dropped.

  210 checks pass on this branch on Linux (sm_75) and on Windows (MSVC, sm_86).

• conversation_snapshot_test passes unchanged (3901 checks, Linux and Windows), and so does conversation_file_test (185, Linux). The existing conversation_kv_validate and
  conversation_kv_restore now share their size check and residency step with the new streaming functions.

Docs

docs/DETAILS.md gets a new paragraph, "Prefix snapshots on disk", after the parked-snapshot notes. That section
now says parked snapshots do not survive a restart, but the session files and the prefix snapshots do.

🤖 Generated with Claude Code

[konijiwa110]

Found a hole while looking at rope scaling: the snapshot identity didn't cover the rope config (nor KV_ROT, the control vector or the arithmetic switches), so turning on --rope-scaling yarn without a rebuild would restore K rotated with the old angles. Pushed a fix that reuses the session files' config fingerprint, so both are keyed on the same settings.

Checked on main + this PR on a 2080 Ti: a same-config restart still restores (30,028 tokens from disk in 357 ms); a restart with --rope-scaling yarn --rope-scale 2 deletes the old file at startup and reads the prompt in full.

[konijiwa110]

Rebased onto current main (e8ca9af). The only conflict was in generate.cpp, around the new pin= sibling check (pin_sibling): a request that restores a prefix snapshot overwrites the session, so it still parks the current conversation first, and pin_sibling now also requires prefix.tokens == 0.

Also added a small fix on top: the snapshot identity now includes the model files' fingerprint, the same one session files use (each file's size plus its first and last MiB). Before this, if a model file was replaced in place under the same path, the old snapshots still counted as valid.

Retested on the rebased branch (RTX 2080 Ti, --kv int8 --kv-resident 32768, 256K context):

• prefix_store_test 210, conversation_snapshot_test 3901 and conversation_cache_test 4198 checks pass
• 30K-token system prompt: first chat reads it in 24.5 s and saves it (576.5 MB, 0.5 s). After a restart, the same prompt restores it from disk in 0.53 s and reads 18 tokens; the answer is identical (content and reasoning). A new chat with the same system prompt also restores it, and a different prefix in between doesn't disturb it.
• With --prefix-cache-ram-mib 2048, the restore is 0.71 s the first time (disk>ram) and 0.10 s from RAM after that.

[Niko1221]

Thank you. This is one of six open disk-persistence designs (#1090, #1331, #1480, #1937, #2077, #2119). We will pick one design for 0.1.46 rather than take them piecemeal; we have not measured this one.

[Interpause]

Thank you. This is one of six open disk-persistence designs (#1090, #1331, #1480, #1937, #2077, #2119). We will pick one design for 0.1.46 rather than take them piecemeal; we have not measured this one.

Just a few factors I hope you will consider:

• Whether the PR can easily be built upon in the future to all features listed below.
• Persist cache across restarts.
  • Maybe some LRU-based disk budget to evict stale cache that is config-agnostic in case requests frequently switch between different model params.
• Gracefully skips over RAM tier cache to disk cache if remaining RAM too little for conversation even when all cache has been evicted to disk.
• Some form of history-based prefix or checkpointing so edits to the conversation performs minimal prompt reprocessing.
  • Even better if disk cache is aware of checkpoints/prefixes/chunks in a way where common history is stored once, with each "fork" being stored separately.
• Possibility of disabling RAM cache altogether in favour of just disk cache for people with high durability drives like Optane SSDs or RAID setups.

EDIT: Also asking GitHub Copilot, it seems quite a few other PRs exist and are active. #2510 seems to be the most ambitious in doing everything. Nonetheless, as much as I also wish we could have all features now, it would make sense to pick the simplest yet most extensible one as the foundation I think.

[G0-0000]

A data point for the 0.1.46 decision, from a UD-Q4_K_XL box (V100 32 GB + 128 GB RAM, dual NUMA nodes):

1. RAM capacity is our scarcest resource — a RAM parking tier is simply unusable here. The resident expert complement alone pins ~48 GiB, so we run with --conversation-cache-mib 0; nothing is left for RAM parking. For this class of machine, a disk-only design (serve: --conversation-cache-disk-only - the conversation cache on disk, with no RAM budget (depends on #1271, #1269) #1480 / Add disk-backed conversation cache parking (--conversation-cache-storage disk) #2119 / [Engine] Conversation cache: park and restore conversations through session files #2915) is not a nice-to-have but the only workable option. Please weigh them heavily, or at least keep the RAM tier optional in whichever design wins.

2. Moving cache I/O to NVMe also relieves the memory subsystem. Expert streaming already keeps RAM read traffic busy on every token; RAM parking adds more memory read/write on top. We have recurring whole-system freezes on this box and suspect the extra cross-node NUMA memory traffic is a contributing factor. Every byte of cache traffic moved from RAM to NVMe is traffic removed from the memory subsystem — and NVMe has ample headroom: session files run ~1–3 GB, so a restore is seconds versus minutes of cold re-prefill.

3. Whatever design is picked, please take [Engine] restore a session file without holding its K/V in RAM #1269 (streaming restore, constant peak RAM) along with it — today RESTORE buffers the whole snapshot in host RAM, effectively doubling the cost of a restore and risking 503 on RAM-constrained machines.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.