[ANBAL534]

The RAM conversation cache (--conversation-cache-mib) parks whole conversations so a later request reuses their prefill instead of reading the prompt again. But a parked conversation is dropped the moment RAM pressure or --conversation-cache-slots evicts it, and every parked conversation is gone when the engine restarts. On a small-RAM PC the budget is tight, so the conversations that matter are the ones evicted; after a restart the long prompt is read again from token 0.

--conversation-cache-spill-dir DIR keeps what the RAM cache evicts: the evicted conversation is written to DIR as an ordinary session file (the same format and model/config identity as the slot save/restore API) plus a small metadata sidecar, so a later request - or a restart - reads it back instead of re-reading the prompt.

TL;DR

• a parked conversation evicted from RAM is written to disk and can be resumed later, also after a restart
• matching a disk conversation costs a small sidecar read, not a K/V read
• a spilled file is an ordinary session file: one format, one identity rule, interchangeable with the slot save/restore API
• opt-in and off by default; single-GPU, like the parking it extends

What changes

• A conversation the RAM cache evicts (make_room) is handed to a spill callback that writes it to the directory; on a clean QUIT or stdin close the parked conversations spill before exit (spill_all).
• A new request matches the directory from the sidecars only (token and image lists, no K/V read) and, when a disk conversation offers a longer resume than RAM, the batch slots and the live state, reads it back whole - subject to the RAM budget and the --conversation-cache-min-free-mib floor.
• --conversation-cache-similarity F and --conversation-cache-n-min N filter weak matches (least common-prefix fraction / token count); both default to 0, which keeps every exact prefix the RAM cache would have used.
• The spill reuses conversation_file.cpp rather than adding a second on-disk format: a spilled conversation is byte-for-byte what the slot save/restore API would write, so the two are interchangeable and there is one format, one model/config identity rule, one checksum.

Why it helps

• Less RAM, same reuse. Set a small --conversation-cache-mib and let the disk tier hold the rest: the RAM cache stays a small hot working set, and the conversations it evicts stay resumable on NVMe instead of being recomputed. On a 12-16 GB card where the cache budget competes with the expert cache, this trades a little disk for prompt-read time.
• No cold load after a restart. The engine logs spill dir ready (N conversations, ...) at start, and the first request after a restart resumes from disk instead of reading the whole prompt again.
• Prefill shared across conversations. Because a spilled file is a session file, a hand-saved session and a spilled conversation are the same artifact; and similarity matching lets a new conversation that shares a long prefix with a spilled one (the same system prompt, the same document) resume from that prefix rather than from token 0.

Flags

flag	default	meaning
--conversation-cache-spill-dir DIR	off	keep conversations the RAM cache evicts on disk, across restarts
--conversation-cache-disk-mib N	8192	the directory's limit for this model (0 = off)
--conversation-cache-similarity F	0	least common-prefix fraction a disk hit may offer (range [0,1))
--conversation-cache-n-min N	0	least common-prefix tokens a disk hit may offer

Testing

• conversation_spill_test (41 checks, wired into STRATA_BUILD_CONVERSATION_TESTS): spill/match/load round-trip with the K/V compared byte for byte, similarity and n_min filtering, refusal of a foreign model/config identity, disk-budget eviction, reindex on reopen, and the RAM cache handing its evictions to the tier (make_room callback + spill_all).
• The existing conversation cache / file / memory / split tests still pass.
• generate.cpp and the module compile in the CUDA build and strata links; the flags appear in --help and bad values are refused.
• A comparative end-to-end run (the same workload with and without the patch, measuring prompt-read time and RAM held) is still pending; I will add the measured numbers.

Scope

Single-GPU only: with --layer-split the disk tier stays off, matching the existing parking limit. A disk hit is read into host RAM before restore, so it must fit the RAM cache budget and leave the min-free floor available, or the request reads the prompt normally.

[ed-hch]

May I ask is it possible to use disk to park conversation in the first place and skip RAM parking entirely? For a 64GB machine, really don't have much headroom after load IQ3_XXS. And I'm hitting a lot of park rejection in the first RAM parking gate.

[midhatn]

The disk tier helped in a local v0.1.40.2 port on Windows, but I found a recovery case worth fixing before merging.

With one RAM cache slot, a request can have a shorter RAM match and a longer disk match. parked is computed first; the disk attempt then calls conversations.make_room(...), which can evict that RAM entry. If loading the disk file fails, the fallback still calls conversations.take(parked.index) using the old index.

I reproduced this using only synthetic snapshots: A and B share a system root, A is on disk, B is the sole parked RAM entry, and an unrelated conversation is active. After changing A's file header, requesting A logs discard unreadable disk conversation (session file: not a Strata session file (magic)), then fails instead of rereading the prompt. Windows recorded a fatal application exit; the test watchdog subsequently stopped the owned process as available RAM fell below its guard. I have not established the cause of that transient RAM drop, so I am not calling it a memory leak.

Recomputing the RAM match after an unsuccessful disk attempt avoids the stale index:

if (!incoming)
    parked = conversations.best(ids, req_imgs, want_cvec,
                                 o.conversation_cache_similarity,
                                 o.conversation_cache_n_min);

This belongs immediately before the RAM take/restore decision. It also covers admission/limit failures after make_room has changed the cache. My combined local cache port with this change rejected the damaged file, reread the prompt and returned the correct A-specific answer; all checks in that run passed, with at least 10.67 GiB RAM and 353 MiB actual VRAM free. That candidate also includes #1163, #1164 and #1394, so this is not an isolated binary A/B of those three lines.

The normal disk path and restart checks passed separately. In a deliberately constrained one-slot seven-request sequence, newly read tokens fell from 37,345 to 18,690 and prompt processing from 127.18 to 67.61 seconds. Total wall time was 150.65 vs 93.44 seconds, with different sampled output totals (513 vs 550). An already-spilled conversation also restored after restarting the process. This demonstrates the eviction/reuse case, not a universal Hermes speedup or persistence of the live RAM-only conversation.

Hardware: Ryzen 9 7940HS, RTX 4070 Laptop 8 GB, 64 GB RAM; IQ3_S, 64K context. Tests and the damaged file were isolated from production.

Update after reviewing the current branch: commit 1904b8a adds the RAM-match recomputation after a failed disk attempt, and it is present in head 1a9e8b9. That addresses the stale-index path described above in source. I have not rerun the damaged-snapshot case on this new head, so the execution evidence above still refers to the earlier control and local combined fix, not to a newly verified upstream binary.

[ANBAL534]

May I ask is it possible to use disk to park conversation in the first place and skip RAM parking entirely? For a 64GB machine, really don't have much headroom after load IQ3_XXS. And I'm hitting a lot of park rejection in the first RAM parking gate.

With the current code is not possible I'm afraid, this functionality it would need more extensive code changes worth another PR.
Your best bet is to set in the launch flags a single ram slot and multiple disk slots, so every conversation switch spills to disk directly.

The disk tier helped in a local v0.1.40.2 port on Windows, but I found a recovery case worth fixing before merging.

With one RAM cache slot, a request can have a shorter RAM match and a longer disk match. parked is computed first; the disk attempt then calls conversations.make_room(...), which can evict that RAM entry. If loading the disk file fails, the fallback still calls conversations.take(parked.index) using the old index.

I reproduced this using only synthetic snapshots: A and B share a system root, A is on disk, B is the sole parked RAM entry, and an unrelated conversation is active. After changing A's file header, requesting A logs discard unreadable disk conversation (session file: not a Strata session file (magic)), then fails instead of rereading the prompt. Windows recorded a fatal application exit; the test watchdog subsequently stopped the owned process as available RAM fell below its guard. I have not established the cause of that transient RAM drop, so I am not calling it a memory leak.

Recomputing the RAM match after an unsuccessful disk attempt avoids the stale index:

if (!incoming)
    parked = conversations.best(ids, req_imgs, want_cvec,
                                 o.conversation_cache_similarity,
                                 o.conversation_cache_n_min);

This belongs immediately before the RAM take/restore decision. It also covers admission/limit failures after make_room has changed the cache. My combined local cache port with this change rejected the damaged file, reread the prompt and returned the correct A-specific answer; all checks in that run passed, with at least 10.67 GiB RAM and 353 MiB actual VRAM free. That candidate also includes #1163, #1164 and #1394, so this is not an isolated binary A/B of those three lines.

The normal disk path and restart checks passed separately. In a deliberately constrained one-slot seven-request sequence, newly read tokens fell from 37,345 to 18,690 and prompt processing from 127.18 to 67.61 seconds. Total wall time was 150.65 vs 93.44 seconds, with different sampled output totals (513 vs 550). An already-spilled conversation also restored after restarting the process. This demonstrates the eviction/reuse case, not a universal Hermes speedup or persistence of the live RAM-only conversation.

Hardware: Ryzen 9 7940HS, RTX 4070 Laptop 8 GB, 64 GB RAM; IQ3_S, 64K context. Tests and the damaged file were isolated from production.

Update after reviewing the current branch: commit 1904b8a adds the RAM-match recomputation after a failed disk attempt, and it is present in head 1a9e8b9. That addresses the stale-index path described above in source. I have not rerun the damaged-snapshot case on this new head, so the execution evidence above still refers to the earlier control and local combined fix, not to a newly verified upstream binary.

I added the fix to the code and applied upstream main to my branch. Thanks for the test and good catch!

[henrikno]

@ed-hch looks like disk only is happening in #1480

[shahrokhzargarpour]

Hi! We're building the same disk tier (our PR #2510): spill = sidecar + ordinary session file, same SCSM format as here.

The one difference: ours is v2 of this format - it adds a stage count so --layer-split spills one file per stage, and it still reads the v1 sidecar this PR introduces (kMetaVersion = 1).

We'd rather converge than race two copies of the same file: v2 is backward-compatible with v1, so your sidecars keep working unchanged. Want to coordinate so we land a single conversation_spill.{hpp,cpp,test.cpp}, crediting you for the v1 design?

#2510

[shahrokhzargarpour]

The call to make the spill file the same artifact as the API save/restore session file is a good one, ANBAL534 —
a file written by one path stays usable by the other, and ours does not do that. The similarity match is the other
thing ours does not have: --conversation-cache-similarity F and --conversation-cache-n-min N, where our match
is an exact prefix from the first token. A tester in this thread put numbers on what the branch buys: on a local
port, a one-slot seven-request sequence fell from 37,345 to 18,690 newly read tokens and from 127.18 s to 67.61 s of
prompt processing (150.65 s to 93.44 s wall), and a spilled conversation came back after a process restart.

Heads-up, so nobody is surprised when the maintainer picks one: our branches add the same three new paths
(include/strata/core/conversation_spill.hpp, src/core/conversation_spill.cpp,
src/core/conversation_spill_test.cpp), marked added on both sides, so whichever one merges, the other needs a
rebase. That is what the diff shows; it is not a judgement about either.

Two things of ours that may be useful to you, offered as material rather than as a comparison. Our sidecar is
version 2 and still reads version 1, so it accepts the files this branch writes; the version check and the reader
branch are in conversation_spill.cpp if cross-reading ever matters. And we keep a separate, lower RAM floor for
anything already on disk (--conversation-cache-park-min-free-mib, 512 MiB against the parking floor's 2560), which
came out of a read-back refusing its own write on a small-RAM machine — possibly relevant here, where the read-back
sits under the same large floor.

The choice between the two routes is the maintainer's, and either one reaches the same place.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.