[dsenchankau]

Problem and result

I took this PR idea from Maya, a GLM5.3-Flash fork of Strata, where the disk conversation cache is on by default.

Right now the conversation cache has 2 options: keep it in RAM, or don't use it at all.

RAM is fine with a smaller quant (IQ3_S) and plenty of system RAM (96GB with that quant). It gets tight with a bigger quant like UD-Q4_K_XL, or with less system RAM.

UD-Q4_K_XL's experts are way bigger, and after startup the OS overhead + model + engine + coding harness have already eaten (almost) all the RAM. That leaves you two bad options: offload some experts to disk and lose decode and prefill speed, or skip the RAM cache and re-process the prompt over and over (slower the bigger the model gets).

For performance, the better move is to move the conversation cache to disk and keep more of the model's experts in RAM.

Without a cache, a conversation that comes back re-reads its whole prompt. With the cache on disk it's read back from its file and picks up where it left off. The re-reading gets worse the longer the session runs, but reading from the file stays about as cheap no matter how long it's been, so the disk cache pays off across a working session. You notice it most when you're using subagents a lot and bouncing between the main session and side ones. I measured the park/restore part (see Validation); I didn't run a reprocessing-vs-restore timing test, so I'm not putting a speedup number on it.

With this change, --conversation-cache-disk-mib N parks conversations into conv-*.sess files under <pack>/conversation_cache, and the budget counts the file bytes, so a lot more stay parked than would fit in RAM. A conversation that comes back on a later turn (same run, or after a restart) is read back from its file and continues from its own state instead of re-reading its prompt. It's off by default, the way the repo wants new features.

Approach

• include/strata/core/conversation_cache.hpp - disk mode: enable_disk, put_disk (budget counts file bytes), the slim folder-index entry, and the budget check that refuses an oversized entry before it touches anything already parked.
• src/core/conversation_file.{hpp,cpp} - meta-only read (session_file_read_meta) and the startup folder scan (conversation_cache_index) that keeps this model's fresh conv-*.sess and deletes foreign, stale or unreadable files. Reuses the session-file format that's already there.
• src/program/generate.cpp - the park and restore calls in the serve loop, plus the guards that skip parking (snapshot bigger than the whole budget, and RAM admission).
• include/strata/core/{conversation_snapshot.hpp,session_save_reclaim.hpp}, src/core/conversation_state.cpp - the meta/checkpoint plumbing the index and restore need.
• No serve/ change: the engine flags go straight through the run-config args, so server.py is untouched.
• Tests: a disk-mode budget-refusal check in conversation_cache_test.cpp; conversation_file_test.cpp already covers the meta-only read and the folder scan.
• docs/DETAILS.md - the flag and the measured numbers.

Validation

Revision 1aa35f01. Environment: Linux home-pc 7.0.0-38-generic #38~24.04.4-Ubuntu SMP PREEMPT_DYNAMIC x86_64, RTX 4080 SUPER (driver 615.71.10), CMake 3.28.3, Ninja 1.11.1, CUDA 13.4, Python 3.12.3.

Passed:

• Build (CUDA backend only; the change is common code, no backend sources or #ifdef):

  cmake -G Ninja -S . -B build-tests -DCMAKE_BUILD_TYPE=Release \
    -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_CONVERSATION_TESTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES=89
  cmake --build build-tests -j "$(nproc)"

• The disk-cache unit tests (CPU-only), 5/5: conversation_cache_test (4789 checks), conversation_file_test (217 checks), plus conversation_split_failure_test, conversation_memory_test and session_save_reclaim_test:

  ctest --test-dir build-tests -R 'conversation_(cache|split_failure|memory|file)_test|session_save_reclaim_test' --output-on-failure

  5/5 passed, 1.46s.

• The engine conversation tests, 3/3. conversation_snapshot_test uses the GPU (passed in 1.11s on an idle card; it skips with code 77 when it can't get one). conversation_validation_test and conversation_transfer_test run with CUDA_VISIBLE_DEVICES=-1:

  ctest --test-dir build-tests -R 'conversation_(snapshot|validation|transfer)_test' --output-on-failure

  3/3 passed, 1.17s.

• Full suite. ctest --test-dir build-tests --output-on-failure on the branch gives the same result as master: 96% passed, 6 failed out of 157, 1 skipped (expert_multi_test). Those 6 are the usual environment/hardware ones on both branches (ple_parity and tq_cpu_parity want fixtures that aren't in the published source; ggml_tq_overlay, native_hadamard_test, tq_cpu_parity_avx2 want AMD/trellis hardware; isa_guard_check is a no-ISA check). So nothing new broke. Run this on a free GPU, or conversation_snapshot_test OOMs on whatever else is using the card.

• Engine run on one RTX 4080 SUPER, IQ3_S pack, greedy. I launched the model the normal way from its config, adding only --conversation-cache-disk-mib 16384 to the run-config args:

  .venv/bin/python serve/server.py --engine strata --config strata-iq3_s.json --port 1337
  

  It parked 1962 tokens to disk (266,114,352 bytes), I restarted it, and it took back the same 1962 tokens from disk. The next turn only read the new part (cached_tokens 1962, prompt_n 17). The guards hold too: a snapshot bigger than the whole budget is refused and the folder stays empty, foreign files and a hand-edited conv-hacked-9.sess are deleted at startup, and --conversation-cache-days 0 empties the folder.

• serve suite:

  PYTHONPATH=. .venv/bin/python -m unittest discover -s serve

  881 run, 14 skipped, 1 failed. The one failure (test_lifeline: "the child outlived a killed server") is a process-watcher timing race. It's not from this change: git diff upstream/main HEAD -- serve/ is empty, so I touched no serve file, and the test fails the same on master.

Failed: nothing here is caused by this change. The 6 C++ failures above are the same environment/hardware ones master has, with or without the branch, and the serve test_lifeline failure is a watcher race in a file this PR doesn't touch.

Not run: HIP and SYCL, build and run - I don't have AMD hardware. The change is common code only (no backend sources, no #ifdef), so it should be backend-agnostic, but that's me reading the code, not a build - call it unconfirmed static analysis. Someone with that hardware should build it, and run it once on each if they can. Metal I didn't build or run. The park/restore numbers above are NVIDIA/CUDA only.

Limits

The park/restore measurement is one pack on one card (RTX 4080 SUPER). I'm not claiming a speedup on other hardware or quants beyond the reasoning above. The disk cache is opt-in, and I didn't change the RAM cache path or how its budget is counted. HIP and SYCL are unverified because I have no hardware for them.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[dsenchankau]

/reopen

[github-actions]

Not reopened yet. Still missing:

• Fill in "Validation": the exact commands, the revision and environment tested, what you saw. Keep Passed, Failed and Not run apart.

Edit the pull request and comment /reopen again. Details: CONTRIBUTING.md

[dsenchankau]

/reopen

[github-actions]

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.