[jeremiahritchey]

RAM-evicted conversations can retain reusable state on disk and return after an engine restart. This adds an opt-in disk tier with byte/entry quotas, using the snapshot representation and GPU restore path released in #189.

Replaces #190 following the request to rebase onto the shipped core. Based on v0.1.30 (30ec18e), including e46ccca's session carve. The diff contains only the dependent disk changes; #189 has shipped. Refs #57 and #52. Delta writes remain a possible follow-up from #265.

• Spill only on RAM eviction, with bounded staging, an exclusive process lock, interrupted-file cleanup and atomic publication. A continuing or latest active turn is not automatically durable.
• Select compatible prefixes across active state, RAM and disk. Validate the full envelope and integrity before GPU application; rejected/corrupt/unadmitted entries fall back to prompt processing.
• The v2 envelope preserves the session layer range introduced by Multi-GPU: the session carve and the per-stage prompt loans #216. Named identity fingerprints bind full asset contents, engine, geometry, per-layer expert quantization, KV layout/residency, resolved RoPE, split/device/runtime and inference settings. Log the first differing field before reading state.
• Disabled persistence performs no cache-directory I/O. OpenSSL Crypto is required only with STRATA_ENABLE_CONVERSATION_DISK=ON. This remains single-GPU, matching the released RAM cache.

Validation at 1b26ecf (the final commit only adds Windows setup documentation and the v2 ignore rule):

Check	Result
Linux CUDA builds, disk support ON and OFF	PASS; OFF has no libcrypto dependency and rejects activation before cache-directory I/O
Cache-disabled comparison with untouched v0.1.30	PASS, 72 requests: release / disk-enabled build / disk-disabled build × INT8 batched / K8V4 batched / INT8 ring/spec4; identical tokens, all nine main-state fingerprints, reuse and known answers
Six-phase disk lifecycle, INT8 batched / K8V4 batched / wrapped INT8 ring	PASS ×3: eviction-only writes, restart reuse, draft read-back, quota bounds, admission denial, changed-tokenizer and corruption fallback
Long disk restart	68,571 prompt tokens restored; draft read-back covers 68,572 cells with only 32,840 resident
GPU snapshot fixture	PASS, 1,781 checks
Long RAM growth/rewind with spec4; RAM off/on parity; HTTP reuse/eviction/cancellation/recovery	PASS ×3
Linux C++ host tests	PASS: codec, store (including fault injection), cache, memory, checkpoint policy, validation and transfer; the five CPU-only suites also pass ASan/UBSan
Python	41 cache-harness tests pass normally and with -O; 53 server tests pass
CLI	26 invalid-argument/configuration cases rejected before cache-directory I/O
Windows host build and tests	PASS, MSVC / OpenSSL 3.6.4 on windows-2022: all five CPU-only C++ suites; run and logs

Linux model testing used one RTX 4090, IQ3_S, CUDA 13.4, CPUs 0–31 with NUMA interleaving, fixed expert residency, and diagnostic state/draft read-back. All ten GPU/model jobs passed. These are correctness gates, not throughput measurements. The Windows workflow ran on a separate validation branch whose only addition to the tested code is that workflow; it is not part of this PR.

Remaining platform gate: Windows CUDA engine/model lifecycle execution is still needed. The hosted Windows run verifies OpenSSL/MSVC and CPU/file-store behavior, not GPU restore. docs/DETAILS.md now includes Windows OpenSSL discovery, build and runtime-DLL requirements.

Full asset hashing adds startup I/O; path/settings binding is intentionally strict. Disk files contain conversation content. Older draft v1 snapshots are not reused and their directory is left untouched. Windows GPU/HIP execution, power-loss survival and full-model weight-swap/cross-quant cache exchange are not established by the local tests. No multi-GPU disk support, page deduplication, planning files or general Pi benchmark tooling is included.

The streaming-envelope approach follows Marmaduke Woodman's (@maedoc) NVMe work in #52 (6648be7), adapted to the shared snapshot core. The RAM admission work from @midhatn is in the released base; the identity diagnostics address the owner/@QilinWan review in #57.

Reproduction (Linux, CUDA 13.4, RTX 4090, IQ3_S)

Build with the setup-pinned llama.cpp checkout and OpenSSL development headers available:

cmake -S . -B build-review-cuda -DCMAKE_BUILD_TYPE=Release \
  -DSTRATA_ENABLE_CUDA=ON -DSTRATA_ENABLE_HIP=OFF \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.4/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=89 -DSTRATA_PORTABLE=ON \
  -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_PREFILL_MMQ=OFF \
  -DSTRATA_BUILD_TESTS=OFF -DSTRATA_BUILD_CONVERSATION_TESTS=ON \
  -DSTRATA_ENABLE_CONVERSATION_DISK=ON \
  -DSTRATA_GGML_DIR=/absolute/path/to/llama.cpp
cmake --build build-review-cuda -j 2 --target strata conversation_snapshot_test \
  conversation_file_test conversation_store_test conversation_cache_test \
  conversation_memory_test conversation_validation_test conversation_transfer_test conv_cache_test
ctest --test-dir build-review-cuda --output-on-failure \
  -R '^(conversation_(file|store|cache|memory|snapshot|validation|transfer)_test|conv_cache_test)$'
python -m unittest discover -s tools -p 'test_conversation_cache*.py'
python -O -m unittest discover -s tools -p 'test_conversation_cache*.py'
python -m unittest serve.test_server

GPU/model tests require an exclusive GPU window. Use absolute asset paths in the engine configs, a fixed --expert-cache 6000 and --pcie-frac 0.55. The tested modes are:

Config	Context	KV	Resident cells
int8-batched.json	16384	int8	0
k8v4-batched.json	16384	k8v4	0
int8-ring.json	131072	int8	32768

Each output directory must be new. Each lifecycle starts six private engines and changes only copied tokenizer files/generated snapshots beneath that directory:

mkdir -p logs/pr190-repro
export STRATA_SNAPSHOT_VERIFY=1
export STRATA_MTP_BATCH=1
for mode in int8-batched k8v4-batched; do
  numactl --interleave=all --physcpubind=0-31 python tools/conversation_cache_disk.py \
    --config "$mode.json" --engine build-review-cuda/strata \
    --draft-path batched --output "logs/pr190-repro/$mode" --run
done
numactl --interleave=all --physcpubind=0-31 python tools/conversation_cache_disk.py \
  --config int8-ring.json --engine build-review-cuda/strata \
  --draft-path ring --paragraphs 4096 --output logs/pr190-repro/int8-ring --run

The lifecycle gate requires actual batched/ring evidence, restart draft read-back, exact output/main-state parity, known answers, quota bounds, cold fallback and the named tokenizer-identity rejection. docs/DETAILS.md documents the serving configuration and persistence limits.

[Niko1221]

Thanks! Please rebase onto 0.1.31 (after #309, which touches park_current); with STRATA_ENABLE_CONVERSATION_DISK=OFF the build must stay byte-identical, and we can run the Windows lifecycle gate on our 5070. Planned for 0.1.33. Closing until the rebase.

[BlueKingMuch]

@Niko1221 @jeremiahritchey - the Windows lifecycle gate you planned to run on your 5070: I ran it on #275 rebased onto 0.1.36, on Windows 11 with an RTX 4080 SUPER (sm_89, 32 GB), CUDA 13.3, MSVC 14.44 and IQ3_S. All of it passes, so you may not need to repeat it; or run it on the 5070 as well, as you prefer. Since then the branch has moved onto 0.1.37, where the ctest suites and the OFF and ON comparison against the untouched 0.1.37 pass again.

The branch: BlueKingMuch:conv-disk-0137-gate is Jeremiah's three commits on 0.1.37 and only what the Windows build and the gate needed on top. The rebase had conflicts only in serve/server.py, all of one kind: his block that binds the tokenizer and template paths, next to code 0.1.36 and 0.1.37 added there (the learned-profile return, the silence watchdog of #481). The two fixes, each its own commit:

• With the option ON, generate.cpp did not compile with MSVC: _get_environ is not in the Universal CRT (C3861). It now reads _environ, the narrow environment table main() gets. The hosted Windows run had built only the CPU-only suites, so this was the first MSVC build of the engine with the option on.
• conversation_cache_disk.py, and upstream's own conversation_cache_disabled.py and conversation_cache_growth.py, call engine.proc.wait() after engine.close(). Since 0.1.32, close() waits for the engine itself and then sets engine.proc to None, so each gate stopped with an AttributeError after its first engine. They now keep the process from before close().

OpenSSL is linked as Jeremiah has it, as a DLL (OpenSSL 4.0.3 from winget install -e --id ShiningLight.OpenSSL.Dev, its bin on PATH).

The gate on 0.1.36, as the PR describes it (absolute asset paths, a fixed --expert-cache 6000 and --pcie-frac 0.55):

	int8-batched (16K)	k8v4-batched (16K)	int8-ring (131K, 32768 resident)
OFF and ON against the untouched 0.1.36, conversation cache off (conversation_cache_disabled.py): same tokens, main state, reuse and answers	PASS	PASS	PASS
lifecycle (conversation_cache_disk.py --run): eviction-only persistence, restart, draft read-back, known answer, main-state parity, admission, foreign and corrupt fallback	PASS	PASS	PASS
back after an engine restart, against reading the prompt again	1.4 s / 3.2 s (2,018 tokens)	1.4 s / 3.2 s	5.1 s / 26.0 s (68,602 tokens)

The conversation ctest suites, including the GPU snapshot test, pass 7 of 7. On Linux, on 0.1.37: the five CPU-only suites, the 41 cache-harness tests (also with -O) and the 116 server tests pass, and generate.cpp builds with the option ON and OFF.

Faster, separately: BlueKingMuch:conv-disk-0137 has six more commits on top, each to take or leave, in this PR or in a follow-up once it is in. Measured on the same PC on 0.1.36, ring mode (the lifecycle passes there too); back after a restart in 4.1 s instead of 5.1 s:

• Startup hashes every asset in full, as before, but in 64 MiB pieces on all but one CPU thread: about 36 s instead of about 100 s for the 86 GB here, on every start with the disk cache on. The per-asset digest becomes SHA-256 over the pieces' SHA-256s, so snapshots written before are declined once; cheapest to take before anyone has snapshots.
• An eviction no longer forces the snapshot to the drive (no fsync/FlushFileBuffers): 0.8-0.95 s for 1.52 GB instead of 2.1-2.3 s in the gate run and 3.4-5.8 s in earlier ones (the flush takes as long as the drive's state says), time the request that caused the eviction waits before its first token. A snapshot that a power failure cuts short fails its footer when read and is now removed, instead of matching and being read in full again on every request with its prefix.
• A snapshot's larger spans are read on the threads the CPU expert pool leaves free, straight into the image, with the SHA-256 check in file order: 0.95-1.13 s instead of 1.9-2.2 s to load 1.42 GiB. Same file format. (MSVC's std::ifstream::read hands fread 4,095 bytes at a time, one 4 KiB ReadFile each; the readers use fread directly.)
• On Windows, OpenSSL is linked statically, so a disk-enabled engine needs no DLL, and a missing OpenSSL stops configuration with the winget command instead of FindOpenSSL's generic error.

One thing the rebase turned up: 0.1.36's opt-in expert_profile_save (#477) rewrites the expert profile every 10 minutes and has the frontend start from that file. The profile is an identity asset and its path is part of the arguments, so with that option on, no snapshot survives a restart.

@jeremiahritchey - would you like to take the branch into #275 or a new PR yourself, or should I open it with your commits (you stay their author) and link it here? If I haven't heard from you by Tuesday, I'll open it.

[BlueKingMuch]

@Niko1221 @jeremiahritchey - I won't open the PR I offered above after all. 0.1.40 has session files (#668): a file format of its own (v1) with model and config fingerprints and no OpenSSL. Rebased onto 0.1.40, #275 would add a second format and a second identity next to it.

What #275 still adds over 0.1.40 is the automatic part: a conversation the RAM cache evicts goes to disk and comes back on a prefix match, also after a restart, without the client saving it. If you still want that, should it write v1 session files? Then I'd build it on 0.1.40 with #275's rules (written only on eviction, a bounded folder, the longest prefix across the live session, RAM and disk) and the Windows gate. If #718, #751 or #1090 is closer to what you want, I'll leave it there.