Repository navigation
serve: the conversation disk tier writes in the background, keeps the root checkpoint, takes what RAM cannot hold (on #1480) - #1529
konijiwa110 wants to merge 19 commits into
Conversation
The RAM conversation cache drops a parked conversation when RAM pressure or --conversation-cache-slots evicts it. With --conversation-cache-spill-dir DIR it is written to DIR as an ordinary session file (the same format and model/config identity as the slot save/restore API) plus a small metadata sidecar, so a later request - or a restart - reads it back instead of re-reading the prompt. The spill reuses conversation_file.cpp rather than a second on-disk format: a spilled conversation and a hand-saved session file are interchangeable, and the same identity rules refuse a foreign file. The sidecar holds only the token and image lists, so best() matches a disk conversation without reading its K/V. --conversation-cache-similarity and --conversation-cache-n-min filter weak hits; a disk hit must fit the RAM budget and the min-free floor before it is read back. On a clean QUIT the parked conversations spill before exit. Single-GPU only: with --layer-split the disk tier stays off. conversation_spill_test covers spill/match/load round-trip, similarity/n_min, identity refusal, budget eviction, reopen reindex, and the RAM cache handing its evictions to the tier (make_room callback + spill_all).
The RESTORE path materialized the whole snapshot in host memory, so a deep session could not be restored on a machine that serves it: upstream's own documented 503 'when the RAM to read the file is not there'. The restore is now two passes over the same file, format v1 unchanged: - the read pass parses and validates the whole file (running state, K/V headers, every count against this session's limits, the payload hash) without materializing the K/V; it reports each layer's part sizes. - the apply pass re-reads the file and copies the K/V into the authoritative pools a 16 MiB block at a time. It is bound to the read pass: a file whose layer count or any part size differs is refused before that part is applied, and the payload hash is compared to the trailer at the end. The first applied block is the gate - before it a failure refuses cleanly, after it the engine fail-stops rather than decode from bytes it cannot vouch for. Peak RAM is two 16 MiB buffers and the running-state metadata, so a session of any depth restores under the same floor. Measured on a 198,830-token IQ3_S session: 3.05 GiB restored in 3.4-3.5 s, next turn 99.99% reused. The residency refill after the pool writes is mandatory for the modes that keep K/V in VRAM and runs per layer (kv_stream_reset / kv_ring_restore). (cherry picked from commit d438ca5)
…disk # Conflicts: # include/strata/core/conversation_cache.hpp
…pt: make_room may have evicted the entry the stale index named
…k, no RAM budget A conversation switched away from is streamed from its pools straight into --conversation-cache-spill-dir (the SAVE path's capture: the deepest checkpoint and the running state are copied, the K/V is not) and a returning one is streamed back with the two-pass RESTORE path (Niko1221#1269): the read pass checks the file before anything on the device changes, the apply pass writes the pools 16 MiB at a time. No parked host image and no RAM budget: the disk limit is the only one, so a long conversation is never refused for its size. The new file is written before this conversation's older copies are dropped (drop_superseded gains a keep path), so a failed write loses nothing. The agent's next turn of the live conversation (it resumes at the newest turn checkpoint) writes nothing. Shutdown saves the live conversation, flushed. Single GPU, without --batch or --peer-device. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
…-disk-only, measured on Strix Halo 18 checks: a streamed spill writes the same file as the image spill, matches and reads back with the same K/V, refuses a host K/V image or an empty conversation, leaves no file or index entry when a source fails, and drop_superseded spares the `keep` copy (the rule alone would drop it). 59 checks pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
…ge cache Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
… prefix entry A conversation saved to disk also writes its root - the system prompt and tools, up to the first turn boundary at least --prompt-cache-root tokens in (the engine's root rule) - once per distinct root, as strata-conv-prefix-N: the root's K/V cells and, as live state, the root checkpoint's own running state. A new conversation that opens the same way (a coding agent's next session) resumes from it instead of reading the preamble again. Prefix entries persist: drop_superseded never removes them, the disk budget evicts ordinary conversations first, and a reopen (restart) keeps them as prefixes; has_prefix dedupes. The sidecar format is unchanged (the kind is in the name). conversation_spill_test: 14 new checks, 73 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
…pin sibling saves the conversation it leaves A request's "strata_prefix" (pin=N) marks where the part its sessions share ends - for a coding agent that runs into the first user message (its <system-reminder> blocks), past the first turn boundary. The prefix entry is the deepest pinned checkpoint on the live path when there is one, the root rule's otherwise. In disk-only mode a request that resumes at a pinned prefix is another conversation, so the one it leaves is saved (the RAM cache's "N queries cost one prefix" skip would drop it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
…point first
A disk-only read pass that was refused for RAM, storage or I/O went to "discard unusable disk conversation" and
deleted the file, so each refusal lost a conversation. Only a file that fails validation (or is gone) is removed
now; any other refusal keeps it for a later turn ("disk hit skipped", with the MiB asked for and the floor).
The read pass's RAM preflight was sized for --prompt-cache checkpoints (6 by default), capped at the file size; a
disk-only save writes one. The read admits one first; a file the RAM cache spilled into the same directory can
hold more and is read once more under the engine's own limit before it counts as bad. On Strix Halo (~420 MiB
files, 64K context) the preflight asked 260 MiB, about 450 before. Measured: three restores refused under a forced
floor kept all six files, and the next start restored the three.
The INFO line reports conversation_cache_disk_only=0/1 for the monitor.
Reported-by: ed-hch (CUDA test of Niko1221#1480)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
…k-only The card keyed on --conversation-cache-mib > 0, so in disk-only mode it showed no cache and told the user to add --conversation-cache-mib. ConvCacheLog now reads the disk-only lines (spill dir ready / disabled, disk-saved, prefix-saved, disk-restored, refusals with the last reason, discarded files); with conversation_cache_disk_only=1 the card shows a bar for the spill directory against --conversation-cache-disk-mib and those counters instead of the RAM bars, and its note describes the disk tier, or says the directory did not open. test_monitor: 2 new tests, 12 pass; checked against a live engine (8 saves, 6 restores, 3 prefixes on the card). Suggested by ed-hch, whose local fix covered the same ground; this is a separate implementation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XBY17SSRKGmDXc2YFsrmp
…disk # Conflicts: # include/strata/core/conversation_cache.hpp
|
excelente, estoy aún finalizando el merge para la compatibilidad con versión 0.1.41 de esta rama, muchas gracias por los aportes! mi siguiente aporte también en proceso, mtp x slot kv, y el tercero, pipeline multislot + expertos multislot + pipeline parallelism + continouos batching |
…ot checkpoint, takes what RAM cannot hold On top of Niko1221#1271 / Niko1221#1269 / Niko1221#1480 (merged unchanged): - an evicted conversation is handed to a writer thread (spill_async) instead of written - and flushed - inside the request that evicted it; the file is indexed, so matched, only once wait() has joined the write (before a disk match, a streamed save and at shutdown). When the physical-RAM admission refuses a park or a disk hit while a write still holds an evicted image, the request waits for that write first. - a spilled file keeps the deepest checkpoint (as SAVE) and the shallowest one (the chain's root - the end of the system prompt a new chat resumes from); the ones between only serve edits further back. - with the RAM cache on, a conversation it cannot park (over its budget, or refused by the RAM admission) is streamed to the directory as --conversation-cache-disk-only writes it, and a disk hit over the RAM budget is streamed back as that mode reads it, instead of both being dropped; put() hands what it evicts to the tier too (it could evict past make_room's estimate without spilling). - --conversation-cache-disk-min-tokens N (default 0): shorter conversations are not written. - every spill write leaves --session-min-free-mib free on the disk. conversation_spill_test: 34 new checks (93 pass, also under ThreadSanitizer); the other conversation tests pass. End to end on an RTX 2080 Ti (docs/DETAILS.md): later turns 5.20 -> 4.51 s median with a 4 GiB RAM cache, and 14.83 -> 5.01 s with 256 MiB (0 -> 12 of 12 turns resumed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… not refused A disk hit within the RAM budget was read into RAM whole; when make_room could not free the room or the physical-RAM admission refused it, the request fell back to a prefix snapshot or a full read. It now takes the streamed read instead, which holds 16 MiB blocks (single GPU, without --batch or --peer-device). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2b779b1 to
86ec596
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hi! Same layer here (our PR #2510), same Your background write + root-checkpoint retention is the right shape - our GC only deletes at budget and never drops an oversized conversation. Proposal: one spill file (v2, backward-compatible) with the background-write / root-checkpoint policy as flags on top. Happy to coordinate and credit you, #1271 and #1480 for the v1 base and these policies. |
|
why those design choices? |
|
Keeping the root checkpoint beside the deepest one is the choice here that ours does not make, konijiwa110, and the Heads-up, so nobody is surprised when the maintainer picks one: our branches add the same three new paths One thing I will mention because it changes what a rebase looks like, without reading anything into it: at the heads And one piece of ours that may be useful to you: our sidecar is version 2 and still reads version 1, so it accepts The decision is the maintainer's, and either route reaches the same place. |
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
Builds on #1480 by @routhjim, which stacks #1271 by @ANBAL534 and #1269 by @pspranger-throw. This branch sits on #1480's current head (8c0a5af, on v0.1.41) with their commits unchanged; mine are the three on top (d9beea0 and 86ec596 for the code, 42cb955 for the numbers in DETAILS.md). If #1480 lands first, this rebases down to those three.
I had a similar disk tier running locally for a while. When I compared it with yours, the session-file base and the streamed save and restore were better than mine, so I moved the parts of mine that still helped onto your branch instead of opening a competing PR.
What changes
spill_async). Before, it was written and flushed inside the request that caused the eviction. The writer thread only callssession_file_write, and the index is only touched on the main thread:wait()joins the write and indexes the file. This happens before a disk match, before a streamed save, and at shutdown, so a file is never matched before it is complete. A failed write removes its file.--conversation-cache-disk-onlywrites it. A disk hit larger than the RAM budget is streamed back the way that mode reads it. This applies on one GPU, without--batchor--peer-device, like disk-only.put()calledmake_roomwithout the spill callback. An image larger than its estimate could evict conversations that were then lost. They now go to the disk tier too, and so does an imageput()refuses.make_roomcould not free the room or the physical-RAM admission said no, so the request fell back to a prefix or a full read. It is streamed back now. I hit this on my own box: a 166K-token conversation came back with only its 46K prefix and re-read 122K tokens (99 s).--conversation-cache-disk-min-tokens N(default 0, off). Conversations shorter than N are not written.--session-min-free-mibfree, as SAVE does.With the defaults and the RAM cache off, behaviour is the same as #1480.
Measured
Measured after the rebase onto v0.1.41: #1480 at 8c0a5af against this branch at 86ec596, same machine, same flags.
Setup. RTX 2080 Ti 22 GB, i5-12490F, 62 GB RAM, Ubuntu, NVMe. Swift 1.5 IQ3_S,
--kv int8 --kv-resident 32768 --spec 4 --mtp.Workload. Three agent-style conversations take turns with
--conversation-cache-slots 1. They share a system prompt; each first prompt is about 16K tokens, and each later turn adds about 2.2K. There are 3 rounds, an engine restart (SIGTERM), then 2 more rounds. Decoding is greedy with 24 output tokens. The table shows prompt-read time for the 12 later turns.--conversation-cache-mib 4096--conversation-cache-mib 4096--conversation-cache-mib 256--conversation-cache-mib 256serve/server.py's 10 s wait before it kills the engine was never reached.Tests.
conversation_spill_testhas 34 new checks of mine; on the rebased branch all 107 pass (93 before the rebase, also under ThreadSanitizer). The checks cover the background write, the index waiting for it, branching at the root,put()evicting through the callback, waiting in the destructor, and refusal on the free-space floor. The other conversation tests pass. Before the rebase it also built on Windows (CUDA 12.6, VS 2022) with the tests passing there; I've been running it on a 2080 Ti and a 3080 since.Not included
The discussion in #1331 (@shahrokhzargarpour) and #1489 (@CC-David-CC) also helped shape this. Thanks to all of you. Happy to rebase or split it differently if that suits the stack better.
🤖 Generated with Claude Code