Repository navigation
Support bounded multi-conversation caching to avoid full prompt rereads in agent workflows #57
Description
I'm using Strata with Pi agent, where controller and worker conversations alternate against the same server. Multi-conversation caching made a substantial difference in a local prototype: my agent benchmark completed much more quickly.
Problem
The existing conversation cache works well for continuing one conversation, but switching between independent conversations can discard reusable state.
A typical sequence is:
- Process conversation A with a long history.
- Process an unrelated conversation B.
- Continue A with its previous history unchanged.
At step 3, the server may need to process A's history again. In my traces, three controller requests reread approximately 46.9K, 47.0K, and 47.6K tokens, spending roughly 143 seconds in prompt processing.
This is about preserving state across conversation switches, not avoiding the processing of genuinely new tool output.
Proposed feature
An opt-in, bounded host-RAM cache for inactive conversations:
- Match the longest exact token prefix, including image identity and steering mode.
- Preserve running state, checkpoints, used KV pages, indexer state, and draft-layer KV.
- Restore into existing GPU allocations without loading another model.
- Bound both memory usage and parked-conversation count, evicting the least recently active entries.
- Fall back to ordinary prompt processing when a snapshot cannot fit.
No client-specific session IDs or Pi changes should be necessary. Requests would still execute serially; this is not concurrent inference.
Working prototype and results
I've pushed the implementation to jeremiahritchey/Strata, branch feat/multi-conversation-cache, in commit 9d154ed, based on upstream commit b742ff9.
Testing used IQ3_S on an RTX 4090, with INT8 KV, a 262,144-token context, and 32,768 GPU-resident KV cells.
The prototype adds:
--conversation-cache-mib 8192
--conversation-cache-slots 4
These are prototype flags, not existing upstream options. The default budget is zero, leaving parking disabled.
In a synthetic A → B → A test:
- Initially processing A's 49,909 tokens took 48.809 seconds.
- Returning to A reused those 49,909 tokens and processed 22 new tokens in 1.961 seconds, including parking/restoration overhead.
- A's parked snapshot occupied approximately 1.26 GiB of host RAM.
These are prompt-processing measurements, not a general decoding-speed claim or a matched uncached A/B/A replay benchmark. My actual Pi benchmark also finished noticeably faster, but I haven't collected controlled end-to-end timing comparisons yet.
Validation passed for output and main-model state fingerprints, checkpoint recovery, byte/slot eviction, oversized-snapshot fallback, in-flight restore memory accounting, cancellation recovery, image identity/grid changes, and steering-mode isolation. Image tests used synthetic embeddings to exercise the engine's cache behavior, not image-recognition quality.
The branch includes design and validation notes, CPU/GPU component tests, and reproducible private-engine and HTTP test harnesses. Model-level harnesses default to a dry run and require an explicit --run.
Would you be interested in this as an opt-in upstream feature? I'd welcome feedback on the approach and can share the synthetic test logs or prepare a pull request if useful.
Activity
Niko1221 commentedon Sep 28, 2026
Thanks @jeremiahritchey - yes, I'm interested, and the numbers (49 s -> 2 s on the return to A) are exactly the problem agent users hit.
There are now three efforts on the same state: #41 (@midhatn: one primary conversation parked in RAM around calls marked auxiliary), #52 (@maedoc: whole-session snapshots on NVMe, surviving restarts) and this one (several conversations in RAM, least-recently-used first, matched by prefix with no client changes). They all capture the same thing - QSA KV incl. the streaming maps and host copies, the indexer, GDN/PLE state, the MTP ring, the checkpoints - so merging them separately would mean three copies of the most delicate code in the serve loop. I'd like to take one design, and I'd suggest:
kv_nvmeand Support bounded multi-conversation caching to avoid full prompt rereads in agent workflows #57's slots both have most of this; serve: preserve primary conversation state across auxiliary calls #41's review listed what must be in it.--max-context(only the used pages), and admission from free RAM on Windows and Linux.What I'd need to merge it: one PR against current main (0.1.16 changed the prompt path, the stdin reader and added the Coder's 256-expert geometry), the three authors agreeing on the split above (or a better one), tests that check restore exactness (the state hash) and the answer text across A -> B -> A, including with KV streaming and images, and a before/after on a real agent run.
@midhatn @maedoc - would you be up for converging on this? #52's NVMe tier on top of #57's RAM tier looks like the natural shape; happy to review the design doc before the code.
midhatn commentedon Sep 28, 2026
@Niko1221 @jeremiahritchey @maedoc — yes, I am up for converging on this. One capture/restore core, #57's RAM policy, and an optional NVMe tier is the direction I support. I am happy for #41 to be superseded once a combined implementation covers its use case and passes validation; there is no need to preserve a separate auxiliary-only snapshot mechanism merely to keep that PR independent.
My proposed contribution is Windows memory admission review and generic agent acceptance tests, together with the complete-state checklist and tests from #41. This is a proposed split, not an assignment of work to the other authors. Below is a concrete design/validation proposal for review before implementation.
Shared design proposal
GlobalMemoryStatusExon Windows andMemAvailableon Linux. Allow a configurable free-RAM floor; unavailable telemetry should skip admission. The cache budget must include staging/transfer buffers or expose and enforce a separate explicit bound for them. Skip/evict/spill instead of relying on allocation failure to prevent paging.--prompt-cache 0prevents activation. Expose hits, reused tokens, capture/restore time, byte occupancy, evictions, admission skips and disk bytes/time, so real agent runs can be diagnosed without dumping conversation content.Two observations from the current prototypes motivate the memory points: #57 already counts the incoming image during an exchange, but the inspected parking path does not yet check system free RAM; #52 currently buffers the entire snapshot file in RAM during restore, as its design notes acknowledge. Both matter when inference and snapshot buffers share limited physical RAM.
Proposed acceptance contract
For deterministic fixtures, require exact answer text as well as a complete restored-state fingerprint. For normal speculative/adaptive runs, also use independent task validators and report output differences; do not treat an equal state hash alone as proof that the task succeeded. A 30-cycle soak is a proposed starting gate, not proof of indefinite stability.
Status
This is a design and acceptance proposal, not a claim that the combined implementation already exists or has passed these gates. The existing #41 patch has passed bounded compatibility checks against v0.1.16, as described in its follow-up comment; those checks do not cover unmarked multi-conversation caching, disk persistence or the proposed soak.
For hardware coverage, a Ryzen 9 7940HS with RTX 4070 Laptop 8 GB and 64 GB RAM is available as a constrained-memory Windows test case. Cache capacity must be selected through physical-memory admission rather than copied from a larger machine's configuration.
I will keep #41 in draft as the implementation/evidence reference while we agree on the shared design and integration owner. No unified branch or consensus from the other authors is implied yet. This avoids pushing a competing rebase while the convergence discussion is open.
maedoc commentedon Sep 28, 2026
yes ofc 👍
the design and most of implementation in #52 was done with strata itself (a bit of help from glm-5.3), the design is a bit simpler minded than that, trying to get it working before the day was over.
jeremiahritchey commentedon Sep 28, 2026
@Niko1221 @midhatn @maedoc — yes, I support converging on one capture/restore core, the automatic bounded RAM policy from #57, and an optional NVMe tier. Keeping three independent implementations of session restoration would be difficult to maintain.
I can contribute the RAM policy, the existing snapshot/parity harnesses, and work toward extracting the complete capture/validation/restore boundary for review. That is an offer of work, not an assumption that I own the combined integration. @midhatn's proposed Windows admission review and acceptance-test coverage would be very welcome. We should agree on the shared interface/design before wiring the disk tier into it.
An update on evidence: locally I merged upstream
d551edf(engine 0.1.15), then reran all eight private full-model gates and the 12-request HTTP smoke test. Single-token and long speculative A/B/A, checkpoint recovery, byte pressure, oversized entries, in-flight exchange budgeting, synthetic image identity/grid changes, and add/project steering isolation passed. HTTP cancellation and subsequent cached recovery passed without restarting the frontend or engine.In the new synthetic long-context run, A had 51,133 tokens: initial processing took 46.558 s; returning after B reused 51,133 tokens and processed 22 new ones in 1.363 s including parking/restoration. Checkpoint recovery reused 51,148 tokens and processed seven in 0.580 s. Restored output and main-model state fingerprints matched the uninterrupted baseline. This is not a matched speed comparison against the old binary or a controlled real-agent before/after benchmark.
The gaps identified above are real:
I'll publish the validated branch update and bring a separate development branch onto current main (which has moved beyond 0.1.15), keeping the validation claims tied to the tested revision. I can start with a reviewable shared-core design, admission checks, and failure/geometry tests without introducing another persistence implementation. Would that split work, and is there a preferred owner/branch for the eventual unified PR?
jeremiahritchey commentedon Sep 28, 2026
I've published an initial shared-core/interface proposal on a separate development branch based on upstream
b38c183(0.1.18). The previously validated 0.1.15 implementation and results are now also pushed onfeat/multi-conversation-cache.The proposal separates snapshot capture/validation/restore from RAM admission/LRU and the future disk adapter. It explicitly distinguishes invalid-image rejection (before writes) from a CUDA transfer failure (fail closed, not a claim of rollback), and lists disk identity/staging/durability questions for review. No combined PR or integration ownership is assumed.
One audit correction worth highlighting: the original #57 snapshot did not preserve the indexer's per-sequence spare key (
idx_dead), and its state fingerprint did not include it. Fixtures sharing opening tokens can mask that omission. I'm adding that state, spare-row reconstruction on checkpoint rewind, and a deliberately different-spare-key regression. Earlier parity evidence should not be read as coverage of those previously unmeasured fields.Admission, validation and expanded regression changes are being tested separately before publication. Feedback on the proposed boundaries is welcome, especially how #52 would consume the snapshot without unbounded promotion staging.
jeremiahritchey commentedon Sep 28, 2026
The RAM/shared-core development pass is now available on feat/conversation-cache-shared-core, based on upstream
b38c183(0.1.18). The implementation commit is3657b8f; revision-specific evidence and limitations are recorded separately.This extracts capture/whole-image validation/restore from the serve loop, preserves the indexer spare-key state noted above, and adds Linux/Windows physical-memory admission with unknown telemetry declining admission. Invalid images are rejected before writes; transfer failures are a distinct fatal outcome, not a rollback claim. The NVMe adapter and persistent identity/format are intentionally not implemented here.
Validation on Linux/RTX 4090/IQ3_S includes nine paired full-model gates: single-token and long speculative reuse/checkpoints, byte pressure, oversized entries, incoming/outgoing exchange, synthetic image identity/grid isolation, both steering modes, and deliberate physical-memory denial. The denial test continued inference with matching outputs/state. Component fixtures also cover zero-QSA, 256/512-expert metadata, FP16/INT8/Q4 and distinct spare keys; host fault injection and ASan/UBSan passed.
The new known-answer soak also passed: 30 returns across 2,026 / 39,985 / 119,987-token contexts, with expected secret-word answers, exact baseline output/state parity, and stable retained payload per context. This covers streamed KV and 91.5% of configured maximum context; it is not a peak-memory or indefinite-stability claim.
The separate 12-request HTTP smoke passed with normal adaptive settings, including slot eviction, streaming disconnect and cached recovery; frontend/engine process identities remained unchanged throughout.
At 51,133 cached tokens, returning after B reused the full prefix and processed 22 new tokens in 1.237 s; checkpoint return processed seven new tokens in 0.566 s. Outputs and main-model fingerprints matched the uninterrupted baseline. This is synthetic evidence, not a version-to-version speed comparison or a controlled real-agent benchmark.
@midhatn: the admission helper and deny-admission harness are ready for Windows/constrained-memory review. @maedoc: the proposed shared image/core is available for interface feedback before wiring in disk promotion/spill. There is no competing unified PR or ownership assumption. Remaining work includes Windows runtime coverage, a real vision encoder smoke, full-model Coder coverage, the controlled Pi/Hermes task, and the agreed bounded-staging/persistence design.
Niko1221 commentedon Sep 28, 2026
Great to see everyone agree - thank you all. @jeremiahritchey, please own the unified PR: open it from
feat/conversation-cache-shared-corerebased onto 0.1.19, with @midhatn reviewing the Windows memory admission and bringing the acceptance tests (A->B->A, KV streaming, images/steering, memory pressure, cancellation, the 30-cycle soak, a real agent before/after). @maedoc, once the core lands, could you port the NVMe tier onto its snapshot format as spill-when-evicted (#47)?Separately, for 0.1.20 I'm merging #62 with #65's end-of-system-prompt checkpoint folded in: that's about which checkpoints one conversation keeps, and it doesn't overlap with the shared core.
midhatn commentedon Sep 28, 2026
That split works for me. Thanks for taking the lead, @jeremiahritchey. I'll review Windows memory admission and run the acceptance checks against the rebased branch, with results tied to the exact commit. The distinct spare-key regression is a useful catch; I'll make sure the Windows tests include it too.
First Windows check on
64bb557: the 23 admission tests pass under MSVC, and a separate live probe confirmsGlobalMemoryStatusExreturns usable physical-memory data and impossible allocations are denied. This only covers the admission helper; full restore and agent tests are still pending.midhatn commentedon Sep 29, 2026
I've sent the Windows admission test as a small PR against the shared-core branch: jeremiahritchey#1
It catches one test gap: unknown telemetry was allowed to pass the real-provider check even on Windows. The new check requires a known physical-memory sample, without assuming how much RAM is free or allocating pressure memory. MSVC run against
64bb557plus the test: 25 checks passed.This is admission-helper coverage only. Full restore, streamed-KV, image/steering, cancellation, soak and real-agent comparisons still need their own acceptance results. I'll keep #41 as the reference draft rather than create another snapshot implementation.
kv-nvme, generate: one pooled-row formula, and the state hash now spa…
core, kv-nvme: a transfer failure stops the engine; a refusal re-pref…
maedoc commentedon Sep 29, 2026
i think i've got the nvme stuff working on top of this api now, and i'm running some tests. the size of my pr (#52) will go down dramatically once the propose api is merged i guess.
Niko1221 commentedon Sep 29, 2026
0.1.22 is out: @jeremiahritchey, whenever the unified PR is ready, please base it on main (0.1.22). Two new things the snapshot must cover: with a layer split (several GPUs) each card holds its own session state (KV, indexer, GDN) for its layers, and the prompt path's chunked indexer append keeps the same per-sequence spare key you already handle. The multi-GPU part can come later if it's simpler to land the single-GPU core first.
hendrikp commentedon Sep 29, 2026
I also wonder in subagent use case what about the experts, woultd it make sense to remember which experts were last active per conversation and/or possibly restore that when a subagent finishes/handsoff. I guess it'll depend could go in either direction (e.g. a subagent reads info/explores - then that also means in turn those topics later on the main convo needs).
Also might be interesting to have different presets of expert profiles that could be initialized/restored as initial state in general if workload is known.
Niko1221 commentedon Sep 29, 2026
@jeremiahritchey, whenever the unified PR is ready, please base it on 0.1.26. Three things have landed since the plan, and the snapshot and its state hash need to cover them (single-GPU first is fine):
--kv k8v4(int8 K + q4 V);#143 (OpenClaw re-reading 150K+ prompts) is the same pain, so this is high on our list.
QilinWan commentedon Sep 29, 2026
No PR from us here, and the unified core is @jeremiahritchey's to own - this is review input only, on
one clause that I can't find a written criterion for yet (happy to be told it exists and I missed it).
Niko's condition for the disk tier says snapshots must be "tied to the model and quant, not only the
geometry". We'd like that clause to become a validation step with named fields, because in our runs
it's the one that fails silently rather than loudly.
1. Suggest the snapshot be addressable by a state-layout identity key, not a model name. Fields
that can change what a snapshot's bytes mean while leaving
general.architectureand the geometryuntouched:
block_count,embedding_length,expert_count,expert_used_count,attention.head_count{,_kv});token_embd,outputand the PLE table - since the per-expert blob size follows the type, and--expert-cachesizes slots per layer rather than to the largest expert;int8/q4_0),--kv-resident,the layer split and device assignment,
--prefillchunk size,--spec, the steering/control-vectormode, the turn token;
2. A disagreement would be better as a refusal than a partial load. Fail closed, re-read the
prompt, and log the first field that disagreed. The failure mode we worry about is not a crash: a
restored-but-misaligned state is finite, correctly sized and wrong, and that shape of wrong passes
every size check and every finite-value check.
3. Two fixtures sharing an opening prefix should be provably unable to restore onto each other.
@jeremiahritchey's
idx_deadaudit correction is the same class of problem from the other side - ashared prefix hides an uncovered field. Worth a regression whose fixtures differ only in a field the
snapshot didn't previously carry.
4. Might it be worth naming the fields whose meaning depends on a coordinate system? Under a ring
or a non-identity page table, selection ranks and physical page/slot ids are different number
spaces; anything stored in one space and restored into the other is silently wrong. We hit exactly
this once on our own tree and had to fix it (a ranked-coordinate correction for a ring-style window,
plus per-page host->device materialization that we then verify byte-for-byte against a paired dense
baseline). If it's already in the interface doc, ignore this item.
Context on who we are, for weighting: we run this same model (
qwen4exp) on our own llama.cpp-basedengine, with a host-side full-history page store and a device working set. Snapshot and residency
correctness is our main failure surface, not a side feature we touch occasionally. No code is offered
in this comment; if item 3 would be useful as a small fixture/regression against the unified branch,
say so and we'll send it as its own piece.
Disclosure: this comment was drafted with AI assistance (LLM-assisted writing and code search) and
reviewed by a human; the claims about our own measurements are ours to reproduce.
midhatn commentedon Sep 29, 2026
Updated and retested against 0.1.27. One useful result for the shared core: on Windows, Ryzen 9 7940HS / RTX 4070 Laptop 8GB / 64GB RAM, with Swift IQ3_XXS in the new low-RAM mode:
This is bounded evidence, not a full agent benchmark: 64K was configured, but the longest test prompt was 3,223 tokens. File-cache warmth varied, so I'm not using the decode rates to compare binaries.
For now the local patch is restricted to one CUDA GPU and INT8 KV. I haven't validated K8V4, HIP, or the new batched MTP prompt path; this profile uses streamed MTP. Those still need the coverage Niko listed in the unified core. I also agree with validating the state-layout identity before restoring anything, including model/quant identity and page/ring coordinates. The existing Windows admission tests remain available on the shared-core branch.
jeremiahritchey commentedon Sep 29, 2026
The local implementation is based on 0.1.26. Full GPU/model validation is still pending.
Thanks @midhatn for the Windows admission contribution and test results, @maedoc for the NVMe work informing this integration, and @QilinWan for the compatibility review.
Niko1221 commentedon Sep 30, 2026
Thanks, all. @jeremiahritchey: #189 is next in our queue. 0.1.29 adds no session state, so the snapshot fields stay the same; we'll restore the old meaning of
pooled=in STATE_HASH ourselves (the new extent gets its own key) and run our gates, including Windows. @QilinWan: agreed, validate a layout-identity key before any byte is applied (geometry, per-layer quant census, KV format,--kv-resident, split/devices, chunk, spec, steering, engine version), fail closed, log the first field that differs, and re-read the prompt. @midhatn: thanks for the 0.1.27 numbers. @hendrikp: the default expert cache is ranked from the profile and never evicts, so it holds no per-conversation state to save or restore.kv-nvme, generate: one pooled-row formula, and the state hash now spa…
core, kv-nvme: a transfer failure stops the engine; a refusal re-pref…
kv-nvme, generate: one pooled-row formula, and the state hash now spa…
core, kv-nvme: a transfer failure stops the engine; a refusal re-pref…
Niko1221 commentedon Sep 30, 2026
The RAM tier shipped in 0.1.30 (#189,
--conversation-cache-mib, off by default); the disk tier continues in #190. Closing the tracker.generate: make a dump describe the file it writes, and refuse where i…