Skip to content

Support bounded multi-conversation caching to avoid full prompt rereads in agent workflows #57

Closed

Description

@jeremiahritchey

I'm using Strata with Pi agent, where controller and worker conversations alternate against the same server. Multi-conversation caching made a substantial difference in a local prototype: my agent benchmark completed much more quickly.

Problem

The existing conversation cache works well for continuing one conversation, but switching between independent conversations can discard reusable state.

A typical sequence is:

  1. Process conversation A with a long history.
  2. Process an unrelated conversation B.
  3. Continue A with its previous history unchanged.

At step 3, the server may need to process A's history again. In my traces, three controller requests reread approximately 46.9K, 47.0K, and 47.6K tokens, spending roughly 143 seconds in prompt processing.

This is about preserving state across conversation switches, not avoiding the processing of genuinely new tool output.

Proposed feature

An opt-in, bounded host-RAM cache for inactive conversations:

  • Match the longest exact token prefix, including image identity and steering mode.
  • Preserve running state, checkpoints, used KV pages, indexer state, and draft-layer KV.
  • Restore into existing GPU allocations without loading another model.
  • Bound both memory usage and parked-conversation count, evicting the least recently active entries.
  • Fall back to ordinary prompt processing when a snapshot cannot fit.

No client-specific session IDs or Pi changes should be necessary. Requests would still execute serially; this is not concurrent inference.

Working prototype and results

I've pushed the implementation to jeremiahritchey/Strata, branch feat/multi-conversation-cache, in commit 9d154ed, based on upstream commit b742ff9.

Testing used IQ3_S on an RTX 4090, with INT8 KV, a 262,144-token context, and 32,768 GPU-resident KV cells.

The prototype adds:

--conversation-cache-mib 8192
--conversation-cache-slots 4

These are prototype flags, not existing upstream options. The default budget is zero, leaving parking disabled.

In a synthetic A → B → A test:

  • Initially processing A's 49,909 tokens took 48.809 seconds.
  • Returning to A reused those 49,909 tokens and processed 22 new tokens in 1.961 seconds, including parking/restoration overhead.
  • A's parked snapshot occupied approximately 1.26 GiB of host RAM.

These are prompt-processing measurements, not a general decoding-speed claim or a matched uncached A/B/A replay benchmark. My actual Pi benchmark also finished noticeably faster, but I haven't collected controlled end-to-end timing comparisons yet.

Validation passed for output and main-model state fingerprints, checkpoint recovery, byte/slot eviction, oversized-snapshot fallback, in-flight restore memory accounting, cancellation recovery, image identity/grid changes, and steering-mode isolation. Image tests used synthetic embeddings to exercise the engine's cache behavior, not image-recognition quality.

The branch includes design and validation notes, CPU/GPU component tests, and reproducible private-engine and HTTP test harnesses. Model-level harnesses default to a dry run and require an explicit --run.

Would you be interested in this as an opt-in upstream feature? I'd welcome feedback on the approach and can share the synthetic test logs or prepare a pull request if useful.

Activity

  1. Niko1221 commented on Sep 28, 2026

    @Niko1221
    Owner

    Thanks @jeremiahritchey - yes, I'm interested, and the numbers (49 s -> 2 s on the return to A) are exactly the problem agent users hit.

    There are now three efforts on the same state: #41 (@midhatn: one primary conversation parked in RAM around calls marked auxiliary), #52 (@maedoc: whole-session snapshots on NVMe, surviving restarts) and this one (several conversations in RAM, least-recently-used first, matched by prefix with no client changes). They all capture the same thing - QSA KV incl. the streaming maps and host copies, the indexer, GDN/PLE state, the MTP ring, the checkpoints - so merging them separately would mean three copies of the most delicate code in the serve loop. I'd like to take one design, and I'd suggest:

    1. The capture/restore core, once: one module that snapshots and restores the complete session state (atomically: validated before anything is applied, a failed restore falls back to a clean session). nvme kv cache #52's kv_nvme and Support bounded multi-conversation caching to avoid full prompt rereads in agent workflows #57's slots both have most of this; serve: preserve primary conversation state across auxiliary calls #41's review listed what must be in it.
    2. Tier 1 - RAM (this proposal): automatic longest-prefix match, a byte budget and a slot count, LRU eviction, off by default. That also covers serve: preserve primary conversation state across auxiliary calls #41's side-call case without a client marker.
    3. Tier 2 - NVMe (nvme kv cache #52), optional: an evicted snapshot spills to disk instead of being dropped, and is promoted back on a prefix match - so it survives restarts and isn't limited by RAM.
    4. Sized by the conversation, not by --max-context (only the used pages), and admission from free RAM on Windows and Linux.

    What I'd need to merge it: one PR against current main (0.1.16 changed the prompt path, the stdin reader and added the Coder's 256-expert geometry), the three authors agreeing on the split above (or a better one), tests that check restore exactness (the state hash) and the answer text across A -> B -> A, including with KV streaming and images, and a before/after on a real agent run.

    @midhatn @maedoc - would you be up for converging on this? #52's NVMe tier on top of #57's RAM tier looks like the natural shape; happy to review the design doc before the code.

  2. midhatn commented on Sep 28, 2026

    @midhatn
    Contributor

    @Niko1221 @jeremiahritchey @maedoc — yes, I am up for converging on this. One capture/restore core, #57's RAM policy, and an optional NVMe tier is the direction I support. I am happy for #41 to be superseded once a combined implementation covers its use case and passes validation; there is no need to preserve a separate auxiliary-only snapshot mechanism merely to keep that PR independent.

    My proposed contribution is Windows memory admission review and generic agent acceptance tests, together with the complete-state checklist and tests from #41. This is a proposed split, not an assignment of work to the other authors. Below is a concrete design/validation proposal for review before implementation.

    Shared design proposal

    1. One snapshot representation and capture/restore implementation. The RAM cache and disk store consume the same validated representation. Capture all required QSA/indexer/GDN/PLE/MTP state, streaming metadata and checkpoints, but only the pages required by the saved conversation/checkpoints. Retain existing graph-bound device addresses. No extra model instance or parallel inference is needed.
    2. Automatic prefix selection. Choose the longest compatible exact prefix across active state, RAM and optional disk entries; prefer the active state on ties. Images and control-vector identity must match too. Preserve a valid live resume point. A rewritten compression prefix remains a miss where it no longer matches. No client marker should be required; the marker used by serve: preserve primary conversation state across auxiliary calls #41 must not bypass compatibility checks.
    3. Separate logical capacity from physical admission. Enforce byte/slot budgets, including retained checkpoints and snapshots temporarily held during exchange. Also check available physical RAM immediately before allocation: GlobalMemoryStatusEx on Windows and MemAvailable on Linux. Allow a configurable free-RAM floor; unavailable telemetry should skip admission. The cache budget must include staging/transfer buffers or expose and enforce a separate explicit bound for them. Skip/evict/spill instead of relying on allocation failure to prevent paging.
    4. Transactional restore behavior. Validate the complete snapshot, all extents and compatibility before applying any layer. Invalid/corrupt entries should be discarded and start clean. A transfer failure must never continue from partially restored state: reset fully only if the CUDA context remains usable; otherwise return an error and restart the engine safely. Validation alone cannot make a failed GPU transfer transactional.
    5. Optional disk tier with bounded staging. Spill on RAM eviction rather than writing every continuing turn by default. Bind persisted snapshots to the exact model/weights, tokenizer, geometry, KV format, image/control-vector identity and compatible state-schema version. Validate after restart. Use bounded buffers and account for save/flush/validation costs, not only reads. Interrupted writes must not become valid cache entries. Persistence policy should explicitly state when an active conversation is durable; eviction-only storage does not automatically preserve every latest turn.
    6. Keep the feature off by default. Warn when --prompt-cache 0 prevents activation. Expose hits, reused tokens, capture/restore time, byte occupancy, evictions, admission skips and disk bytes/time, so real agent runs can be diagnosed without dumping conversation content.

    Two observations from the current prototypes motivate the memory points: #57 already counts the incoming image during an exchange, but the inspected parking path does not yet check system free RAM; #52 currently buffers the entire snapshot file in RAM during restore, as its design notes acknowledge. Both matter when inference and snapshot buffers share limited physical RAM.

    Proposed acceptance contract

    Scenario Required evidence
    Cache disabled Existing behavior retained; no snapshot allocations or disk writes
    Unmarked A → B → A; A → worker → approval → A Compare with uninterrupted A using restore-state fingerprints and known-correct answer text; exercise live and checkpoint resume
    KV streaming Repeat above with histories exceeding resident KV, at multiple context lengths near the configured context limit
    Images and steering Real image encoder smoke plus synthetic state tests; changed pixels/grid/control vectors reject incompatible reuse
    Used-page sizing and geometry Short/long histories, partial pages, indexer boundaries, zero-QSA component fixture, canonical/pruned geometry fixtures; bounded checkpoints included
    Memory pressure Byte cap, slot cap, oversized entry, unknown/low available RAM, incoming + outgoing exchange and disk staging; inference survives denied admission
    Cancellation/error Disconnect during generation and test capture/restore failures; subsequent unrelated and resumed requests remain correct, or the engine restarts cleanly
    Disk persistence Save/restart/restore, eviction, truncated/corrupt/interrupted files and incompatible model/schema; no stale or mixed-state continuation
    Soak Proposed minimum: 30 A/B/A cycles across several context lengths with expected outputs, no silent failure and no growing retained allocation
    Real Hermes workload Same bounded task, starting files, prompts and tool-output limits with cache disabled/enabled; record verified success, total wall time, prefill/decode/auxiliary/tool time and peak memory

    For deterministic fixtures, require exact answer text as well as a complete restored-state fingerprint. For normal speculative/adaptive runs, also use independent task validators and report output differences; do not treat an equal state hash alone as proof that the task succeeded. A 30-cycle soak is a proposed starting gate, not proof of indefinite stability.

    Status

    This is a design and acceptance proposal, not a claim that the combined implementation already exists or has passed these gates. The existing #41 patch has passed bounded compatibility checks against v0.1.16, as described in its follow-up comment; those checks do not cover unmarked multi-conversation caching, disk persistence or the proposed soak.

    For hardware coverage, a Ryzen 9 7940HS with RTX 4070 Laptop 8 GB and 64 GB RAM is available as a constrained-memory Windows test case. Cache capacity must be selected through physical-memory admission rather than copied from a larger machine's configuration.

    I will keep #41 in draft as the implementation/evidence reference while we agree on the shared design and integration owner. No unified branch or consensus from the other authors is implied yet. This avoids pushing a competing rebase while the convergence discussion is open.

  3. maedoc commented on Sep 28, 2026

    @maedoc
    Contributor

    would you be up for converging on this?

    yes ofc 👍

    the design and most of implementation in #52 was done with strata itself (a bit of help from glm-5.3), the design is a bit simpler minded than that, trying to get it working before the day was over.

  4. jeremiahritchey commented on Sep 28, 2026

    @jeremiahritchey
    ContributorAuthor

    @Niko1221 @midhatn @maedoc — yes, I support converging on one capture/restore core, the automatic bounded RAM policy from #57, and an optional NVMe tier. Keeping three independent implementations of session restoration would be difficult to maintain.

    I can contribute the RAM policy, the existing snapshot/parity harnesses, and work toward extracting the complete capture/validation/restore boundary for review. That is an offer of work, not an assumption that I own the combined integration. @midhatn's proposed Windows admission review and acceptance-test coverage would be very welcome. We should agree on the shared interface/design before wiring the disk tier into it.

    An update on evidence: locally I merged upstream d551edf (engine 0.1.15), then reran all eight private full-model gates and the 12-request HTTP smoke test. Single-token and long speculative A/B/A, checkpoint recovery, byte pressure, oversized entries, in-flight exchange budgeting, synthetic image identity/grid changes, and add/project steering isolation passed. HTTP cancellation and subsequent cached recovery passed without restarting the frontend or engine.

    In the new synthetic long-context run, A had 51,133 tokens: initial processing took 46.558 s; returning after B reused 51,133 tokens and processed 22 new ones in 1.363 s including parking/restoration. Checkpoint recovery reused 51,148 tokens and processed seven in 0.580 s. Restored output and main-model state fingerprints matched the uninterrupted baseline. This is not a matched speed comparison against the old binary or a controlled real-agent before/after benchmark.

    The gaps identified above are real:

    • The current RAM prototype enforces its byte/slot limits but does not yet admit allocations based on available physical RAM.
    • Restore validates/applies individual KV layers and fails closed on transfer errors; it does not yet prevalidate the whole session before the first write, or provide a clean-session fallback for every invalid entry.
    • Our image tests use synthetic embeddings, not a real encoder smoke. We have not yet completed the proposed soak, new geometry coverage, or controlled agent-task comparison, and there is no disk-persistence claim.

    I'll publish the validated branch update and bring a separate development branch onto current main (which has moved beyond 0.1.15), keeping the validation claims tied to the tested revision. I can start with a reviewable shared-core design, admission checks, and failure/geometry tests without introducing another persistence implementation. Would that split work, and is there a preferred owner/branch for the eventual unified PR?

  5. jeremiahritchey commented on Sep 28, 2026

    @jeremiahritchey
    ContributorAuthor

    I've published an initial shared-core/interface proposal on a separate development branch based on upstream b38c183 (0.1.18). The previously validated 0.1.15 implementation and results are now also pushed on feat/multi-conversation-cache.

    The proposal separates snapshot capture/validation/restore from RAM admission/LRU and the future disk adapter. It explicitly distinguishes invalid-image rejection (before writes) from a CUDA transfer failure (fail closed, not a claim of rollback), and lists disk identity/staging/durability questions for review. No combined PR or integration ownership is assumed.

    One audit correction worth highlighting: the original #57 snapshot did not preserve the indexer's per-sequence spare key (idx_dead), and its state fingerprint did not include it. Fixtures sharing opening tokens can mask that omission. I'm adding that state, spare-row reconstruction on checkpoint rewind, and a deliberately different-spare-key regression. Earlier parity evidence should not be read as coverage of those previously unmeasured fields.

    Admission, validation and expanded regression changes are being tested separately before publication. Feedback on the proposed boundaries is welcome, especially how #52 would consume the snapshot without unbounded promotion staging.

  6. jeremiahritchey commented on Sep 28, 2026

    @jeremiahritchey
    ContributorAuthor

    The RAM/shared-core development pass is now available on feat/conversation-cache-shared-core, based on upstream b38c183 (0.1.18). The implementation commit is 3657b8f; revision-specific evidence and limitations are recorded separately.

    This extracts capture/whole-image validation/restore from the serve loop, preserves the indexer spare-key state noted above, and adds Linux/Windows physical-memory admission with unknown telemetry declining admission. Invalid images are rejected before writes; transfer failures are a distinct fatal outcome, not a rollback claim. The NVMe adapter and persistent identity/format are intentionally not implemented here.

    Validation on Linux/RTX 4090/IQ3_S includes nine paired full-model gates: single-token and long speculative reuse/checkpoints, byte pressure, oversized entries, incoming/outgoing exchange, synthetic image identity/grid isolation, both steering modes, and deliberate physical-memory denial. The denial test continued inference with matching outputs/state. Component fixtures also cover zero-QSA, 256/512-expert metadata, FP16/INT8/Q4 and distinct spare keys; host fault injection and ASan/UBSan passed.

    The new known-answer soak also passed: 30 returns across 2,026 / 39,985 / 119,987-token contexts, with expected secret-word answers, exact baseline output/state parity, and stable retained payload per context. This covers streamed KV and 91.5% of configured maximum context; it is not a peak-memory or indefinite-stability claim.

    The separate 12-request HTTP smoke passed with normal adaptive settings, including slot eviction, streaming disconnect and cached recovery; frontend/engine process identities remained unchanged throughout.

    At 51,133 cached tokens, returning after B reused the full prefix and processed 22 new tokens in 1.237 s; checkpoint return processed seven new tokens in 0.566 s. Outputs and main-model fingerprints matched the uninterrupted baseline. This is synthetic evidence, not a version-to-version speed comparison or a controlled real-agent benchmark.

    @midhatn: the admission helper and deny-admission harness are ready for Windows/constrained-memory review. @maedoc: the proposed shared image/core is available for interface feedback before wiring in disk promotion/spill. There is no competing unified PR or ownership assumption. Remaining work includes Windows runtime coverage, a real vision encoder smoke, full-model Coder coverage, the controlled Pi/Hermes task, and the agreed bounded-staging/persistence design.

  7. Niko1221 commented on Sep 28, 2026

    @Niko1221
    Owner

    Great to see everyone agree - thank you all. @jeremiahritchey, please own the unified PR: open it from feat/conversation-cache-shared-core rebased onto 0.1.19, with @midhatn reviewing the Windows memory admission and bringing the acceptance tests (A->B->A, KV streaming, images/steering, memory pressure, cancellation, the 30-cycle soak, a real agent before/after). @maedoc, once the core lands, could you port the NVMe tier onto its snapshot format as spill-when-evicted (#47)?

    Separately, for 0.1.20 I'm merging #62 with #65's end-of-system-prompt checkpoint folded in: that's about which checkpoints one conversation keeps, and it doesn't overlap with the shared core.

  8. midhatn commented on Sep 28, 2026

    @midhatn
    Contributor

    That split works for me. Thanks for taking the lead, @jeremiahritchey. I'll review Windows memory admission and run the acceptance checks against the rebased branch, with results tied to the exact commit. The distinct spare-key regression is a useful catch; I'll make sure the Windows tests include it too.

    First Windows check on 64bb557: the 23 admission tests pass under MSVC, and a separate live probe confirms GlobalMemoryStatusEx returns usable physical-memory data and impossible allocations are denied. This only covers the admission helper; full restore and agent tests are still pending.

  9. midhatn commented on Sep 29, 2026

    @midhatn
    Contributor

    I've sent the Windows admission test as a small PR against the shared-core branch: jeremiahritchey#1

    It catches one test gap: unknown telemetry was allowed to pass the real-provider check even on Windows. The new check requires a known physical-memory sample, without assuming how much RAM is free or allocating pressure memory. MSVC run against 64bb557 plus the test: 25 checks passed.

    This is admission-helper coverage only. Full restore, streamed-KV, image/steering, cancellation, soak and real-agent comparisons still need their own acceptance results. I'll keep #41 as the reference draft rather than create another snapshot implementation.

  10. maedoc commented on Sep 29, 2026

    @maedoc
    Contributor

    i think i've got the nvme stuff working on top of this api now, and i'm running some tests. the size of my pr (#52) will go down dramatically once the propose api is merged i guess.

  11. Niko1221 commented on Sep 29, 2026

    @Niko1221
    Owner

    0.1.22 is out: @jeremiahritchey, whenever the unified PR is ready, please base it on main (0.1.22). Two new things the snapshot must cover: with a layer split (several GPUs) each card holds its own session state (KV, indexer, GDN) for its layers, and the prompt path's chunked indexer append keeps the same per-sequence spare key you already handle. The multi-GPU part can come later if it's simpler to land the single-GPU core first.

  12. hendrikp commented on Sep 29, 2026

    @hendrikp
    Contributor

    I also wonder in subagent use case what about the experts, woultd it make sense to remember which experts were last active per conversation and/or possibly restore that when a subagent finishes/handsoff. I guess it'll depend could go in either direction (e.g. a subagent reads info/explores - then that also means in turn those topics later on the main convo needs).

    Also might be interesting to have different presets of expert profiles that could be initialized/restored as initial state in general if workload is known.

  13. Niko1221 commented on Sep 29, 2026

    @Niko1221
    Owner

    @jeremiahritchey, whenever the unified PR is ready, please base it on 0.1.26. Three things have landed since the plan, and the snapshot and its state hash need to cover them (single-GPU first is fine):

    • --kv k8v4 (int8 K + q4 V);
    • the AMD HIP backend;
    • the draft layer's batched prompt pass, which now writes the drafter's K/V during the prompt.

    #143 (OpenClaw re-reading 150K+ prompts) is the same pain, so this is high on our list.

  14. QilinWan commented on Sep 29, 2026

    @QilinWan

    No PR from us here, and the unified core is @jeremiahritchey's to own - this is review input only, on
    one clause that I can't find a written criterion for yet (happy to be told it exists and I missed it).

    Niko's condition for the disk tier says snapshots must be "tied to the model and quant, not only the
    geometry". We'd like that clause to become a validation step with named fields, because in our runs
    it's the one that fails silently rather than loudly.

    1. Suggest the snapshot be addressable by a state-layout identity key, not a model name. Fields
    that can change what a snapshot's bytes mean while leaving general.architecture and the geometry
    untouched:

    • the geometry keys already assumed (block_count, embedding_length, expert_count,
      expert_used_count, attention.head_count{,_kv});
    • the quant census of tensors that touch resident state - which gate/up/down type per layer, plus
      token_embd, output and the PLE table - since the per-expert blob size follows the type, and
      --expert-cache sizes slots per layer rather than to the largest expert;
    • flags that move bytes or change a coordinate system: KV format (int8/q4_0), --kv-resident,
      the layer split and device assignment, --prefill chunk size, --spec, the steering/control-vector
      mode, the turn token;
    • the engine version that produced the bytes.

    2. A disagreement would be better as a refusal than a partial load. Fail closed, re-read the
    prompt, and log the first field that disagreed. The failure mode we worry about is not a crash: a
    restored-but-misaligned state is finite, correctly sized and wrong, and that shape of wrong passes
    every size check and every finite-value check.

    3. Two fixtures sharing an opening prefix should be provably unable to restore onto each other.
    @jeremiahritchey's idx_dead audit correction is the same class of problem from the other side - a
    shared prefix hides an uncovered field. Worth a regression whose fixtures differ only in a field the
    snapshot didn't previously carry.

    4. Might it be worth naming the fields whose meaning depends on a coordinate system? Under a ring
    or a non-identity page table, selection ranks and physical page/slot ids are different number
    spaces; anything stored in one space and restored into the other is silently wrong. We hit exactly
    this once on our own tree and had to fix it (a ranked-coordinate correction for a ring-style window,
    plus per-page host->device materialization that we then verify byte-for-byte against a paired dense
    baseline). If it's already in the interface doc, ignore this item.

    Context on who we are, for weighting: we run this same model (qwen4exp) on our own llama.cpp-based
    engine, with a host-side full-history page store and a device working set. Snapshot and residency
    correctness is our main failure surface, not a side feature we touch occasionally. No code is offered
    in this comment; if item 3 would be useful as a small fixture/regression against the unified branch,
    say so and we'll send it as its own piece.

    Disclosure: this comment was drafted with AI assistance (LLM-assisted writing and code search) and
    reviewed by a human; the claims about our own measurements are ours to reproduce.

  15. midhatn commented on Sep 29, 2026

    @midhatn
    Contributor

    Updated and retested against 0.1.27. One useful result for the shared core: on Windows, Ryzen 9 7940HS / RTX 4070 Laptop 8GB / 64GB RAM, with Swift IQ3_XXS in the new low-RAM mode:

    • A 3,223-token lookup, interrupted by three unrelated auxiliary calls, reread the prompt each time on the official binary: zero tokens reused, 10.7–11.9 seconds per return.
    • With the local auxiliary snapshot patch, each return reused 3,216 tokens and completed in 0.8–1.4 seconds, with the correct answer in both cases.
    • The deployed build also passed tool call/result checks, image restoration, changed-image invalidation and rewritten-history checks. An earlier run correctly declined an image snapshot under RAM pressure and produced the right answer through a cold reread.

    This is bounded evidence, not a full agent benchmark: 64K was configured, but the longest test prompt was 3,223 tokens. File-cache warmth varied, so I'm not using the decode rates to compare binaries.

    For now the local patch is restricted to one CUDA GPU and INT8 KV. I haven't validated K8V4, HIP, or the new batched MTP prompt path; this profile uses streamed MTP. Those still need the coverage Niko listed in the unified core. I also agree with validating the state-layout identity before restoring anything, including model/quant identity and page/ring coordinates. The existing Windows admission tests remain available on the shared-core branch.

  16. jeremiahritchey commented on Sep 29, 2026

    @jeremiahritchey
    ContributorAuthor

    The local implementation is based on 0.1.26. Full GPU/model validation is still pending.

    1. S1 — Implementation: RAM and optional NVMe caching use the same snapshot and restore code. Disk writes happen on RAM eviction, with bounded staging, compatibility and integrity checks, atomic writes, and quotas.
    2. S2 — Validation: Host tests pass for serialization, corruption, restart behavior, eviction, and memory admission. Next are a linked engine build and GPU/model tests covering K8V4, batched draft KV, and disk restart/restore parity. Earlier model results apply only to the revisions tested. I'm also reviewing the state-layout identity feedback.
    3. S3 — Hardware coverage: I can test Linux with one NVIDIA GPU. I don't have Windows or multi-GPU hardware. Those paths need contributor testing, and HIP remains untested locally.

    Thanks @midhatn for the Windows admission contribution and test results, @maedoc for the NVMe work informing this integration, and @QilinWan for the compatibility review.

  17. Niko1221 commented on Sep 30, 2026

    @Niko1221
    Owner

    Thanks, all. @jeremiahritchey: #189 is next in our queue. 0.1.29 adds no session state, so the snapshot fields stay the same; we'll restore the old meaning of pooled= in STATE_HASH ourselves (the new extent gets its own key) and run our gates, including Windows. @QilinWan: agreed, validate a layout-identity key before any byte is applied (geometry, per-layer quant census, KV format, --kv-resident, split/devices, chunk, spec, steering, engine version), fail closed, log the first field that differs, and re-read the prompt. @midhatn: thanks for the 0.1.27 numbers. @hendrikp: the default expert cache is ranked from the profile and never evicts, so it holds no per-conversation state to save or restore.

  18. Niko1221 commented on Sep 30, 2026

    @Niko1221
    Owner

    The RAM tier shipped in 0.1.30 (#189, --conversation-cache-mib, off by default); the disk tier continues in #190. Closing the tracker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions