[KeyanHu-git]

Problem and result

A client that pins the same system prompt on each request can fill every conversation-cache slot with a different branch of that prefix. Upstream protects every holder from eviction, so later conversations cannot be parked even when the byte budget has room.

This change lets redundant holders leave while keeping every pinned prefix recoverable. On 0.2.2, the same 15-request sequence produced these results:

	Upstream	Patched
Refused parking attempts	8	0
Restored conversations	0	6
Input tokens processed on six revisits	1,778	90
Output token mismatches	—	0

Related discussion

Fixes #2255. The repeated-pin case reported there remains reproducible on current main. Related admission work: #2253 and #2082; this PR supplies cache-owned reclaimable-byte accounting for those callers.

Approach

Evict unpinned entries first, then the oldest entry whose pinned prefixes all have another holder. Prefix identity includes token IDs, image identities and steering mode. A borrowed donor stays protected and counts as a surviving holder.

flowchart LR
    A[Old branch A] --> P[Shared pinned prefix]
    B[Branch B] --> P
    C[Branch C] --> P
    A --> E[Evict A to admit a new branch]
    B --> K[Prefix remains recoverable]
    C --> K

Loading

evictable_bytes() simulates the same removals as eviction, so it stops counting a prefix's last holder once earlier copies have gone. The borrow precheck uses that result before making room. Retained active K/V is accounted separately. No cache settings, budgets or snapshot formats change.

The ownership rule is shared by slot/byte admission, physical-memory reclamation and session saving. The additional session-save regression checks that reclamation stops when enough memory is available, preserves a borrowed donor, and leaves the selected checkpoint's content and storage intact without allocating memory.

Validation

Base 011c9e60 (0.2.2 / engine 0.1.47); production patch af1e09dd. This update adds tests, documentation and recorded evidence; the production headers are unchanged. Results, inputs, reproduction commands and source hashes.

Passed:

• Windows 11, Ryzen 5 7600X, RX 7900 XTX, Coder IQ1_M: 15 fixed requests per arm, three cache slots, a 4 GiB cache budget and fixed expert-placement settings. All output IDs match; parking and reuse counts are in the table. Snapshot verification was enabled, so this is a correctness/reuse result rather than a speed claim.
• Cache-policy tests under ASan/UBSan: 45,153 checks, covering shared/distinct pins, images, steering mode, guarded donors and preview-versus-actual reclamation.
• Session-save tests: 161 checks under Apple Clang with ASan/UBSan, Windows HIP Clang and Intel icpx. The new shared-pin reclamation case fails on unmodified main.
• Full CUDA (sm_120), HIP (gfx1100), SYCL and Metal engine builds. The corrected CUDA build also passes both cache and session-reclamation ctests.
• git diff --check and recomputation of the recorded result table.

Failed: the first CUDA invocation omitted the conversation-test build option and could not find that target. The corrected command completed the engine build and both tests successfully.

Not run: CUDA, SYCL and Metal model execution; the real-model comparison was on HIP. The portable replay driver was checked with a protocol fixture; the recorded model run used the original sequential driver with the same commands and settings.

Limits

Eviction searches for redundant holders after no unpinned victim is available; guarded admission and reclaimable-byte queries also simulate this policy. The work depends on entries, checkpoints and prefix length. The real-model evidence covers one GPU and serial requests; it establishes restored cache reuse and matching outputs for that workload. The PR does not change the active checkpoint-retention policy or make physical-memory reclamation equal to logical freed bytes.

[qiuzan2001]

The eviction rule reads right to me (every pinned prefix keeps one holder, and each eviction re-checks). Two follow-ups, found by reading (C++ test not built here):

• docs/DETAILS.md:987 still says a pinned prefix means "a parked conversation holding it stays parked"; with this change only one holder per prefix is kept, as the updated header comment says.
• Open [Engine] serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394) #2253 and fix: make conversation parking admission fail safely #2082 add ConversationCache::evictable_bytes() ("What evict_oldest() could give back in all: everything parked that is not a pinned shared prefix") and refuse an admission when short_by > evictable_bytes(). After this PR evict_oldest() can also free redundant pinned holders, so that check would refuse parks that eviction could make room for. git merge-tree also shows a textual conflict with fix: make conversation parking admission fail safely #2082 in src/core/conversation_cache_test.cpp. Whichever lands second needs to count the redundant pinned holders as evictable.

[KeyanHu-git]

Thanks @qiuzan2001 for checking both interactions. The documentation fix is in 3e0780a.

You're right about evictable_bytes(). I reproduced the mismatch: with three holders of the same pin, the admission patches report zero evictable bytes, while this eviction rule can release two holders and retain the third.

The accounting needs to agree with repeated evict_oldest() calls. Simply summing every holder that is currently redundant would overcount: after earlier removals, the last holder must become non-evictable. I tested local integration prototypes for both #2253 and #2082 using the same pin-matching rule and tracking simulated removals without copying snapshot payloads. Both combined CPU test suites pass under ASan/UBSan, including 4,096 pin-membership combinations checking that the estimate matches actual eviction and preserves every pin. The same added tests fail with the original unpinned-only accounting. These are local CPU integration checks; the follow-up code still needs to be integrated and reviewed in whichever PR lands second. I also confirmed the textual conflicts and retained both sets of tests in the local combinations.

Since neither admission API is on main yet, I suggest landing this focused fix first, then updating the accounting in the admission PR that follows. If an admission PR lands first, #2443 will need that adjustment before merging.

@Niko1221, could you review the current fix and let us know whether that landing order works for you?

[KeyanHu-git]

Follow-up: the cache-side accounting is now implemented in 613f78f. Keeping it in the cache defines the eviction contract in one place, while the admission policies remain in their respective PRs.

evictable_bytes() shares the pin-matching rule with eviction and tracks simulated removals, so it excludes retained K/V and the final holder of each pin. The query does not mutate the cache or copy snapshot payloads; up to 64 entries need no heap allocation. I also corrected the comment about memory release: logical evictable bytes do not guarantee an immediate RSS reduction.

The updated suite passes 44,581 checks on macOS, Windows and Linux, and full CUDA/HIP/SYCL builds pass. Local combinations with #2253 and #2082 pass 44,641 and 44,651 CPU checks under ASan/UBSan, including admission by evicting a redundant pinned holder. The earlier real-model comparison remains from 44877b3; this follow-up was validated through CPU tests and backend builds.

When rebasing #2253 or #2082, please retain this implementation instead of the unpinned-only definition, keep a single copy of the method, and preserve both sets of regression tests. The textual merge conflicts still need resolution. The PR description now records the implementation and validation details.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[KeyanHu-git]

/reopen

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.