[ravel7524]

Overview

This PR clears a slot when a cache_prompt = false task ends and skips its prompt-reuse checkpoints. This prevents automatic RAM-cache saves of that task's own state and avoids checkpoint work on recurrent, hybrid and SWA models.

This extends the clear-on-release approach used for stateless tasks in #30498, without adding per-slot state.

Before overwriting the slot, a no-cache task saves the displaced conversation when RAM caching is available. It also skips loading cached prompts it cannot use.

Policy change: later requests cannot reuse a no-cache task's state, even on full-attention models, and a subsequent explicit slot save finds an empty slot. With --cache-ram 0, even a failed no-cache request can discard the previous conversation. The new behavior is documented in tools/server/README.md.

Benchmark Summary

• AMD Radeon PRO W7800 48 GB (gfx1100 / RDNA3); Vulkan, RADV / Mesa 26.0.8-1ubuntu0.3, kernel 7.0.0-38-generic
• Qwen3.8 27B Q8_0 + MTP Q4_0 draft
• --ctx-size 32768 --cache-ram 20480 -b 2048 -ub 2048 --parallel 1
• -ctk f16 -ctv f16 --flash-attn on --spec-type draft-mtp --spec-draft-n-max 3
• Upstream 0ce8cd114 versus 22e3b6236 (tested tree 9342ff0d); A-B-B-A
• Per build: 60 no-cache probes (n_predict = 1) and 12 following cache-enabled requests with unique 1.4-1.9k-token prompts; no cache hits

Measurement	Baseline	Candidate
Probe prompt time, p50	268.5 ms	109.0 ms (-59.4%)
Following request: slot selection to launch, median	153.96 ms	0.03 ms
Following request: wall time, median	2821.5 ms	2667.5 ms

Following-request prompt evaluation stayed similar: 2663.1 -> 2664.1 ms. A separate conversation follow-up after a probe reused 8400 tokens with identical output on both builds.

The policy has a measured cost: continuing a 2.9k-token no-cache prompt reprocessed 2880 tokens instead of reusing 2879; prompt time increased from 148 to 4496 ms in that check.

Testing: 28/28 CPU cases pass across test_kv_keep_only_active.py and test_slot_save.py. Four new tests fail on master; the checkpoint test also catches removal of the checkpoint gate. On GPU, a 2.9k-token no-cache request created 0 checkpoints versus 2 on master.

Requirements

• Yes, I have read and agree with the contributing guidelines.
• AI usage disclosure: YES, AI was used for implementation, tests, benchmarks, review. I reviewed the final diff and can explain all code changes.

[ravel7524]

Hi @ngxson, could you take a look at this cache_prompt=false change when you have time? I'd appreciate your feedback on the clear-on-release approach and the documented caching trade-off. Thanks!

[schuyler]

So it looks like the merge of master in 208449a dropped the release() change from 077c71f. #31269 edited the same condition to add SERVER_TASK_TYPE_REALTIME, and the merge kept master's version. As a result, a cache_prompt: false task's slot isn't cleared when the task ends, and its prompt is saved to the RAM cache when the next task takes the slot. Restoring || !task->params.cache_prompt alongside the REALTIME check fixes it.

[ravel7524]

Good catch. The merge in 208449aeb dropped the cache_prompt=false cleanup check in release(). c17aea23d restores it alongside the realtime check. Both regression tests now pass on the fixed build.