serve: optionally checkpoint an existing chunk near the prompt tail

[imanu86]

A branched request can share almost the whole previous prompt but diverge before its final cached state, leaving only an earlier periodic checkpoint usable. Optional --prompt-cache-tail saves at most one extra checkpoint near the prompt's end, on a chunk boundary the existing batched prefill already reaches. It defaults to off, uses the existing bounded cache, and does not split chunks or advance the periodic schedule early. Reread diagnostics and layer splits retain their existing behavior.

For a 33823-token prompt, 6144-token chunks and a 16384-token periodic interval, a branch sharing 32769 tokens can reuse 30720 rather than 18432 tokens. The benefit depends on prompt/branch position and cache capacity.

Validation against pristine upstream 0.1.38 (99f3dbd) on a modified RTX 2080 Ti 22 GB, KV capacity 262144, int8, 32768 resident cells:

• Pristine / tail-enabled / pristine-repeat, fixed placement and chunk geometry: 2560 teacher-forced rows, 2557 scored positions per arm, complete code/doc/chat corpora. All logits bitwise identical, KL 0, top1 agreement 100%, identical NLL/PPL; fresh/follow first rows bitwise and six complete recall answers correct on repeated fixtures. This is a local regression gate, not a general model-quality score.
• Three interleaved shipping-binary timing pairs, A1/B1, B2/A2, A3/B3: branch TTFT medians 17.047 → 4.470 seconds (−73.78%). Fresh-prompt medians 34.178 → 34.497 seconds (+0.93%). Reference spread 1.91% follow / 0.90% fresh; 12/12 complete answers correct. These numbers describe this 33.8k fixture, not a universal speedup or a prefill/decode record.
• Combined automatic operation with the independent elastic-cache port: real grow/shrink and prefill relayout; correctly closed conversations at 131072 and 250000 each generate 1024 decode tokens and recognize distinct later user markers. That elastic port is not part of this PR.

This is a conservative new alternative, not the original Daily checkpoint-grid patch: that patch changed chunk boundaries and failed its local paired numerical gate. Its failure remains documented. This proposal does not claim to repair or validate that implementation. The Daily and launcher have not been changed.

Pinned report, complete results/logs, reproducibility scripts, fixture IDs and SHA manifest. Raw multi-GB logits remain local with hashes/sizes recorded in result metadata. The initial incomplete dialogue attempt is preserved separately from the corrected deep-conversation result. AMD/multi-GPU and stock 11 GB hardware are unvalidated.

Related work checked before submission: #62/#65 retain a root/shared-prefix checkpoint, and #203 proposed asynchronous checkpoint copies (closed after no measured gain on current main). This option instead adds one existing near-tail chunk boundary for branches. The broader elastic-cache work in #37, #378 and #563 is outside this focused PR.

[imanu86]

@Niko1221 I kept this submission focused on one optional near-tail checkpoint at an existing prefill boundary. The paired code/doc/chat gate is bitwise identical on the recorded fixed-placement setup; the three timing pairs show 17.047 → 4.470 s branch TTFT on the 33.8k fixture, with a +0.93% fresh-prompt cost. The PR links the complete evidence and its limits.

Before submitting, I also checked the related work: root retention (#62/#65), async checkpoint copying (#203), and VMM/cache work (#37, #378, #563). I found no equivalent near-tail checkpoint in the public branches inspected.

We also have a separately validated automatic elastic expert-cache port (stable VMM addresses, fixed core/rotating tail, growth/shrink at safe points), but I have not opened another cache PR because that overlaps your planned work in #533 and the existing VMM proposals. Your stated preference there is manual VRAM control. If the automatic variant's measurements or a focused part of its implementation would be useful, the same linked report includes the patch, provenance, CUDA functional checks and isolated numeric comparison. We can shape a follow-up around the direction you prefer.

[Niko1221]

Thank you for the careful evidence. An opt-in near-tail checkpoint fits the conversation cache, but it is not part of 0.1.39. It is planned for review in a later release.

[imanu86]

A correction to my note above: the report it links lives in a private repository, so the link did not open for
anyone else. The elastic-cache evidence (and the first 0.1.38 port it describes) is now public at
https://github.com/imanu86/Strata-2080Ti/tree/evidence/elastic-cache, and the cache itself is proposed separately in #1030.

[Niko1221]

In 0.1.40: --prompt-cache-tail is in (opt-in). The tail checkpoint now uses the real chunk size of the call, and it is evicted before periodic checkpoints (only one stays alive).