Repository navigation
feat(ds4): ondisk prefix cache export/adopt; fix continued checkpoint keying - #716
Merged
Merged
Conversation
DeepSeek V4 snapshots were memory-only: the backend left the ModelBackend snapshot_ref/snapshot_adopt hooks at their defaults, so DiskPrefixCache detected a memory-only backend and disabled itself for this model. - Name every snapshot tensor (ds4_snap_*) so the disk cache can serialize the context and rebind it after a reload. - Store the host-side decode state (last logits, DSpark feature tail) and the per-layer compressed row counts as sidecar tensors in the same context. Variable lengths live in ne[1], which the layout fingerprint normalizes, so snapshots of different lengths share one layout. - Add deepseek4_snapshot_bind() to rebind a deserialized context by name, with optional per-shard prefix and shared-storage ownership. - Implement snapshot_ref/snapshot_adopt on DeepSeek4Backend. Adopt validates against the compression schedule of the loaded weights, so a valid file is never dropped as corrupt while the target is parked. Paged concurrent serving stays memory-only. - Implement the same hooks on DeepSeek4LayerSplitAdapter by merging the shard snapshots into one named context; mixed remote-shard splits stay memory-only. - Unit tests: real DiskPrefixCache save/lookup/adopt/restore round trip with exact-hit decode and layout stability across lengths (monolithic), and a merged-context round trip for the layer-split adapter.
A disk entry is keyed by the hash of its prompt prefix and a hit restores the whole snapshot. maybe_store_continued() keyed files by the interval-aligned prefix while the snapshot extended to the generation's final position, so a later prompt sharing only the key prefix could restore a snapshot covering tokens it never contained. - Key continued checkpoints by the full token prefix the snapshot covers; the interval only paces them. - Refuse to write a file whose snapshot position exceeds its key, and reject such files on read so existing ones are dropped. - Tests cover both guards and the new keying; the identity-salt tests now use a key that covers the mock snapshot.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 9 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…ss review
Move the DeepSeek V4 snapshot code out of deepseek4_graph.cpp into
deepseek4_snapshot.{h,cpp} with a small API: declare (tensors only, into a
caller-owned context), fill, save (declare + allocate + fill), restore, bind
and validate. A DeepSeek4LayerGeometry derived from the weights is the single
source of truth for cache allocation, snapshot declaration and validation.
- Adopt validates every tensor's type and shape against the weights'
geometry (and capacity when the live cache exists), so a malformed or
foreign file is rejected on lookup and removed instead of failing every
later restore.
- Layer-split adapter: one merged host context per slot holds the shard
snapshots (aliased, owns_storage=false) plus the adapter-level tensors,
and is what the disk cache serializes. No second copy per slot, no
rebuild step, no reader backend to hand over.
- DiskPrefixCache::read_file() allocates from the CPU buffer type instead of
a CPU backend object, removing the per-lookup backend leak for all
backends.
- Tests: continued checkpoints now cover crossing into a new interval
bucket; DeepSeek tests use std::filesystem instead of a shell, exercise
geometry rejection on both paths, and assert the split snapshot aliases
the merged context.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 13 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…hard slot on split save failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeepSeek V4 snapshots were memory-only: the backend left
snapshot_ref/snapshot_adoptat their defaults, soDiskPrefixCachedetected a memory-only backend and disabled itself for this model (the gating added in #579). This PR implements the two hooks for DeepSeek V4 on the same shared disk cache used by Qwen3 / Qwen3.5 / Laguna, and fixes a keying inconsistency in the shared continued-checkpoint path.DeepSeek V4 ondisk prefix cache (
server/src/deepseek4)ds4_snap_*), so the disk cache can serialize the context and rebind it after a reload.ne[1], which the layout fingerprint normalizes, so snapshots of different lengths share one layout id.deepseek4_snapshot_bind()rebinds a deserialized context by name, with an optional per-shard prefix and shared-storage ownership.DeepSeek4Backend::snapshot_ref/snapshot_adopt: adopt validates against the compression schedule of the loaded weights (not the live cache), so a valid file is never dropped as corrupt while the target is parked. Paged concurrent serving stays memory-only, as in the Qwen3.5 backend.DeepSeek4LayerSplitAdapter: shard snapshots are merged into one named context (ls<i>_prefix plus adapter-level meta / HC state / prefill logits), mirroring the Qwen3.5 adapter. Mixed remote-shard splits stay memory-only.Continued checkpoint keying (
server/src/server/disk_prefix_cache.cpp)A disk entry is keyed by the hash of its prompt prefix and a hit restores the whole snapshot.
maybe_store_continued()keyed files by the interval-aligned prefix while the snapshot extended to the generation's final position, so a later prompt sharing only the key prefix could restore a snapshot covering tokens it never contained. Now:save()refuses a snapshot whose position exceeds its key, andread_file()rejects such files so existing ones are dropped on lookup.Validation
Unit (lucebox5, HIP gfx1151;gfx1201 build on top of #598):
test_deepseek4_unit: all cases pass, including newtest_monolithic_snapshot_disk_roundtrip(realDiskPrefixCachesave → lookup → adopt → restore → exact-hit decode, layout stability across lengths, rejection of a bogus context) andtest_layer_split_snapshot_disk_roundtrip.test_server_unit: 453 passed, 0 failed, including newtest_disk_cache_rejects_snapshot_past_keyandtest_disk_cache_continued_keys_full_prefix.End to end (lucebox5, DeepSeek-V4-Flash-0731 ROCmFPX on R9700 + Strix, long-context profile, greedy, two-turn conversations of ~950 tokens,
--kv-cache-dirset, prefix cache slots enabled):--disk-prefix-cache auto), is adopted, and prefills only the new turn: 17 to 28 tokens in 0.4 to 1.4 s (including the file read) versus 7 to 9 s for a cold prefill of ~950 tokens.Notes for reviewers
--disk-prefix-cache fullkeys the exact full prompt, so a conversation with a new last turn misses; cross-session reuse needsauto(per request viaprefix_cache.scope, or as the server default).