Skip to content

feat(server): add prefix cache to paged concurrency (qwen AR) - #655

Merged
davide221 merged 1 commit into
Luce-Org:mainfrom
Graffioh:codex/paged-prefix-cache-ar
Sep 12, 2026
Merged

feat(server): add prefix cache to paged concurrency (qwen AR)#655
davide221 merged 1 commit into
Luce-Org:mainfrom
Graffioh:codex/paged-prefix-cache-ar

Conversation

@Graffioh

@Graffioh Graffioh commented Aug 24, 2026 •
edited by cubic-dev-ai Bot
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Enable copied in-memory prefix checkpoints for Qwen continuous batching with paged attention. Full-prefill and disk snapshots remain disabled in paged mode.
  • Keep token matching, boundary selection, pinning, LRU, and hit accounting in PrefixCache while Qwen owns checkpoint payloads, page-table restore, and recurrent state.
  • Introduce an opaque checkpoint/ticket protocol and a move-only capture transaction so the scheduler never manipulates backend snapshot storage.
  • Gather and scatter logical K/V rows through fresh sequence-owned pages, copy the matching recurrent slab, coalesce consecutive physical blocks, and publish replacements atomically.

Correctness review

The pre-draft review fixed these lifecycle hazards:

  • stale restore cleanup now happens before busy or failed admission returns;
  • unsupported engines use cold admission without deleting a valid checkpoint;
  • malformed or mismatched events can abort only their own capture ticket;
  • fatal step failures discard the pending destination instead of preserving potentially mutated state;
  • failed replacement leaves the incumbent checkpoint intact;
  • cache hits and MRU promotion are recorded only after a validated restore;
  • exact-prompt checkpoints cannot mask a shorter restorable prefix when next-token logits are unavailable;
  • tensor type, contiguity, extents, vector topology, recurrent slab shape, and per-layer tensor pairs are validated before destination writes.

Verification

  • HIP Release build of dflash_server
  • test_server_unit: 418 passed
  • test_seq_slot_manager: 186 checks
  • test_parallel_prefix_txn: 21 checks
  • test_seq_engine_contract: 19 checks
  • test_recurrent_snapshot: 2 passed
  • git diff --check

Draft gates

  • Measure capture and restore pause time plus snapshot memory at realistic long-prefix sizes while decode traffic is live.
  • Run real Qwen GPU parity for a partial final page and a forced mid-message boundary, restoring into non-identity fresh pages.
  • Decide the copied-checkpoint byte budget before broad production enablement.

This PR targets main as the AR ownership/mechanism base. Open speculative concurrency PRs #642 and #654 overlap in SeqEngine, Qwen, scheduler, and target-graph files; they should consume this opaque prefix-store boundary during their rebase rather than duplicate snapshot ownership.

Review in cubic

@Graffioh
Graffioh force-pushed the codex/paged-prefix-cache-ar branch 3 times, most recently from aeaeb5b to 8aaa6ae Compare August 25, 2026 08:07
@Graffioh

Graffioh commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 25, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 25 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/src/common/concurrency/seq_engine.h
Comment thread server/src/common/concurrency/prefix_store.h
Comment thread server/src/server/parallel_prefix_txn.h
Comment thread server/src/qwen35/concurrency/qwen35_seq_engine.cpp Outdated
Comment thread server/src/server/server_main.cpp Outdated
Comment thread server/src/common/concurrency/seq_engine.h Outdated
Comment thread server/test/test_parallel_prefix_txn.cpp
@Graffioh
Graffioh marked this pull request as ready for review August 25, 2026 15:03
@Graffioh Graffioh changed the title feat(server): add prefix cache to paged concurrency feat(server): add prefix cache to paged concurrency (qwen AR) Aug 25, 2026
cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 25, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 26 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/src/server/prefix_cache.cpp
Comment thread server/src/common/concurrency/prefix_store.h Outdated
Comment thread server/src/qwen35/qwen35_target_graph.cpp
Comment thread server/test/test_server_unit.cpp Outdated
Comment thread server/src/server/server_main.cpp
cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 26, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/src/server/scheduler.cpp
@Graffioh
Graffioh force-pushed the codex/paged-prefix-cache-ar branch from ca218d5 to 99129bb Compare August 26, 2026 13:03
@Graffioh Graffioh mentioned this pull request Aug 31, 2026
Closed
davide221 pushed a commit to Graffioh/lucebox-hub that referenced this pull request Sep 12, 2026
mrciffa
Merge main into paged prefix cache (Luce-Org#655 rebase)
64937e0 
Resolve the prefix-cache, scheduler, engine, and slot-manager overlap
with Luce-Org#665, Luce-Org#720, Luce-Org#598, and Luce-Org#625/Luce-Org#642:

- reserve_inline_snap() takes restore_source_slot (from Luce-Org#665) ahead of
  the resident-byte estimator; the restore source is never the eviction
  or budget victim and the free-slot path skips it. The Luce-Org#665 tests are
  ported to the reservation API.
- The scheduler builds the prefix plan and admits with it inside the
  admit-first flow from Luce-Org#720; every non-admitted outcome and a failed
  SSE header send cancel the capture transaction.
- pending_capture and seed_restored_prefix() move to the shared
  SeqSlotManager; the Qwen aliases stay.
- The engine keeps its out-of-line constructor and initialises pool_.
- The new flag is documented in server/README.md.
@Graffioh mrciffa
feat(server): add prefix cache to paged concurrency (qwen AR)
3a981ce 
Rebased onto main after Luce-Org#665, Luce-Org#598, Luce-Org#625/Luce-Org#642, and Luce-Org#720. Original
commits by Graffioh (8aaa6ae, 0493354, 171015f, 99129bb); rebase
resolution by mrciffa:

- reserve_inline_snap() takes restore_source_slot ahead of the resident
  byte estimator; the restore source is never the eviction or budget
  victim and the free-slot path skips it. The Luce-Org#665 tests are ported to
  the reservation API.
- The scheduler builds the prefix plan and admits with it inside the
  admit-first flow; every non-admitted outcome and a failed SSE header
  send cancel the capture transaction.
- pending_capture and seed_restored_prefix() live in the shared
  SeqSlotManager.
- The engine keeps its out-of-line constructor and initialises pool_.
- The new flag is documented in server/README.md.
@davide221
davide221 force-pushed the codex/paged-prefix-cache-ar branch from 4daa228 to 3a981ce Compare September 12, 2026 18:52
@davide221
davide221 merged commit 18329b8 into Luce-Org:main Sep 12, 2026
8 checks passed
davide221 pushed a commit to Graffioh/lucebox-hub that referenced this pull request Sep 12, 2026
@Graffioh mrciffa
fix(server): unify generation failure responses
d427a90 
Rebased onto main after Luce-Org#720 and Luce-Org#655. Original commit by Graffioh
(9938ab1); rebase resolution by mrciffa: the scheduler's admit-first
block keeps main's structure and its capacity-exceeded and failed
outcomes use the error envelope, and the deferred-admission shutdown
reply sends the envelope without an SSE preamble (no bytes are on the
wire before admission).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Graffioh @davide221