perf(ds4): gfx1151 fast path, exact q5 rollback, confidence-driven verify width by davide221 · Pull Request #729 · Luce-Org/lucebox

davide221 · 2026-09-13T20:30:21Z

perf(ds4): gfx1151 fast path, exact q5 rollback, confidence-driven verify width

Makes DeepSeek V4 Flash speculative decoding on Strix Halo (gfx1151) the default performance a bare launch gets, on top of main 2e24149 (PR 728; the fast-path base was PR 684, and no commit since touches these files). The kernels come from the September 6 lucebox8 snapshot that produced 284 / 37.4 at 123K and never reached a PR; they were rebased, reduced to device-class rules and profile defaults, reviewed for merge, and two behavior defects found in review were fixed and validated separately. 8 commits (5 logical plus three follow-ups from CI and review), 30 files.

Commits

38e80b9 perf(hip): DS4 gfx1151 fast-path kernels and ggml core hooks
eee1f6c feat(common): non-absorbing adaptive verify width driven by a calibrated confidence head
f55e758 perf(ds4): gfx1151 fast path as device-profile defaults with exact q5 rollback
ee275e3 test(ds4): fast-path kernel, rollback, lifecycle and width-policy coverage
dfd5eae docs(ds4): gfx1151 fast-path defaults and surviving switches
f105340 test(ds4): skip the D512 position replay test off HIP
e53d778 fix(ds4): review fixes, verified-only calibration, bounded width exploration
ad64216 fix(ds4): second review round

What a bare launch gets on gfx1151

A plain launch (DFLASH_DS4_SPEC=1, DFLASH_DS4_DRAFT=<dspark draft>, --chunk 8192) installs one device profile, two constexpr tables applied by one helper, a user-set value always winning:

fused q5 DSpark verify with a 24-slot resident verify-graph cache (every width across the four ratio-4 phases; device scratch is shared at about 0.8 MiB per slot, and each built slot keeps a 32 MiB host metadata arena, up to 768 MiB when fully populated);
verify width q2 to q5 chosen per step from the drafter's confidence head (three depths; the fourth learned from target feedback), calibrated online per depth against the target's actual acceptance, scoring only candidates the target actually verified; after a clean draft the controller may step one width up when that width is within 3 percent of the best, so a near tie cannot freeze it below the cap, while prose's larger q2-to-q3 gap keeps prose at q2; DFLASH_DS4_CONFIDENCE_WIDTH=0 falls back to learned acceptance;
exact q5 rollback: single-boundary rejections truncate directly in both cases, the seed row recovered from the rotated previous half when the flush is undone; checkpoint restore across two boundaries, replay only without a checkpoint;
sparse-prefill kernel profile: rocWMMA streaming and dense high-ratio MLA attention, sparse value skip, segmented KV, m32 indexer with cached B operand, F16 indexer query and KV transport, analytic causal window for SWA layers, maskless ratio-4 prefill admitted under the kernel's contract;
F16 MMVF admission up to five columns on RDNA 3.5 (the env ceiling is a floor), ROCmFP4 x4+1 MMVQ, ROCmFP3 wide two-pass three-row tiles;
chain-graph cache keyed by a drafter lifecycle generation, so a reloaded drafter never replays a graph over freed tensors.
Unchanged from main: PR 684 GLU fusion admission, F32 prefill rows, sparse decode flash stays opt-in, chunk size stays a deployment setting (release ships 3072; 8192 is what the tables measure).

Measurements

lucebox8, Strix Halo gfx1151, ROCm 7.2.4, ROCmFPX MIX target + Q4RMFP4 DSpark draft, 128K alloc, bare launch, five-key retrieval fixture, expected response hash on every run. Vulkan: Nathan v0.7.5 portable on the same box with the published DSpark drafter, --spec-draft-n-max 4, q8_0 KV, max-perf flags, memory-guarded runner.

Context	main (PR 684)	this branch	Vulkan v0.7.5 same box
8K prefill / decode	256.6 / 34.2 (spec, 9K)	320.4 / 41.9	240.7 / 27.1
32K	n/a	314.6 / 36.9 (pre-review tree; no clean-step path changed since)	239.9 / 27.0
123K	150.8 / 24.3 (chunk 4096)	283.5 / 36.3	193.8 / 24.4

8K prefill reads about 10 percent lower when measured right after a page-cache drop; the values above are warm. Vulkan's published v0.7.0 claims (254 / 37.7 at 8K, 226.8 / 35.7 at 123K) did not reproduce on this box.

Real workloads, same box, four prompts per suite, 512-token cap, temperature 0, server decode tok/s:

Suite	main (shipped q4)	this branch	accuracy
HumanEval	33.6	41.5	4/4 both
GSM8K	30.6	36.8	4/4 both
MATH	30.1	40.6	4/4 both
agent	29.6	38.9	n/a
prose, 500-word essays	18.1	24.2	Vulkan same prompts: 17.1
prose then code, one request	n/a	29.0 to 29.3 (learned-acceptance policy: 27.2)	n/a

On code and math the controller holds q5 on 691 of 795 steps (3.0 of 4 candidates accepted on average, 39.4 tok/s pooled against 30.5 on main); on prose it sits at q2 on about 300 of 310 steps per essay; on the mixed request it runs the essay at q2 and switches to q5 when the code starts (q5 on 110 to 114 of about 320 steps in two runs), which the learned-acceptance policy never did. Verify graphs are reused: 15 to 36 builds per 700 to 1,250 steps.

Correctness findings from the review, both fixed

q5 rollback. A five-token verify wraps the ratio-4 rolling state, so the fifth token's slot is the seed's. The direct truncation first used for speed overwrote the seed row with stale data whenever the flush had to be undone. Fixed-q5 prose output is byte-reproducible run to run here, and the old rule diverged from a replay-always reference within the first 5 to 50 tokens of every essay. The fused graph rotates the completed window, seed row included, into the previous half before the tail rewrites the current half, so the fix copies that one row back per rolling tensor before restoring the previous half; rejected rows that alias a committed row are never restored. The fixed rule matches the old output exactly up to the first undone-flush rejection of every essay and tracks the replay reference as far or farther than a replay-on-undo fix, at truncation cost: replaying instead had cost the real suites 32.4 tok/s because the controller charged the replays to q5 and retreated to q2/q3.
Adaptive width. Prefix-survival estimates beyond the offered width were frozen, so once narrowed to q2 the controller could not re-widen within a request. A rejection now zeroes every deeper prefix and a clean draft extrapolates the next depth geometrically (an optimistic first version probed q3 on a third of prose steps and cost 12 percent there). On top of that the width now comes from the DSpark confidence head, which the q4 path already used and the q5 path had skipped; the head is honest at depth 1 and increasingly optimistic deeper on code (depth 4: 0.78 predicted against 0.64 accepted), so depth 4 stays learned and every depth is calibrated online. DFLASH_DS4_TIMING prints the per-depth calibration after each request.

Tests

gfx1151 (HIP_VISIBLE_DEVICES=1 on lucebox8): CI ctest set 9/9, extended GPU set 18/18, test_server_unit 544/544.
New coverage: indexer score m32 specializations, sparse value skip, segmented KV, gallocr release and rebind, q5 replay alignment guard, seed-row restore contract, chain-graph cache generation, device-aware maskless launch expectation, and eight adaptive-width contract tests (widening and narrowing at both bounds, full-rejection collapse, re-widening within 32 clean steps, deeper zeros, prose-like Bernoulli(0.6) traffic stays narrow, short confidence vector extension, online calibration).
Pre-existing on main and outside the CI regex: test_rope_tail's CPU fixture aborts on an unaligned std::vector pointer (file byte-identical to main).

Switch audit

Every experimental switch the snapshot added was A/B-measured on the final build (8K and 123K retrieval, real suites). Every off-by-default switch measured as noise or worse was removed with its code path (fused compressor, compressor lean variants, MMVQ x5, x4-max-k, fp2-dual and expert-union variants, ROCmFP3 row4, F16 indexer verify query, verify-graph retention, block-size and warps-per-block tuning knobs, the test-only ggml_get_rows_f16 API). Every switch that remains is documented in DS4.md and ENVIRONMENT.md as a kill switch for a defaulted path or a diagnostic. HIP graph replay was also measured and refuted on this GPU (0.2 µs saved per kernel, capture cost never amortized on short replies); the tree keeps GGML_HIP_GRAPHS off as the CMake default does, and the three DS4.md build recipes that still pass it on are a separate doc fix.

For reviewers

Shared-code changes that also affect other backends: ggml core hooks (ggml_flash_attn_ext_set_ds4_kv_segments on src[7]/src[8], causal ratio), segmented KV defaulting on for every HIP backend, fused verify and decode meta arenas 256/192 MiB to 32 MiB, need_logits skipping the output projection on non-terminal prefill chunks. CI on CUDA, gfx1201 and Windows has not run on this tree; a static scan found no HIP-only construct outside HIP guards in the added kernel code.
Known limit: the confidence head's fourth depth is learned rather than trusted; a head calibrated on q5 traffic would remove that.

Kernel side of the Strix Halo (gfx1151) DeepSeek V4 fast path, reviewed for merge: - MLA D512 attention: rocWMMA streaming and dense high-ratio paths, sparse value skip, segmented compressed/preserved-tail KV operands via ggml_flash_attn_ext_set_ds4_kv_segments (src[7]/src[8]; src[6] stays the rope positions), analytic causal ratio. Segmented KV always takes the split-KV schedule; the dense fall-throughs assert instead of indexing past the buffer. One gfx1151 predicate replaces raw compares. - Indexer: rocWMMA m32 score kernels with a cached B operand, F16 query input, named crossovers (256 scored tokens, 6144 rows). - MMVQ: ROCmFP4 x4+1 dense kernel for the five-column q5 verifier. - MMVF: the F16 column ceiling is an admission floor, and RDNA 3.5 admits F16 MMVF up to five columns as a device-class rule; the width-5 block-size table is a measured shape table, not a tunable. - ROCmFP3 MIX: wide two-pass and three-row tiles for q3 to q5 shapes. - CPU indexer score accepts an F16 query. Every switch that survives is a kill switch for a defaulted path or a diagnostic (documented in the docs commit); tuning knobs measured as noise were removed with their code paths.

…ted confidence head AdaptiveSpecWidth changes, all covered by the unit suite: - Prefix survival beyond the offered width is no longer frozen: a rejection is a real zero for every deeper prefix and a clean draft extrapolates the next depth geometrically from the two deepest observed survivals, so a narrow width never becomes absorbing while low acceptance still keeps it narrow. - A per-candidate confidence vector shorter than the proposal is extended with the learned conditional acceptance of the uncovered depths (ratio of consecutive survivals, floored so two near-zero estimates cannot read as certainty). - Confidence scores are calibrated online per depth: observe_confidence() tracks predicted against observed acceptance of offered candidates and each score is scaled by their ratio, bounded and reset per request. - Named constants for the priors, rates, warmup and bounds. DSpark head: the chain-graph cache is keyed by an atomic drafter lifecycle generation bumped on every drafter free path, so a reloaded drafter never replays a graph over freed tensors; a build counter lets the unit test assert hit and miss.

… rollback Server side of the Strix Halo fast path. A bare launch gets the qualified configuration because the gfx1151 profile is two constexpr tables applied by one helper with one banner each, a user-set value always winning: fused q5 verify, adaptive width q2 to q5 from the DSpark confidence head, 128-row graph padding, pinned rollback, GPU argmax, and the sparse-prefill kernel profile (rocWMMA attention, cached m32 operand, F16 indexer query and KV transport, analytic causal window). - Fused verifier cache: 24 resident slots in q5 mode so every width stays warm across the four ratio-4 phases (about 0.8 MiB per slot). - Maskless ratio-4 sparse prefill admitted only under the kernel's contract (named cap, ratio, prior rows, live compressed rows). - q5 rollback is exact and cheap: single-boundary rejections truncate directly in both cases. When the flush is undone the seed's rolling row is recovered from the rotated previous half with one device row copy per rolling tensor (the fifth token's slot aliases the seed's), rejected rows that alias a committed row are never restored, and replay remains only for two boundaries without a checkpoint. - Width per step from the confidence head (three depths; the fourth learned from target feedback), DFLASH_DS4_CONFIDENCE_WIDTH=0 falls back to learned acceptance; DFLASH_DS4_TIMING prints the per-depth calibration. - Drafter free paths note the lifecycle generation (DS4 and Qwen3). - Named contracts replace literals (comp pad stride, ratio-4 boundary helpers, width bounds and cost table, analytic causal admission). lucebox8, bare launch, expected hash on every run: 8K 316.6 / 42.1, 123K 284.5 / 37.9, prose 24.3 tok/s at q2, real suites 39.4 tok/s pooled at q5 (4/4 on every scored suite), prose-then-code 28.4. Same-box Vulkan v0.7.5 with the published drafter: 240.7 / 27.1, 193.8 / 24.4, 17.1.

…erage - test_deepseek4_unit: indexer score m32 specializations, sparse value skip, segmented KV, gallocr release and rebind, q5 replay alignment guard, seed-row restore contract (test_dspark_seed_row_restore_cpu), chain-graph cache generation (hit within a generation, miss after a bump and for a different output set, kill switch honoured), shared DSparkChainFixture and LCG helpers instead of copied bodies. - test_ds4_maskless_prefill expects the gfx1151 profile's explicit-mask streaming launch count on that device. - test_adaptive_spec_width: widening and narrowing at both bounds, full-rejection collapse, re-widening within 32 clean steps, deeper zeros, prose-like Bernoulli(0.6) traffic staying narrow, short confidence vector extension, online calibration. - GPU test helpers updated for the F16 MMVF admission and the grouped MoE benchmark accounting.

DS4.md and ENVIRONMENT.md: rows for every default the gfx1151 profile installs and every switch that survives the audit, each classified as a kill switch for a defaulted path or a diagnostic; rows for removed tuning knobs deleted.

The DeepSeek4 D=512 flash-attention kernel is HIP-only and aborts on CUDA; every other new GPU test already skips there, this one did not and took the DGX verifier job down.

cubic-dev-ai

2 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_backend.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_backend.cpp:363">
P3: The new gfx1151 profile exposes three undocumented environment controls, so operators cannot discover or disable these defaults from the full inventory. Add each variable with its defining file and override semantics.

(Based on your team's feedback about environment inventory consistency.)</violation>
</file>

<file name="server/test/test_ds4_dspark_load.cpp">

<violation number="1" location="server/test/test_ds4_dspark_load.cpp:202">
P3: When a benchmark iteration fails, `need()` prints `FAIL: missing benchmark draft forward`, but the forward ran and failed rather than being absent. Use a phrase that matches the failure, e.g. `"benchmark draft forward failed"` (or a dedicated check) so the diagnostic reads correctly.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

…oration Review findings on PR 729, all validated on gfx1151 (tests 18/18 and 562/562, prose 24.6, real suites 39.3 at q5 with 4/4, 8K hash): - DSpark chain-graph cache recreates its allocator when the backend changes instead of allocating the new graph in the old device's buffers. - An explicit DFLASH_CUDA_MMVQ_MOE_FP3_PACKED24 value wins on gfx1151; only an unset variable takes the device default. - The gfx1151 verifier profile propagates a failed default and init() aborts, like the sparse-prefill profile. - The streaming rocWMMA epilogue writes zero for a state with no visible rows instead of dividing by zero. - ggml_flash_attn_ext_set_ds4_kv_segments asserts that K and V alias; the header states the precondition. - deepseek4_step clears out_logits when logits were not requested on the non-hybrid path. - Confidence calibration scores only candidates the target verified: after a rejection the deeper conditional scores have no outcome. - After a clean draft the controller takes the next wider width when it is within kExploreMargin (3%) of the best. A near tie between two widths is otherwise decided by measured step costs, and a width never offered never gets its survival or calibration measured; on a prose-then-code request that left the controller at q3 on the code in one run and at q5 in another. With the margin both runs reach q5 (29.0 and 29.3 tok/s against 27.2 for the learned policy); prose's q2-to-q3 gap is larger, so prose stays at q2. - Tests: unknown DFLASH_TEST_QUANT and DFLASH_MMID_TEST_TYPE filters fail instead of passing vacuously, the parent rejects the type filter like the width filter, one shared DS4 cost curve, clearer benchmark diagnostic. - Docs: verifier cache slots default 8 and 24 with q5, per-variable semantics of the rocWMMA switches, rows for the three profile-installed variables, the adaptive-width paragraph rewritten; the slot comment now states the 32 MiB host metadata arena per built slot.

cubic-dev-ai

All reported issues were addressed across 15 files (changes from recent commits).

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

- observe() clears the clean-draft flag on a one-row step, so exploration follows only an actual clean draft. - The cost-aware decision keeps its per-width utilities in a member scratch sized from the width cap: every width up to the cap is evaluated and the const path allocates nothing. - DFLASH_CUDA_MMVQ_MOE_FP3_PACKED24: only a missing variable is unset; an empty value is an explicit setting. - DFLASH_DS4_Q5_VERIFY has an inventory row. - The grouped MMID child fails when a type or width filter selects no runnable case; the last literal DS4 cost curve in the width tests uses the shared constant. gfx1151: tests 18/18 and 562/562, prose 25.0, mixed 29.0 at q5, 8K hash.

cubic-dev-ai

All reported issues were addressed across 4 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

…tory row

…ay from the build recipes HIP graph replay was measured on gfx1151 (ROCm 7.2): 0.2 us saved per kernel, 25 to 30 ms to capture a 7,000-node graph per shape, short replies 8 to 10 percent slower, long ones unchanged. The three DS4 build recipes no longer pass -DGGML_HIP_GRAPHS=ON (the CMake default is off, and the appliance and Docker builds already build that way). The docs now state the qualified launch: DFLASH_DS4_SPEC, the DSpark draft, --chunk 8192, 128K context, nothing else.

cubic-dev-ai

All reported issues were addressed across 1 file (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

…agraph

The README's Strix Halo rows and the recommended-setups page still carried the blog recipe (fixed verify width 4, exact prefill) and its figures. The qualified launch is now the plain one, so the page shows that command and the README rows cite what it measures.

mrciffa added 6 commits September 13, 2026 18:54

test(ds4): skip the D512 position replay test off HIP

f105340

The DeepSeek4 D=512 flash-attention kernel is HIP-only and aborts on CUDA; every other new GPU test already skips there, this one did not and took the DGX verifier job down.

cubic-dev-ai Bot reviewed Sep 13, 2026

View reviewed changes

Comment thread server/docs/ENVIRONMENT.md Outdated

mrciffa added 2 commits September 14, 2026 00:38

docs(ds4): list deepseek4_graph.cpp on the DFLASH_DS4_Q5_VERIFY inven…

6755839

…tory row

cubic-dev-ai Bot reviewed Sep 13, 2026

View reviewed changes

Comment thread server/docs/DS4.md

mrciffa added 2 commits September 14, 2026 00:57

docs(ds4): describe the q5 MMVQ default ahead of the plain-launch par…

9e85e84

…agraph

davide221 merged commit 1a7ccdf into main Sep 14, 2026
8 checks passed

davide221 deleted the perf-ds4-gfx1151-fast-path branch September 18, 2026 15:50

Navigation Menu

perf(ds4): gfx1151 fast path, exact q5 rollback, confidence-driven verify width - #729

perf(ds4): gfx1151 fast path, exact q5 rollback, confidence-driven verify width#729
davide221 merged 12 commits into
mainfrom
perf-ds4-gfx1151-fast-path

davide221 commented Sep 13, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

davide221 commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

perf(ds4): gfx1151 fast path, exact q5 rollback, confidence-driven verify width

Commits

What a bare launch gets on gfx1151

Measurements

Correctness findings from the review, both fixed

Tests

Switch audit

For reviewers

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

davide221 commented Sep 13, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading