Repository navigation
Optimize E4B shared-KV prefill on Vulkan - #1861
Closed
chris-beckman wants to merge 1 commit into
Closed
chris-beckman wants to merge 1 commit into
chris-beckman wants to merge 1 commit into
Conversation
Signed-off-by: Chris Beckman <chris@paintedwolf.ai>
This was referenced Nov 5, 2026
Contributor
Author
|
Delivered on main through #1987 ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
E4B prompt ingestion computes query/output rows in upper layers that share earlier KV state and do not populate new cache entries. Prune those rows before the shared-KV tail, while retaining full rows for requested hidden-state or embedding outputs.
Adapt only row pruning from upstream ggml-org/llama.cpp#31356 at
df6b47945e46ff7558c0a8fe88f2698861ee2aca, keeping the pinned PLE scaling/layout unchanged. Pin the patch digest for both development and Nix builds and include it in executable provenance fixtures. Dependency versions are unchanged.Normal Nix baseline/candidate builds and 20 provenance/harness tests pass. On AMD RX 7900 XT, the identical 35,269-token prototype request reaches first token in 10.035 s versus 15.537 s (35.4% reduction); uncached repeat is 9.750 s versus 15.525 s. Decode stays about 100 tokens/s, and process peak RSS stays about 3.05 GiB. Rendered input and greedy output match across builds. Candidate cache/replay/disk-restore parity passes with both 35,251- and 30,010-token checkpoints, including sliding-window rollover; a 187-command lifecycle/grammar/cancellation probe also passes. These are manual timings, not automated gates or model-quality evidence. AMD application/Bialy overlap also passes: two non-abstaining decisions complete while generation remains active (736/455 ms reported decision work). This uses the candidate native package and corrected packaged Bialy with the retained composed host; it is not final installed-desktop qualification.
Depends on #1734 → #1811 → #1812 → #1735 → #1859. Final application integration also includes #1737 → #1736 → #1846 → #1761 → #1762 and #1858 → #1860 after #1736. Lands after other open prerequisite work. Keep draft under the human hold.
RTX 3070 8GB / NVIDIA 610.57.04 paired qualification also passes after the independent ICD dispatch fix (#1862): 35,269 tokens, 30,010-token checkpoint, 49,152 context, chunk 128. Cold first token falls from 20.437 to 12.288 seconds (39.9%); partial-prefix repeat from 4.126 to 2.478 seconds; restored prefix from 4.654 to 2.994 seconds. All 64 greedy output tokens match across both builds and five trials each. Decode is approximately 73–74 tokens/s and peak RSS approximately 3.08 GiB in both. The 187-command lifecycle/grammar/cancellation probe passes. These are prototype direct-runner measurements; the separate AMD overlap probe qualifies resident Bialy coexistence.
Combined regression verification passes 1,196 Go test/subtest results across the affected runtime packages. The composed native suite passes 15 Python and 70 Rust tests; final combined frontend typecheck and budgets also pass. Receipts retain exact source trees. These scoped checks do not replace protected CI on the eventual landing composition.