Repository navigation
meta : raise the compute view limit for buffers with few tensors - #31447
Merged
Merged
Conversation
Assisted-by: Claude Opus 5.5
github-actions
Bot
added
testing
developer0hye
marked this pull request as ready for review
October 31, 2026 00:30
developer0hye
requested review from
JohannesGaessler and
ggerganov
as code owners
October 31, 2026 00:30
am17an
approved these changes
Nov 1, 2026
am17an
left a comment
am17an
left a comment
Contributor
There was a problem hiding this comment.
@ggerganov this may be a better solution
JohannesGaessler
approved these changes
Nov 5, 2026
ggerganov
reviewed
Nov 5, 2026
| n_head = 8; | ||
| n_ff = 1024; | ||
| n_layer = 6; | ||
| n_layer = 22; |
Member
There was a problem hiding this comment.
I think this change will make the dummy model file become a few GB. We generally try to keep the dummy models small.
Contributor
Author
There was a problem hiding this comment.
Agreed, it was 437.7 MB in CI (128.5 MB with 6 layers), and it also broke the Metal fusion counts and the WebGPU NMSE check. I reverted the test change in 4079b31, so the PR only changes ggml-backend-meta.cpp now. The smallest model that reproduces the crash has 19 layers (379.7 MB) and is only 1 view over the limit, so I left the test out; I checked locally that 22 layers fail before the fix and pass after it. I can add a regression test if you want one.
Assisted-by: Claude Opus 5.5
am17an
approved these changes
Nov 7, 2026
3 of 6 tasks
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…b and cache engine) Re-based onto a691b23: 7 conflicted files / 12 hunks, all of them upstream's new subsystem-meets-ours rather than a mechanism fight. - common/common.cpp, common/fit.cpp: upstream factored the --fit block into common_fit_apply(); the delivery's draft reserve (wip/slab-transient-absorption) moves INTO it as a `common_draft_reserve_derive()` helper shared with the ctor's MoE preflight, and fit.cpp keeps both sides (upstream's RAII common_fit_log_scope + the delivery's host-expert accounting and its `hp_ngl_skip` / host-expert out-params, which also fixes the call sites that predated upstream's n_skip parameter). - ggml-backend.cpp: upstream's prefetch-split early-out and the delivery's graph-input sync condition coexist; both event families are freed at teardown. - ggml-cuda/common.cuh: the delivery's keyed CUDA-graph cache with upstream's idle eviction window (ggml-org#30662), keyed comparison instead of the node pointer. - ggml-backend-meta.cpp: the delivery's GGML_META_COMPUTE_HEADROOM knob plus upstream's 256-view floor (ggml-org#31447). - llama-context.cpp / llama-cparams.h: upstream's kv_rows auto-resolution and moe_cache_cpu_share, the delivery's drop_compute_buffers, fused_hc_mix/combine and host-table accounting; `model.hparams.no_alloc` -> `model.no_alloc()` (ggml-org#31141). Finding (blocks the host-expert tier on a default build): the movable-boundary slab and therefore the whole expert-cache arena require VMM, and upstream's HIP default is GGML_HIP_NO_VMM=ON. On build-rocm the arena declines ("the device does not report VMM") and the run streams the experts, so any host-expert measurement needs -DGGML_HIP_NO_VMM=OFF. A second build dir (build-rocm-vmm) was made for the D1 numbers. Validated: build green; dense coherence green; and on Qwen3.6-35B-A3B Q8_0 (-ncmoe 20, -c 8192, 16.3 GiB host experts, 1 GPU, VMM build) at equal MiB the arena (4080 MiB) gives prefill 547.6 / decode 53.5 t/s against upstream --moe-cache-mib 4096's 544.2 / 62.9 and streaming's 470 / 49, with byte-identical text (533 chars, sha 8d6e74e98d89) in every arm. That is D1's first measurement, not its decision.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…b and cache engine) Re-based onto a691b23: 7 conflicted files / 12 hunks, all of them upstream's new subsystem-meets-ours rather than a mechanism fight. - common/common.cpp, common/fit.cpp: upstream factored the --fit block into common_fit_apply(); the delivery's draft reserve (wip/slab-transient-absorption) moves INTO it as a `common_draft_reserve_derive()` helper shared with the ctor's MoE preflight, and fit.cpp keeps both sides (upstream's RAII common_fit_log_scope + the delivery's host-expert accounting and its `hp_ngl_skip` / host-expert out-params, which also fixes the call sites that predated upstream's n_skip parameter). - ggml-backend.cpp: upstream's prefetch-split early-out and the delivery's graph-input sync condition coexist; both event families are freed at teardown. - ggml-cuda/common.cuh: the delivery's keyed CUDA-graph cache with upstream's idle eviction window (ggml-org#30662), keyed comparison instead of the node pointer. - ggml-backend-meta.cpp: the delivery's GGML_META_COMPUTE_HEADROOM knob plus upstream's 256-view floor (ggml-org#31447). - llama-context.cpp / llama-cparams.h: upstream's kv_rows auto-resolution and moe_cache_cpu_share, the delivery's drop_compute_buffers, fused_hc_mix/combine and host-table accounting; `model.hparams.no_alloc` -> `model.no_alloc()` (ggml-org#31141). Finding (blocks the host-expert tier on a default build): the movable-boundary slab and therefore the whole expert-cache arena require VMM, and upstream's HIP default is GGML_HIP_NO_VMM=ON. On build-rocm the arena declines ("the device does not report VMM") and the run streams the experts, so any host-expert measurement needs -DGGML_HIP_NO_VMM=OFF. A second build dir (build-rocm-vmm) was made for the D1 numbers. Validated: build green; dense coherence green; and on Qwen3.6-35B-A3B Q8_0 (-ncmoe 20, -c 8192, 16.3 GiB host experts, 1 GPU, VMM build) at equal MiB the arena (4080 MiB) gives prefill 547.6 / decode 53.5 t/s against upstream --moe-cache-mib 4096's 544.2 / 62.9 and streaming's 470 / 49, with byte-identical text (533 chars, sha 8d6e74e98d89) in every arm. That is D1's first measurement, not its decision.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…b and cache engine) Re-based onto a691b23: 7 conflicted files / 12 hunks, all of them upstream's new subsystem-meets-ours rather than a mechanism fight. - common/common.cpp, common/fit.cpp: upstream factored the --fit block into common_fit_apply(); the delivery's draft reserve (wip/slab-transient-absorption) moves INTO it as a `common_draft_reserve_derive()` helper shared with the ctor's MoE preflight, and fit.cpp keeps both sides (upstream's RAII common_fit_log_scope + the delivery's host-expert accounting and its `hp_ngl_skip` / host-expert out-params, which also fixes the call sites that predated upstream's n_skip parameter). - ggml-backend.cpp: upstream's prefetch-split early-out and the delivery's graph-input sync condition coexist; both event families are freed at teardown. - ggml-cuda/common.cuh: the delivery's keyed CUDA-graph cache with upstream's idle eviction window (ggml-org#30662), keyed comparison instead of the node pointer. - ggml-backend-meta.cpp: the delivery's GGML_META_COMPUTE_HEADROOM knob plus upstream's 256-view floor (ggml-org#31447). - llama-context.cpp / llama-cparams.h: upstream's kv_rows auto-resolution and moe_cache_cpu_share, the delivery's drop_compute_buffers, fused_hc_mix/combine and host-table accounting; `model.hparams.no_alloc` -> `model.no_alloc()` (ggml-org#31141). Finding (blocks the host-expert tier on a default build): the movable-boundary slab and therefore the whole expert-cache arena require VMM, and upstream's HIP default is GGML_HIP_NO_VMM=ON. On build-rocm the arena declines ("the device does not report VMM") and the run streams the experts, so any host-expert measurement needs -DGGML_HIP_NO_VMM=OFF. A second build dir (build-rocm-vmm) was made for the D1 numbers. Validated: build green; dense coherence green; and on Qwen3.6-35B-A3B Q8_0 (-ncmoe 20, -c 8192, 16.3 GiB host experts, 1 GPU, VMM build) at equal MiB the arena (4080 MiB) gives prefill 547.6 / decode 53.5 t/s against upstream --moe-cache-mib 4096's 544.2 / 62.9 and streaming's 470 / 49, with byte-identical text (533 chars, sha 8d6e74e98d89) in every arm. That is D1's first measurement, not its decision.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 10, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 10, 2026
…b and cache engine) Re-based onto a691b23: 7 conflicted files / 12 hunks, all of them upstream's new subsystem-meets-ours rather than a mechanism fight. - common/common.cpp, common/fit.cpp: upstream factored the --fit block into common_fit_apply(); the delivery's draft reserve (wip/slab-transient-absorption) moves INTO it as a `common_draft_reserve_derive()` helper shared with the ctor's MoE preflight, and fit.cpp keeps both sides (upstream's RAII common_fit_log_scope + the delivery's host-expert accounting and its `hp_ngl_skip` / host-expert out-params, which also fixes the call sites that predated upstream's n_skip parameter). - ggml-backend.cpp: upstream's prefetch-split early-out and the delivery's graph-input sync condition coexist; both event families are freed at teardown. - ggml-cuda/common.cuh: the delivery's keyed CUDA-graph cache with upstream's idle eviction window (ggml-org#30662), keyed comparison instead of the node pointer. - ggml-backend-meta.cpp: the delivery's GGML_META_COMPUTE_HEADROOM knob plus upstream's 256-view floor (ggml-org#31447). - llama-context.cpp / llama-cparams.h: upstream's kv_rows auto-resolution and moe_cache_cpu_share, the delivery's drop_compute_buffers, fused_hc_mix/combine and host-table accounting; `model.hparams.no_alloc` -> `model.no_alloc()` (ggml-org#31141). Finding (blocks the host-expert tier on a default build): the movable-boundary slab and therefore the whole expert-cache arena require VMM, and upstream's HIP default is GGML_HIP_NO_VMM=ON. On build-rocm the arena declines ("the device does not report VMM") and the run streams the experts, so any host-expert measurement needs -DGGML_HIP_NO_VMM=OFF. A second build dir (build-rocm-vmm) was made for the D1 numbers. Validated: build green; dense coherence green; and on Qwen3.6-35B-A3B Q8_0 (-ncmoe 20, -c 8192, 16.3 GiB host experts, 1 GPU, VMM build) at equal MiB the arena (4080 MiB) gives prefill 547.6 / decode 53.5 t/s against upstream --moe-cache-mib 4096's 544.2 / 62.9 and streaming's 470 / 49, with byte-identical text (533 chars, sha 8d6e74e98d89) in every arm. That is D1's first measurement, not its decision.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 10, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 10, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
With
-sm tensor, DeepSeek-V4.1 aborts on the first decode:The compute containers that hold external views of a meta buffer have room for 16 views per statically allocated tensor. DeepSeek-V4.1 shares the KV cache of a source layer with the following layers. In the 1017 checkpoint (
kv_source_layers= 2, 8, 14, 20), the cache of layer 20 is read by layers 20-39 and by the indexer candidate path, but its meta buffer holds only two static tensors. Its container fits 32 views, but 46 views of it are created before the container is reset.Give every compute container room for at least 256 views. With 8 devices this is at most 2 x 8 x 256 x 368 bytes = 1.5 MB of tensor metadata per meta buffer.
The deepseek41 model in
test-llama-archshas 6 layers, too few to reach this limit. Its Meta (-sm tensor) config fails with the assert above from 19 layers on (15 of them read the ratio-1 KV source), and passes after this change. But a 19-layer model file is 379.7 MB instead of 128.5 MB, so this PR does not change the test model.Additional information
DeepSeek-V4.1-Flash 1017 (Q8_0, MXFP4 experts) on 8 RTX A6000s (NVLink pairs), CUDA 13.0, driver 580.159.03, NCCL 2.32.3 with
NCCL_P2P_LEVEL=NVL,-fa on, all layers on GPU. Median time for a batch of real text on an empty context:-sm layer-sm tensorllama-perplexityover 8 x 512 tokens: 3.3229 +/- 0.158 with-sm layer, 3.3328 +/- 0.160 with-sm tensor. Before this change-sm tensordoes not run for this model.On master 6c01da6, with an RTX 3070 (CUDA) and an Apple M2 (Metal):
test-llama-archs -s 1with 1, 2 and 4 CUDA devices and with 1 and 2 Metal devices: all tests pass.Requirements