Repository navigation
llama : fix on-device state copy sizes for quantized KV caches - #30541
Merged
am17an merged 3 commits intoNov 5, 2026
Merged
Conversation
The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed their element count as size / ggml_element_size(). For a quantized type such as q8_0 or q4_0 that is the number of blocks, not elements, so only part of each range was copied: the restored state was wrong, or the copy crashed. Compute the count as size / ggml_type_size() * ggml_blck_size() for the save, the restore and the copy between differently chunked ranges, and assert that each range covers whole blocks.
ggerganov
approved these changes
Oct 14, 2026
ggerganov
left a comment
ggerganov
left a comment
Member
There was a problem hiding this comment.
Thanks for catching this. Just remove the one-line comments - they are not needed.
lhez
marked this pull request as ready for review
October 14, 2026 20:02
3 of 6 tasks
Contributor
Author
|
@ggerganov, done with the comments. Can you have one more person review? |
CISC
approved these changes
Nov 3, 2026
ServeurpersoCom
approved these changes
Nov 4, 2026
ServeurpersoCom
left a comment
ServeurpersoCom
left a comment
Contributor
There was a problem hiding this comment.
LGTM, please apply CISC's suggestion so the three sites read the same.
am17an
approved these changes
Nov 5, 2026
7 tasks done
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Nov 6, 2026
…org#30541) * llama : fix on-device state copy sizes for quantized KV caches The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed their element count as size / ggml_element_size(). For a quantized type such as q8_0 or q4_0 that is the number of blocks, not elements, so only part of each range was copied: the restored state was wrong, or the copy crashed. Compute the count as size / ggml_type_size() * ggml_blck_size() for the save, the restore and the copy between differently chunked ranges, and assert that each range covers whole blocks. * llama : drop the one-line comments * llama : drop the redundant cast in the state copy size (cherry picked from commit e56f006)
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 9, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 10, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 10, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 10, 2026
…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
With
LLAMA_STATE_SEQ_FLAGS_ON_DEVICE,llama_state_seq_get_data_ext()/llama_state_seq_set_data_ext()keep the sequence state in device-side copies of the KV cache ranges. The element count of each copy was computed assize / ggml_element_size(tensor). For a quantized KV cache (q8_0, q4_0, ...) that is the number of blocks, not elements, so only part of each range was copied (1/32 for q8_0 and q4_0): the restored state was wrong, or the copy crashed.The count is now
size / ggml_type_size(type) * ggml_blck_size(type)in the three places that use it (save, restore, and the copy between differently chunked ranges), with an assert that each range covers whole blocks. f16 and f32 caches are unaffected.Additional information
tests/test-save-load-statewith-ctk q8_0 -ctv q8_0 -fa on(and the same with q4_0), CPU backend: "seq copy (device)" fails and "seq copy (device, scatter)" crashes on master; both pass with this change. f16 results are unchanged. Also checked on the OpenCL backend together with #30471.Requirements