Repository navigation
graph: fix CI realloc abort by gathering the recurrent states once - #29856
Merged
ServeurpersoCom merged 1 commit intoOct 3, 2026
Merged
ServeurpersoCom merged 1 commit into
ServeurpersoCom merged 1 commit into
Conversation
…plit build_rs gathered the extra states (n_rs - n_seqs rows) with their own get_rows. The worst-case reserve has n_rs == n_seqs, so that node was sized at zero rows, and any ubatch whose cells are not contiguous forced a graph reallocation at an unchanged node count, which aborts under GGML_SCHED_NO_REALLOC. A single get_rows now gathers the n_rs states: the ubatch states and the extra states are views of it, and its size only depends on n_rs, which the reserve already sets to the maximum. A custom getter (mamba ssm_scan) gathers from the second state, so a single sequence ubatch copies no state. The views are built once per graph in the input to keep the host overhead of the graph unchanged.
ggerganov
approved these changes
Oct 2, 2026
CISC
approved these changes
Oct 2, 2026
ravi9
added a commit
to ravi9/llama.cpp
that referenced
this pull request
Oct 5, 2026
ggml-org#29856 changed build_rs to gather all recurrent states with one GET_ROWS on the s_copy leaf and take the ubatch and extra states as views of it. The stateful path matched only the previous form, a GET_ROWS per view of s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU ("is_axis_valid(axis, r)" in a Concat). For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the active-state gather, keep the rank-4 layout of reshapes that read a view of it, and map the copy of the empty extra-state view to the single-slot remainder writeback. Do not warn about the dynamic dim of empty views.
5 tasks
ravi9
added a commit
to ravi9/llama.cpp
that referenced
this pull request
Oct 6, 2026
ggml-org#29856 changed build_rs to gather all recurrent states with one GET_ROWS on the s_copy leaf and take the ubatch and extra states as views of it. The stateful path matched only the previous form, a GET_ROWS per view of s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU ("is_axis_valid(axis, r)" in a Concat). For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the active-state gather, keep the rank-4 layout of reshapes that read a view of it, and map the copy of the empty extra-state view to the single-slot remainder writeback. Do not warn about the dynamic dim of empty views.
ggerganov
pushed a commit
that referenced
this pull request
Oct 6, 2026
* ggml-openvino: skip unselected graph branches and support DUP Upstream #29622 adds a mixed token/embd branch to every input embedding graph through ggml_build_forward_select(). Its nodes are not flagged for compute, but the backend translated them anyway, and the DUP in that branch was unsupported, so the scheduler split the graph and passed the embeddings across the split with a fixed token count. The first single-token decode then failed (test-thread-safety on CPU and GPU). Build the OV model from the compute nodes only, and translate a same-type contiguous DUP like CONT so the graph stays on one backend. * ggml-openvino: make inp_scale_rows token dim dynamic #29622 also moves the per-token embedding scale (gemma3, gemma3n, gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim and pad it per chunk on the static (NPU) path. * ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU plugin fails to compile that form for some row counts with "clFinish, error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on the MUL_MAT cases added in #29869 (e.g. m=1000, n=2, k=1024). Model weights use a u4 zero point and are not affected. Report these cases as unsupported on GPU until the plugin is fixed. Op tests check support before allocating, so the check matches unbound weights only; model loading probes with a dummy buffer and keeps its weights on the GPU. * ggml-openvino: create FILL in the output type translate_fill always built an f32 constant, so an f16 FILL produced f32 data and the copy back overran the f16 output buffer. Use the output type for the constant. * ggml-openvino: reject CONCAT with a quantized type Quantized inputs are dequantized when translated, so the backend cannot write a quantized CONCAT output. Report it as unsupported, as for CPY to a quantized type. * ggml-openvino: handle the single recurrent state gather of build_rs #29856 changed build_rs to gather all recurrent states with one GET_ROWS on the s_copy leaf and take the ubatch and extra states as views of it. The stateful path matched only the previous form, a GET_ROWS per view of s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU ("is_axis_valid(axis, r)" in a Concat). For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the active-state gather, keep the rank-4 layout of reshapes that read a view of it, and map the copy of the empty extra-state view to the single-slot remainder writeback. Do not warn about the dynamic dim of empty views. * openvino: align eltwise operand ranks to work around a GPU-plugin defect * openvino: match the MoE fusion on the rank-3 stateful graph * ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract whose operand ranks differ. In gemma-3 the lower-rank operand of the post-attention residual add is the norm output, and unsqueezing it makes the GPU plugin compute the layer wrongly: gemma-3 returns empty answers on GPU with stateful execution. Skip the rewrite when the lower-rank operand is an RMS norm output. * docs : update OpenVINO validated models --------- Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 8, 2026
…plit (ggml-org#29856) build_rs gathered the extra states (n_rs - n_seqs rows) with their own get_rows. The worst-case reserve has n_rs == n_seqs, so that node was sized at zero rows, and any ubatch whose cells are not contiguous forced a graph reallocation at an unchanged node count, which aborts under GGML_SCHED_NO_REALLOC. A single get_rows now gathers the n_rs states: the ubatch states and the extra states are views of it, and its size only depends on n_rs, which the reserve already sets to the maximum. A custom getter (mamba ssm_scan) gathers from the second state, so a single sequence ubatch copies no state. The views are built once per graph in the input to keep the host overhead of the graph unchanged. (cherry picked from commit 436f6f8)
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 8, 2026
* ggml-openvino: skip unselected graph branches and support DUP Upstream ggml-org#29622 adds a mixed token/embd branch to every input embedding graph through ggml_build_forward_select(). Its nodes are not flagged for compute, but the backend translated them anyway, and the DUP in that branch was unsupported, so the scheduler split the graph and passed the embeddings across the split with a fixed token count. The first single-token decode then failed (test-thread-safety on CPU and GPU). Build the OV model from the compute nodes only, and translate a same-type contiguous DUP like CONT so the graph stays on one backend. * ggml-openvino: make inp_scale_rows token dim dynamic ggml-org#29622 also moves the per-token embedding scale (gemma3, gemma3n, gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim and pad it per chunk on the static (NPU) path. * ggml-openvino: skip GPU MUL_MAT op tests with unbound Q4_1/Q4_K weights Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU plugin fails to compile that form for some row counts with "clFinish, error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on the MUL_MAT cases added in ggml-org#29869 (e.g. m=1000, n=2, k=1024). Model weights use a u4 zero point and are not affected. Report these cases as unsupported on GPU until the plugin is fixed. Op tests check support before allocating, so the check matches unbound weights only; model loading probes with a dummy buffer and keeps its weights on the GPU. * ggml-openvino: create FILL in the output type translate_fill always built an f32 constant, so an f16 FILL produced f32 data and the copy back overran the f16 output buffer. Use the output type for the constant. * ggml-openvino: reject CONCAT with a quantized type Quantized inputs are dequantized when translated, so the backend cannot write a quantized CONCAT output. Report it as unsupported, as for CPY to a quantized type. * ggml-openvino: handle the single recurrent state gather of build_rs ggml-org#29856 changed build_rs to gather all recurrent states with one GET_ROWS on the s_copy leaf and take the ubatch and extra states as views of it. The stateful path matched only the previous form, a GET_ROWS per view of s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU ("is_axis_valid(axis, r)" in a Concat). For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the active-state gather, keep the rank-4 layout of reshapes that read a view of it, and map the copy of the empty extra-state view to the single-slot remainder writeback. Do not warn about the dynamic dim of empty views. * openvino: align eltwise operand ranks to work around a GPU-plugin defect * openvino: match the MoE fusion on the rank-3 stateful graph * ggml-openvino: do not unsqueeze an RMS norm output in AlignEltwiseOperandRanks The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract whose operand ranks differ. In gemma-3 the lower-rank operand of the post-attention residual add is the norm output, and unsqueezing it makes the GPU plugin compute the layer wrongly: gemma-3 returns empty answers on GPU with stateful execution. Skip the rewrite when the lower-rank operand is an RMS norm output. * docs : update OpenVINO validated models --------- Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> (cherry picked from commit b9a5a00)
sodre90
added a commit
to sodre90/llama.cpp
that referenced
this pull request
Oct 8, 2026
…match (revert ggml-org#29856) Our GDN state gather, SSM_CONV fold and GET_ROWS->CPY fusions match a GET_ROWS that GATED_DELTA_NET / CONCAT reads directly. Upstream's view of one gather stops them. The abort it fixes only happens under GGML_SCHED_NO_REALLOC, which the fork does not set.
2 tasks
mclrcha
added a commit
to mclrcha/llama.cpp
that referenced
this pull request
Oct 10, 2026
…-10-10) Conflicts: - ggml-backend.cpp: upstream moved input copies to ggml_backend_sched_copy_input with host weights last (ggml-org#29943); the GGML_SCHED_BATCH_INPUTS fast path now runs in the first loop before falling back to it. - ggml-cuda/fattn-common.cuh: launch_fattn takes upstream async_kv_preload, then our parallel_blocks_fixed (GQA-6 tile launcher updated). - ggml-cuda/fattn-mma-f16.cuh: upstream swizzle refactor (ggml-org#29612); its helpers take ncols2 like the rest of the fork's config getters. - ggml-cuda/gated_delta_net.cu: keep the fork's kernel and launch geometry (its columns_per_block template argument is not upstream's cols_per_warp, ggml-org#30087). - ggml-cuda/mmq.cu: upstream per-tile src1 padding (ggml-org#29953); ggml_cuda_mul_mat_q_src1_nbytes and the MMQ input cache use the worst-case padding so producers and projections with other tiles can share the buffer. - ggml-cuda/mmq-vec-dot.cuh: fork vec_dot kernels pass GGML_PREC_Q8 to the config helpers (ggml-org#30168). - ggml-cuda/mmvf.cu/.cuh: keep both includes and the HIP declarations, upstream warp_size argument. - ggml-cuda/rope.cu: upstream grid-stride rms_norm+mul+rope (ggml-org#28175) with the fork's M-RoPE theta (nchannels instead of gridDim.y) and rope_store_cast. - ggml-cuda/top-k.cu: upstream shape-based selection (ggml-org#28713), RDNA4 tournament kept for few long rows. - src/llama-batch.*: mixed token/embd batches (ggml-org#29622) copy embd rows per entry; other batches keep pointing at the ext rows. - src/llama-context.cpp: expert copy callback also on the second MTP scheduler. - src/llama-graph.cpp: build_rs keeps the GET_ROWS of the ubatch states (GDN fusions, deferred state) and gathers the extra states from the tail like upstream's custom getter path (ggml-org#29856).
2 tasks
asf0
pushed a commit
to asf0/strix-llama.cpp
that referenced
this pull request
Oct 12, 2026
halo-box/llama.cpp is archived, so its last master comes in once: the 2026-10-07 upstream sync (halo halo-box#34) with its resolutions, plus the archival README of that repository is not taken (strix keeps its own README). Resolution rules: MTP follows upstream's design; everywhere else strix's side wins where the two collide, and upstream's non-conflicting changes stay. - qwen4exp: strix's QSA stack (selected-key / WMMA block selection, incremental pooled keys, PLE prefetch, HIP prefill work) is kept whole in qwen4exp.cpp and the qwen4exp struct of models.h. strix's graph_mtp is kept: it builds the same draft as upstream's ggml-org#29761 (eh_proj of [enorm(e); hnorm(h)], the draft's own nextn.hc_head_* mixer, sparse QSA attention) and also loads shared sidecars without embeddings / LM head. The converter, gguf-py tensor maps and arch tables are upstream's, so new conversions follow upstream's layout; strix loads both. - llama-memory-hybrid-idx: both APIs kept side by side - strix's qsa_* for qwen4exp and upstream's k-pool for the new GLM5Next (ggml-org#27773 and follow-ups). strix's context set_input_qsa, which upstream deleted, is restored. - llama-graph: strix's in-place gated-delta-net state path is kept; upstream's single recurrent-state gather (ggml-org#29856, avoids a realloc abort under GGML_SCHED_NO_REALLOC) is reverted on this path and needs porting. - vulkan: strix's flash_attn.comp and lightning_indexer.comp are kept whole; upstream's FA shmem out-of-bounds fix (ggml-org#29988) is ported into strix's shader. Upstream's lightning indexer rewrite (ggml-org#29825) is reverted on the Vulkan host side to match strix's shader; its CUDA/Metal parts stay. Upstream's sparse FA path (ggml-org#29639) stays out: strix deleted its shader. - ggml-alloc: upstream's alloc_buffer_n refactor (ggml-org#23671) replaces strix's backport of the view-init fix; strix's regression test is kept. - cuda: strix's RDNA3.5 paths kept; upstream's shared-expert MoE fusion added next to strix's fused GDN decode, with both sets of fusion args. - server: speculative accept keeps strix's replay branch, then upstream's rejection sampling at temp > 0. - tests: both sides' new tests kept (rollback table gets both columns, save-load-state test 10 is strix's, 11 upstream's); strix's rollback test migrated to common_batch / llama_process. - fusion.yml kept (upstream removed it), flash_attn_sparse_compact.comp stays deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
asf0
pushed a commit
to asf0/strix-llama.cpp
that referenced
this pull request
Oct 12, 2026
The 39 upstream commits after halo-box/llama.cpp's final sync. Same rules as the previous merge: strix's side wins where the two collide, upstream's non-conflicting changes stay, and fixes are ported into strix's code. - server: upstream's final ggml-org#26004 (context checkpoints across slot save/restore) replaces strix's earlier port of it, keeping strix's on-device restore flags; test_slot_save.py follows upstream. - cuda: strix's top-k (HIP radix), gated delta net and argsort kernels kept whole; upstream's CCCL major-version guard fix (ggml-org#29453) applied to strix's argsort. MMQ: upstream's out-of-bounds fix and host-side tile choice (ggml-org#29953) with strix's RDNA3.5 tile overrides on top; strix's fused MMQ paths pad src1 for the largest tile instead of the removed get_J_max. strix's 128-column rms_norm fast path and two mmvf calls updated to upstream's new signatures (ggml-org#28175, warp_size). - vulkan: strix's flash_attn_cm2.comp and topk_nary_search.comp kept whole (upstream's sparse cm2 FA ggml-org#30003 stays out with the rest of sparse FA). - llama-context: strix's MTP draft vocabulary next to upstream's MoE cache. - graph: upstream's single recurrent-state gather (ggml-org#29856) is restored and combined with strix's in-place gated delta net: in-place capable consumers keep the full gather and get their state view next to the consumer whether or not the rows are in place, so in-place and gathered ubatches build the same graph; this fixes the realloc aborts under GGML_SCHED_DEBUG_REALLOC in both upstream's shared-seq test and strix's seq cp graph reuse test. - ggml-backend: alloc_buffer_n planning no longer flushes on zero-size tensors, which left views of a tensor larger than max_size uninitialized (caught by strix's test_view_init_after_max_size_split). - SYNC.md: this fork now syncs with ggml-org/llama.cpp directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaetan-puleo
pushed a commit
to rcfa/strix-llama.cpp
that referenced
this pull request
Oct 16, 2026
…e cache build_rs gains an allow_inplace option: when the sequence-copy mapping is the identity, the state input is a view of the cache slot rather than a get_rows copy. qwen35 and qwen35moe both opt in, since they build the same Gated DeltaNet graph through llm_build_delta_net_base. s_copy_main_identity joins the graph-reuse key: a seq_cp between steps changes whether the mapping is the identity, so a graph built for one must not be reused for the other. test-recurrent-state-rollback gains a check that decodes two sequences in shared ubatches while seq_cp swaps them, forks one and rolls it back, with graph reuse on. The logits are compared with a context without graph reuse (bit-exact), one with n_rs_seq = 1 and one that decodes one sequence per ubatch. It uses fresh random weights at scale 1: at the 0.01 scale of the generated models the recurrent branch does not change the logits, so a wrong state would not be seen. The check runs for qwen35 and qwen35moe, and the test now also runs on the generated qwen35moe model. Assisted-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MkuKwgPNgpAg7EbxKR2iWC (cherry picked from commit e3f31cf) [rebase: ported onto the single state gather of upstream ggml-org#29856 with a fixed graph structure (llm_rs_inplace: the same nodes for both mappings), so test-llama-context with debug_realloc passes; the qwen35moe rollback ctest entry is dropped with upstream's removal of test-recurrent-state-rollback]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fix CI / graph
Hybrid models keep a recurrent state per sequence, and build_rs sometimes has to move the states of other sequences back into place. The worst-case reserve did not account for those moved states, so the first time one appeared the graph was reallocated mid-run: an abort under GGML_SCHED_NO_REALLOC (test_systemone since #29818) and a silent reallocation otherwise.
build_rs now gathers all the states at once in a buffer that the reserve always sizes for the maximum, so this reallocation no longer happens. The output is bit-exact and decode speed is unchanged.
Additional information
Follow-up #29818
Requirements