Repository navigation
ggml: fix conv_1d result layout for batched input - #28738
Merged
ggerganov merged 2 commits intoOct 14, 2026
Merged
Conversation
github-actions
Bot
added
testing
This was referenced Sep 15, 2026
ServeurpersoCom
force-pushed
the
ggml/fix-conv-1d-batch
branch
from
September 23, 2026 12:52
1bd25fe to
1365a4f
Compare
Contributor
Author
|
Rebase only |
This was referenced Sep 24, 2026
ServeurpersoCom
added a commit
to ServeurpersoCom/acestep.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/minimaxmusic.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/yue2.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/omnivoice.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/qwentts.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/qwenasr.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/pockettts.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/s2s.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ggerganov
reviewed
Oct 14, 2026
Comment on lines
+4630
to
+4637
| // the GEMM flattens the batch into its rows, so the result comes back as | ||
| // [OC, N, OL] and the batched case needs OC and N swapped back | ||
| if (im2col->ne[2] == 1) { | ||
| result = ggml_reshape_3d(ctx, result, im2col->ne[1], a->ne[2], 1); // [OC, OL] => [1, OC, OL] | ||
| } else { | ||
| result = ggml_reshape_3d(ctx, result, im2col->ne[1], im2col->ne[2], a->ne[2]); // [OC, N, OL] | ||
| result = ggml_cont(ctx, ggml_permute(ctx, result, 0, 2, 1, 3)); // [N, OC, OL] | ||
| } |
Member
There was a problem hiding this comment.
It's better to not have this branch - it would cause different graphs for different shapes.
Contributor
Author
There was a problem hiding this comment.
Done, the branch is gone: conv_1d now always reshapes and permutes like conv_2d, so every batch size builds the same graph. Pushed on my fork + ReTested on qwentts.cpp: outputs are bit-exact, and the extra cont at N == 1 costs about 5% of the codec decode, 0.3% of the whole synthesis.
The GEMM behind conv_1d folds the batch into its rows, so the product sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N]. The shape is right and the contents are transposed, so nothing fires: no assert, no abort, the graph just runs and the channels come out interleaved with the batches. With N == 1 the two readings describe the same buffer, which is why the three vocoders in the tree never hit it, they all squeeze to 2d right after the call. Reshape to the real layout and permute OC and N back. The N == 1 path keeps its direct reshape, so the current callers build the same graph and pay for no extra copy. tests/test-conv-1d.cpp compares one batched convolution against N single batch convolutions built from the same weight and the same input slices. Five of the seven geometries fail before this change and all seven pass after. N == 1 and OC == 1 pass either way, since swapping two dimensions when one of them is 1 changes nothing. The batched path matters to downstream audio projects where several synthesis lanes share a single graph. Nothing in mtmd batches a vocoder today, each one streams one flow and carries its left context between calls, but the op is correct now if one of them grows a lane dimension.
ServeurpersoCom
force-pushed
the
ggml/fix-conv-1d-batch
branch
from
October 14, 2026 07:41
1365a4f to
6c67527
Compare
Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d. The GEMM result is always reshaped to [OC, N, OL] and permuted back, as ggml_conv_2d does, so every batch size builds the same graph. For N == 1 this adds a cont on an already ordered tensor.
ggerganov
approved these changes
Oct 14, 2026
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fix batched conv_1d so vocoders can process several streams in one graph. Nothing in mtmd batches a vocoder today, so no in-tree output changes: it is a ggml level fix for downstream projects that do, such as qwentts.cpp, and for the day mtmd wants to.
Additional information
ggml_conv_1dreads the GEMM result as [OL, OC, N] when it is actually [OL, N, OC]. A reshape only relabels the buffer, so the shape looks right and the values come out interleaved between batches and channels. Nothing asserts, the graph just runs and the audio is scrambled. Every caller in the tree is N == 1, where both readings are the same bytes, which is why nobody hit it.test-backend-opscannot catch this since it compares a backend against the CPU on the same wrong graph.tests/test-conv-1d.cppinstead compares a batched convolution against N single batch ones: 5 of 7 geometries fail before, all 7 pass after.Updated :
Reshape to the real layout and permute back, the same way
ggml_conv_2ddoes, so every batch size builds the same graph. For N == 1 this adds a cont on an already ordered tensor: outputs are bit-exact, and in qwentts.cpp it costs about 5% of the codec decode, 0.3% of the whole synthesis.Requirements