Repository navigation
CUDA: bitonic argsort handles rows wider than one block - #28957
Merged
IMbackK merged 1 commit intoSep 29, 2026
Merged
Conversation
github-actions
Bot
added
ggml
Contributor
|
I think this deserves a test case |
Contributor
Author
The cases at 1025, 2047 and 2048 columns already exist in test-backend-ops, they were just skipped on HIP as unsupported, so this PR is what makes them run there for the first time, and the gpu-rocm check below went through them. |
Contributor
|
I see, the fact that qwentts crashes instead of falling back to CPU for that op threw me off. |
Contributor
Author
Right, qwentts.cpp runs its graphs straight on one backend without the scheduler, so there is no CPU fallback there. That is on purpose: I want the raw performance, and a hard failure surfaces every backend gap through user reports instead of hiding it behind a silent fallback and slow performance |
JohannesGaessler
approved these changes
Sep 17, 2026
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread per padded column, so any row above 1024 entries launched an invalid block configuration. Each thread now owns several columns, every stage of the network runs all owned columns before the barrier, and the block is capped at 1024 threads. Shared memory becomes the only bound, which supports_op checks against the device instead of a fixed 1024. Rows up to 1024 run the same work as before. Bit-exact with the CUB path on rows of 2048.
ServeurpersoCom
force-pushed
the
cuda-argsort-bitonic-wide-rows
branch
from
September 23, 2026 12:50
e301898 to
f8fecff
Compare
Contributor
Author
|
Rebase only |
Contributor
Author
|
@IMbackK could you give it a second approval? It is the HIP/MUSA path you looked at, tested on ROCm by the qwentts.cpp reporter. |
IMbackK
approved these changes
Sep 29, 2026
pierreguillot
pushed a commit
to Ircam-Partiels/llama.cpp
that referenced
this pull request
Oct 1, 2026
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread per padded column, so any row above 1024 entries launched an invalid block configuration. Each thread now owns several columns, every stage of the network runs all owned columns before the barrier, and the block is capped at 1024 threads. Shared memory becomes the only bound, which supports_op checks against the device instead of a fixed 1024. Rows up to 1024 run the same work as before. Bit-exact with the CUB path on rows of 2048.
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 1, 2026
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread per padded column, so any row above 1024 entries launched an invalid block configuration. Each thread now owns several columns, every stage of the network runs all owned columns before the barrier, and the block is capped at 1024 threads. Shared memory becomes the only bound, which supports_op checks against the device instead of a fixed 1024. Rows up to 1024 run the same work as before. Bit-exact with the CUB path on rows of 2048.
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 4, 2026
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread per padded column, so any row above 1024 entries launched an invalid block configuration. Each thread now owns several columns, every stage of the network runs all owned columns before the barrier, and the block is capped at 1024 threads. Shared memory becomes the only bound, which supports_op checks against the device instead of a fixed 1024. Rows up to 1024 run the same work as before. Bit-exact with the CUB path on rows of 2048. (cherry picked from commit 6a2743f)
1 task
frostyautumnleaf
pushed a commit
to frostyautumnleaf/llama.cpp
that referenced
this pull request
Oct 5, 2026
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread per padded column, so any row above 1024 entries launched an invalid block configuration. Each thread now owns several columns, every stage of the network runs all owned columns before the barrier, and the block is capped at 1024 threads. Shared memory becomes the only bound, which supports_op checks against the device instead of a fixed 1024. Rows up to 1024 run the same work as before. Bit-exact with the CUB path on rows of 2048.
ServeurpersoCom
added a commit
to ServeurpersoCom/acestep.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/minimaxmusic.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/yue2.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/omnivoice.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/qwentts.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/qwenasr.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/pockettts.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
ServeurpersoCom
added a commit
to ServeurpersoCom/s2s.cpp
that referenced
this pull request
Oct 7, 2026
The vendored fork rebases onto ggml-org/ggml 0.26.0, which now carries the fixes upstreamed from these projects: - Vulkan: mul_mat and mul_mat_id read strided views such as a KV cache slice with their real batch stride, and mul_mm keeps exact A and B ranges (ggml-org/llama.cpp#28956) - Metal: GGML_OP_PAD supports left and circular padding (ggml-org/llama.cpp#29561) - CUDA: bitonic argsort handles rows wider than 1024 without CUB, on HIP and MUSA (ggml-org/llama.cpp#28957) Two commits remain on top of upstream: - cpu: fused snake activation - ggml: conv_1d result layout for batched input (ggml-org/llama.cpp#28738)
2 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fix argsort on HIP and MUSA builds for rows wider than 1024. Without CUB the bitonic kernel launched one thread per padded column, so a 2048 wide argsort aborted with an invalid block configuration. Nothing in-tree sorts rows that wide, it is a ggml level fix for downstream projects that do, such as qwentts.cpp.
Additional information
Each thread now owns several columns and the block is capped at 1024 threads, shared memory becomes the only bound and supports_op checks it against the device. Rows up to 1024 run the same work as before, CUDA still switches to CUB above 1024. Bit-exact against CUB on NVIDIA with the bitonic kernel forced on rows of 2048, and tested on ROCm by the reporter of ServeurpersoCom/qwentts.cpp#33 on a Radeon 780M.
No new test: the already existing argsort cases at 1025, 2047 and 2048 columns were skipped on HIP as unsupported, they now run on the ROCm ggml-ci.
Requirements