Repository navigation
metal : few-row MMA mat-mul for the remaining src0 types - #30065
Merged
Merged
Conversation
The generic few-row MMA kernel works for any type with a 16-weight dequantizer, so it now also takes BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0 and the IQ types. Each type starts at the row count where it beats the current kernels on an M3 Ultra: 5 rows for TQ2_0, 4 for BF16, 3 for MXFP4, Q2_0, Q2_K and IQ4_NL, and 2 for the others. test-backend-ops perf -o MUL_MAT, m=4096, k=14336, M3 Ultra, time of this change over master (mean of two interleaved runs each): 0.23 to 0.98 from the threshold to 8 rows, 0.24 to 0.33 at 9 to 16 rows, and 0.99 to 1.01 at 1 and 512 rows.
github-actions
Bot
added
testing
ggerganov
marked this pull request as ready for review
October 7, 2026 07:41
ggerganov
approved these changes
Oct 7, 2026
ggerganov
added
the
merge ready
thelittlefireman
pushed a commit
to thelittlefireman/llama.cpp
that referenced
this pull request
Oct 12, 2026
) The generic few-row MMA kernel works for any type with a 16-weight dequantizer, so it now also takes BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0 and the IQ types. Each type starts at the row count where it beats the current kernels on an M3 Ultra: 5 rows for TQ2_0, 4 for BF16, 3 for MXFP4, Q2_0, Q2_K and IQ4_NL, and 2 for the others. test-backend-ops perf -o MUL_MAT, m=4096, k=14336, M3 Ultra, time of this change over master (mean of two interleaved runs each): 0.23 to 0.98 from the threshold to 8 rows, 0.24 to 0.33 at 9 to 16 rows, and 0.99 to 1.01 at 1 and 512 rows.
Achilles1089
added a commit
to Achilles1089/llama.cpp
that referenced
this pull request
Oct 15, 2026
…gml-org#29869 + ggml-org#30065, without their MUL_MAT+ADD fusion); stop accepting draft tokens at EOG (ggml-org#29638); graph capture flag fix (ggml-org#29390) The 2..16-row mat-muls that MTP verify produces ran on mat-vec kernels whose time grows with every row. Measured on the M1 Ultra with Raven's Qwen3.8-27B Q8_0 shapes: 8 rows now cost 1.6-1.9x one row (was 3.3-3.8x), matching upstream within 1-3%. End to end (temp 1.0, 512 tokens): at --spec-draft-n-max 6 --spec-draft-n-min 1 --spec-draft-p-min 0.6, code echo 34.9 -> 54.7 tok/s and tool JSON 35.2 -> 54.9, prose level (25.6 -> 25.9). Greedy output is identical to the previous build, with and without drafting. test-backend-ops MUL_MAT: 1163/1163. The capture fix: the -1 capture counter was passed as a bool, so every node of every graph got debug-group markers.
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Split out of #30047, as asked there: the MUL_MAT part only.
The few-row MMA mat-mul from #29869 now takes every src0 type with a 16-weight dequantizer, adding BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0 and the IQ types. They run on the generic MMA kernel, so the change is new instantiations plus the dispatch. Each type starts at the row count where it beats the current kernels on an M3 Ultra: 5 for TQ2_0, 4 for BF16, 3 for MXFP4, Q2_0, Q2_K and IQ4_NL, and 2 for the rest.
Additional information
test-backend-ops perf -o MUL_MAT, m=4096, k=14336, Apple M3 Ultra (60-core GPU), macOS 15.7.9. Speedups are the time on master divided by the time with this PR, from two interleaved runs of each build. A dash marks a row count below the type's threshold, where the kernel is the same as on master:At 9 to 16 rows the new types run 3.06x to 4.14x faster. The types that already had MMA kernels (F32, F16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q4_K, Q5_K, Q6_K) run the same kernel code as on master and measure 0.97x to 1.03x at 1 to 16 rows, and every type is within 1% at 512 rows. Where the speedup does not grow with n, as for Q3_K at n=3, it follows the time of the kernel on master, which does not grow monotonically with n.
Some cells move between runs on this machine. On master, the MXFP4 and IQ4_NL kernels at 3 to 8 rows differed by up to 22% between two sessions (MXFP4 at n=4: 175 us and 202 us), and with this PR the BF16 kernel at n=4 took between 152 us and 174 us (1.17x to 1.34x).
The
mul_mv_mmalibrary takes longer to compile:xcrun metal -con an Apple M5 takes 0.33 s on master and 0.67 s with this PR.Tests:
test-backend-ops -o MUL_MAT,MUL_MAT_ID -b MTL0: 3314/3314 on the M3 Ultra. On an Apple M5 with the tensor API on and off (GGML_METAL_TENSOR_DISABLE=1): MUL_MAT and MUL_MAT_ADD 2587/2587.Requirements