Repository navigation
vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 - #27952
Merged
Merged
Conversation
github-actions
Bot
added
Vulkan
0cc4m
marked this pull request as ready for review
August 29, 2026 12:10
jeffbolznv
reviewed
Aug 29, 2026
| #define ACC_BIAS_F 12582912.0f | ||
| const bool USE_MAGIC_BIAS = WARP != 32; | ||
|
|
||
| // Accumulator row for element e: RDNA4 blocked, RDNA3/3.5 interleaved. |
Contributor
There was a problem hiding this comment.
Are these layouts entirely based on the hardware definition, or do they depend partly on the compiler. For example, at some point I changed the NVIDIA compiler so the layout of a 16x16 matrix in four registers was permuted to 0,2,1,3 vs earlier compiler versions.
Contributor
Author
There was a problem hiding this comment.
I think it's a little bit of both. I have confirmed that Linux' RADV driver based this layout on what the hardware requires, so it should be fixed. I have also checked that at least RDNA3.5 uses the same layout on the Windows driver, and this branch does improve performance there too.
I think we can replace this with a generic solution once enough drivers support coopmat maintenance1.
Contributor
There was a problem hiding this comment.
On the other PR, you mentioned that you couldn't probe the layout because it used too many registers, but if the matrix is 16x16 then we only need 4 bits each for row+col and that's only a couple registers for wave32. I'm surprised storing those would cause a perf issue.
Contributor
Author
There was a problem hiding this comment.
I removed it in a number of optimizations that reduced overall register use. It's possible I can readd it now, but I don't think it's worth the complexity with coopmat maintenance1 on the way.
| return; | ||
| } | ||
| #else | ||
| // L2-friendly workgroup scheduling |
Contributor
There was a problem hiding this comment.
Would be interesting to separate out what perf gain is from this vs using int8. I've been tempted to do something like this for the other shaders, but with ubatch=512 still being the default it usually doesn't matter much (probably helps more for stable diffusion kind of workloads)
Contributor
Author
There was a problem hiding this comment.
Very small, hard to measure. Maybe 2% on RDNA3.5, but my Strix Halo laptop has a lot of thermal noise in benchmarks, so not sure. Might not be worth it. I didn't measure a difference on RDNA4.
This was referenced Aug 30, 2026
|
Built b10760 + this pr. Using a RX7900XT (RDNA3) with RADV. Running usloth/Qwen3.8-27B-Q4_0.gguf before applying this PR I was seeing: after the same sort of prefill gets (+8% at start to +3% at end of prefill) There is too much variance in token gen to say anything. This has positive effects in a real use case. |
Contributor
|
Apologies if this is not helpful / spam, but I have been running this on my 7900XTX for a few days and it has been a big improvement, so wanted to share a before / after llama bench run: master: this PR: |
Contributor
Author
|
@jeffbolznv Can you take a look at this? I know you can't test it, but at least to check the code is not breaking anything else. |
jeffbolznv
reviewed
Sep 7, 2026
| #define ACC_BIAS_F 12582912.0f | ||
| const bool USE_MAGIC_BIAS = WARP != 32; | ||
|
|
||
| // Accumulator row for element e: RDNA4 blocked, RDNA3/3.5 interleaved. |
Contributor
There was a problem hiding this comment.
On the other PR, you mentioned that you couldn't probe the layout because it used too many registers, but if the matrix is 16x16 then we only need 4 bits each for row+col and that's only a couple registers for wave32. I'm surprised storing those would cause a perf issue.
|
|
||
| const uint CM_ELEMS = (TM * TN) / WARP; | ||
| #define ACC_BIAS_BITS 0x4B400000 | ||
| #define ACC_BIAS_F 12582912.0f |
Contributor
There was a problem hiding this comment.
I assume this is one of these "add an integer to the mantissa of this float value" tricks, but I couldn't easily tell what's going on.
| #include "mul_mmq_cm1_funcs.glsl" | ||
|
|
||
| void main() { | ||
| #if defined(DATA_A_IQ4_NL) |
Contributor
There was a problem hiding this comment.
I assume this will get the spec constant treatment before it goes in?
Contributor
Author
There was a problem hiding this comment.
I checked and this is not easily possible, because unlike the fp16 shaders which dequant into shared memory, the MMQ shaders have to have custom shmem layouts for the quantized data in shared memory. This would get us into trouble with shmem size tricks again that previously especially the Nvidia driver did not like. Do you see a way to handle this?
Contributor
There was a problem hiding this comment.
No, I don't have any clever ideas for this.
| const uint32_t tk_m = device->coopmat_support ? device->coopmat_k : 1; | ||
| const uint32_t tk_s = device->coopmat_support ? device->coopmat_k : 1; | ||
|
|
||
| const uint32_t itm_l = device->coopmat_int_support ? device->coopmat_int_m : 4; |
Contributor
There was a problem hiding this comment.
codex had one interesting finding here:
- [P1] Keep cooperative-matrix dimensions out of scalar MMQ tiles.
ggml/src/ggml-vulkan/ggml-vulkan.cpp:4389 now uses coopmat_int_* for l/m/s_warptile_mmq_int. If integer cooperative matrices are advertised but the new AMD CM1 path is not
selected, these dimensions reach the scalar mul_mmq.comp shader, where TM/TN/TK have different meanings. Preserve the old scalar values and use the queried dimensions only in
*_mmq_cm1_int.
Contributor
Author
There was a problem hiding this comment.
Yes, that was not correct. Fixed.
NickM-27
added a commit
to NickM-27/llama.cpp
that referenced
this pull request
Sep 8, 2026
NickM-27
added a commit
to NickM-27/llama.cpp
that referenced
this pull request
Sep 9, 2026
|
Please rebase on top of master head, there are merge conflicts |
Contributor
Author
No, this is a PR, it exists to facilitate merging this into the master branch, not for you to merge and use. There's a plan to get this merged, I'm following it. |
|
Could you share the plan that you are following? |
Contributor
Author
|
It touches a lot of code, and there will be more merge conflicts with other PRs that get merged first. Rebasing now is just a waste of effort. |
It took me 25 seconds to do rebase to master head, it is less than your answer.
You didn't answer to simple question, where is that plan you are following ? |
Contributor
Author
|
It's in my head. Let me do my job. Stop this discussion or you will get blocked. |
NickM-27
added a commit
to NickM-27/llama.cpp
that referenced
this pull request
Sep 9, 2026
Member
|
@vlforit did we miss you somehow becoming the main coordinator of llama.cpp? If not, please be respectful to maintainers instead of demanding they engage you on things that are purely your personal whims. |
NickM-27
added a commit
to NickM-27/llama.cpp
that referenced
this pull request
Sep 9, 2026
Oh, that explains a lot. |
0cc4m
force-pushed
the
0cc4m/vulkan-coopmat-int8
branch
from
September 10, 2026 06:35
224807f to
8c7611a
Compare
|
Discrete RDNA3 data point, not a bisect. RX 7800 XT (Windows, AMD proprietary Vulkan, KHR_coopmat), Qwen3.6-35B-A3B Q4_K_M, same flags on c77ae69 vs 84e76d8 (this PR is in the second build, plus 145 other commits).pp512 1174 to 1151 t/s (flat, not the Strix Halo +32%). g32 without MTP 42.4 to 36.1 t/s. MTP n=3 tg128: 55.8 to 25.8 t/s at 48k, 39.2 to 27.1 t/s at 130k, acceptance still 1.00. Full writeup: #29410. |
|
Post-merge data point from an RDNA3 APU (unified memory, so please read this as iGPU-specific — it matches the edt-xx RX 7900XT result above in direction but is a very different memory regime). Hardware / software
Method — three binaries, identical build flags, benched interleaved in one session (all three in the same process conditions, so drift cancels):
The workloads are the two models this box actually serves: Gemma 4 E2B IQ4_XS (
* the pp32768 baseline is from an earlier session; the other rows are same-session. Pass-to-pass spread on pp3461 is ±0.3%. Result — on this APU essentially the entire prefill win comes from this PR's coopmat1 path. Reverting it gives back ~94% of the pp3461 gain; the generic int8-dot IQ4_XS MMQ/MMVQ kernels from #28415 measure ~+2.6% (pp3461), ~+1.4% (32K depth) and ~0% on the Q8_0 embedder. Decode is unchanged in every build, consistent with an iGPU being bandwidth-bound on generation. Two notes that may be useful:
Measured with llama-bench through a scripted, counter-balanced harness; raw per-build output retained. |
Jamaika1
reviewed
Sep 25, 2026
| } | ||
| #undef X_CM1 | ||
|
|
||
| if (device->coopmat_int_support && (rdna3 || rdna4)) { |
There was a problem hiding this comment.
Add definitions.
#if defined(GGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT)
Contributor
Author
There was a problem hiding this comment.
No, it doesn't use integer dot.
Jamaika1
reviewed
Sep 25, 2026
Contributor
|
I was also trying to make this work for qwen3.8 27B on strix halo but I was not able to make it work and while trying that found this PR. Glad to see it working. Few learnings from the mistakes I made:
|
|
I'm unfortunately seeing sub 1% improvements on RDNA4, do we need to enable something or I am just unlucky with Q5 weights? |
Contributor
Author
|
See benchmarks in the PR description, RDNA4 improvements are mostly for MoE. |
6 tasks
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 1, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. Adapted to this tree: - pipelines are created with the existing CREATE_MM-era helpers since the upstream pipeline-map refactor (91f6a6c) is not here - the cm1 MUL_MAT_ID shader uses the non-hoisted row ids and push constants of this tree (hoisting, 90c26fc, is not here) - WARP_SIZE_IDX taken from upstream 1945e09 Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 1, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. Adapted to this tree: - pipelines are created with the existing CREATE_MM-era helpers since the upstream pipeline-map refactor (91f6a6c) is not here - the cm1 MUL_MAT_ID shader uses the non-hoisted row ids and push constants of this tree (hoisting, 90c26fc, is not here) - WARP_SIZE_IDX taken from upstream 1945e09 Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 1, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. Adapted to this tree: - pipelines are created with the existing CREATE_MM-era helpers since the upstream pipeline-map refactor (91f6a6c) is not here - the cm1 MUL_MAT_ID shader uses the non-hoisted row ids and push constants of this tree (hoisting, 90c26fc, is not here) - WARP_SIZE_IDX taken from upstream 1945e09 Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 2, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 2, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
gianni-cor
pushed a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 3, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
gianni-cor
pushed a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 3, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
|
This regressed RDNA4 for batch sizes above 4 on Gemma 3 31B QAT by ~40%. Before:
After:
(The command was According to Opus, this is because since this MR RDNA4 is treated as separate arch, so some optimizations are no longer enabled. Specifically, this line: // RDNA3: above four columns, static 4 rows for all types bench faster than the default
const bool is_rdna3 = device->vendor_id == VK_VENDOR_ID_AMD && device->architecture == AMD_RDNA3;From Changing it to also allow RDNA4 restores the performance. |
Contributor
Author
|
You're right, I overlooked that. Will fix. |
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 5, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
gianni-cor
pushed a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 5, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 5, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
1 task
frostyautumnleaf
pushed a commit
to frostyautumnleaf/llama.cpp
that referenced
this pull request
Oct 5, 2026
…gml-org#27952) * vulkan: add int8 coopmat quantized matmul shader * apply scales inline * use scalar sums * probe and directly access coopmat values instead of going through shmem * add q8_0 support * add BK_STEP to shader, default to 2 * use larger workgroups * double buffering * preload scales * coopmat load first, then wmma * use float for scales * add faster RDNA int->float conversion * workgroup scheduling for cache proximity * clean up * use wave32 * restructure for vgpr use * skip computation for inactive tiles * only force subgroup size 32 on AMD RDNA * use BK_STEP 4 * fix compilation * move quant-specific prefetch function out of main file * add q4_1, q5_0, q5_1 support * restructure mmq cm1 functions * enable mul_mat_id support * fix segfault * fix mul_mat_id bug * support iq4_nl and mxfp4 * remove elem row/col fast path, invalid for RDNA4 * use shmem arrays for LUTs * use 4-byte loads where possible * add q3_k, q4_k, q5_k, q6_k and nvfp4 support * fix l warptile * improve performance * improve performance * improvements * dedup b scales * merge shmem arrays * undo uint8_t, gate to RDNA3/4 * add RDNA4 architecture, use for hardcoded coopmat elem thread access, set BK_STEP back to 4 * improve offset application * clean up * fix iq4_nl and nvfp4 performance * rdna4 tuning * use BK_STEP 2 on MUL_MAT_ID * adapt to upstream changes * fix shmem support function, clean up comments * fix warptile logic Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * vulkan: add IQ4_XS support to the coopmat1 integer matmul shader (ggml-org#28440) Adds IQ4_XS to mul_mmq_cm1: dedicated block_a_load/block_a_to_shmem that expand both nibbles of each packed32 word through cm1_kvalues, LOAD_VEC_A 8 and an IQ4_XS-sized a_panel_bytes estimate for the L2-friendly scheduling. Assisted-by: OpenAI Codex Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * avoid compiling f16 acc shader variants --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 7, 2026
…gml-org#27952) * vulkan: add int8 coopmat quantized matmul shader * apply scales inline * use scalar sums * probe and directly access coopmat values instead of going through shmem * add q8_0 support * add BK_STEP to shader, default to 2 * use larger workgroups * double buffering * preload scales * coopmat load first, then wmma * use float for scales * add faster RDNA int->float conversion * workgroup scheduling for cache proximity * clean up * use wave32 * restructure for vgpr use * skip computation for inactive tiles * only force subgroup size 32 on AMD RDNA * use BK_STEP 4 * fix compilation * move quant-specific prefetch function out of main file * add q4_1, q5_0, q5_1 support * restructure mmq cm1 functions * enable mul_mat_id support * fix segfault * fix mul_mat_id bug * support iq4_nl and mxfp4 * remove elem row/col fast path, invalid for RDNA4 * use shmem arrays for LUTs * use 4-byte loads where possible * add q3_k, q4_k, q5_k, q6_k and nvfp4 support * fix l warptile * improve performance * improve performance * improvements * dedup b scales * merge shmem arrays * undo uint8_t, gate to RDNA3/4 * add RDNA4 architecture, use for hardcoded coopmat elem thread access, set BK_STEP back to 4 * improve offset application * clean up * fix iq4_nl and nvfp4 performance * rdna4 tuning * use BK_STEP 2 on MUL_MAT_ID * adapt to upstream changes * fix shmem support function, clean up comments * fix warptile logic Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * vulkan: add IQ4_XS support to the coopmat1 integer matmul shader (ggml-org#28440) Adds IQ4_XS to mul_mmq_cm1: dedicated block_a_load/block_a_to_shmem that expand both nibbles of each packed32 word through cm1_kvalues, LOAD_VEC_A 8 and an IQ4_XS-sized a_panel_bytes estimate for the L2-friendly scheduling. Assisted-by: OpenAI Codex Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * avoid compiling f16 acc shader variants --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 70c4e15)
4 tasks
pratiknarola-t
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 8, 2026
Ports upstream 70c4e15 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4, ggml-org#27952): the mul_mmq_cm1 shaders, AMD_RDNA4 detection, cm1 int8 shared-memory checks and warptiles, and the q8_1 path selection on coopmat_int_support. The shaders are byte-identical to upstream, including the hoisted MUL_MAT_ID row ids and push constants (90c26fc is in this tree). The pipelines are created the way upstream does it, through create_mm_pipelines with the tc_mmq_cm1_int and tc_mmq_cm1_int_k tile tables. Adapted to this tree: - quantize_y also requires that src0 needs no reformat copy and that src1 is dim01-contiguous. This tree sends non-dim01-contiguous quantized src0 through ggml_vk_mul_mat_q_f16 for its quant->f16 copy, and ggml_is_contiguous accepts permuted size-1 dims, so without the old !y_non_contig term the q8_1 path met the reformat copies (wrong MUL_MAT results, and a dispatch of a never-requested copy pipeline). The src0 check also fixes int-dot devices without coopmat, where the parent already paired the q8_1 path with the src0 copy. test_mul_mat_permuted_src0 covers the src0 side. - AMD_RDNA4 goes into the architecture enum in ggml-vulkan.cpp, since ggml-vulkan-types.h (f172be7) is not here - the IQ4_NL and NVFP4 cases upstream adds to ggml_vk_matmul_int_shmem_support are left out: this tree has no int-dot q8_1 pipelines for those types, so they would be dead - the cm1 q8_1 shaders get the device_prefix that the other matmul shaders in vulkan-shaders-gen use - the cm1 q8_1 shaders are generated in the fp16 coopmat pass only. This tree also runs the matmul shader pass for f32 coopmat inputs (e0bee17), which would otherwise build fp32 q8_1 variants that no pipeline creates Upstream e85e15c (ggml-org#29409) needs no change here: its guards cover the Intel FA decode shaders this tree does not have, and the cm1 shaders are only generated when glslc supports coopmat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Currently with tensor/matrix cores we only use fp16 instructions, so dequant to shared memory -> fp16 GEMM. But many devices have higher throughput int8 tensor/matrix cores, so similar to the existing DP4A-based scalar MMQ path it would be good to have a coopmat int8 MMQ path. I have tried to make this work in the past, but failed due to architectural constraints in the coopmat extension.
But based on @pwilkin's ideas in #27493, I finally managed to get it to work with good performance on RDNA3 and 4. This new MMQ cm1 shader supports q4_0, q4_1, q5_0, q5_1, q8_0, q3_k, q4_k, q5_k, q6_k, mxfp4, nvfp4 and iq4_nl on RDNA3 and RDNA4. Strix Halo prompt processing performance is significantly improved, RDNA4 is more neutral, but MoE prompt processing is also up significantly. The shader is limited to those two architectures since it hardcodes their specific coopmat access patterns. More architectures can be added if needed.
q4_1, q5_1, q4_k, q5_k and nvfp4 on RDNA4 are disabled for MUL_MAT since they run slower than existing fp16 matmul. Additionally nvfp4 is also disabled for MUL_MAT_ID.
Benchmarks
AMD Radeon 8060S (Strix Halo, RDNA3.5)
MUL_MAT test-backend-ops perf geomean us (lower = better)
MUL_MAT_ID test-backend-ops perf geomean us (lower = better)
llama-bench pp512 t/s (higher = better)
AMD Radeon AI PRO R9700 (RDNA4)
MUL_MAT test-backend-ops perf geomean us (lower = better)
MUL_MAT_ID test-backend-ops perf geomean us (lower = better)
llama-bench pp512 t/s (higher = better)
Requirements