vulkan: use spec constant for matrix matrix multiplication A-type by 0cc4m · Pull Request #25773 · ggml-org/llama.cpp

0cc4m · 2026-07-16T12:21:23Z

Overview

This is an attempt to reduce the amount of matrix multiplication shaders we need by using a spec constant for the type of the A input, similar to how it works for KV cache types in Flash Attention. Additionally, it replaces the current static s, m and l shapes with a map and allows improved tuning through more shapes and a custom selection function, which should help with work like #24407.

It's still draft because currently it regresses Nvidia, because for dynamic type selection all quants with shared memory requirements must also be available. I thought it would eliminate those that are not used, this works on AMD, but not on Nvidia. @jeffbolznv Do you know why? There are other ways to resolve this, hopefully I'll find something that works.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, AI was used to draft the code, I reviewed, tested and corrected it afterwards.

jeffbolznv · 2026-07-16T15:29:03Z

I had experimented with something like this and ended up with one spirv for all the legacy and K quants and then separate spirvs for all the iq quants. That resolves the shared memory issue and probably isn't a ton of bloat because those spirvs are dominated by the table size anyway.

Fwiw, even if our compiler dead code eliminated the shared memory arrays, they would still count against the spec limit.

jeffbolznv · 2026-07-17T08:31:24Z

I think the shared memory arrays with length 1 could run into this compiler bug again #24924.

0cc4m · 2026-07-24T06:11:55Z

I don't see these CI failures locally, not sure what is causing them. Is that the compiler bug?

jeffbolznv · 2026-07-24T15:59:05Z

I don't see these CI failures locally, not sure what is causing them. Is that the compiler bug?

I also can't reproduce the cm1 failures locally. It's conceivable it's the Turing-only bug. I think it's more likely to happen if you use more fp16 math, is there any place you've changed fp32 math to fp16?

0cc4m · 2026-07-24T16:15:04Z

No, the PR should not contain any functional changes, it's all just restructuring. The running code in the end should be basically identical.

0cc4m · 2026-08-20T06:35:18Z

@ggerganov The Nvidia-CM pipeline is timing out in test-backend-ops, I haven't seen that before. Is that expected? Is it possible to increase the timeout?

ggerganov · 2026-08-20T07:49:54Z

Hm, not sure - I can't see a reason for the increased time. It almost doubled than usual for this job (~15 mins). Also, looking at the most recent runs in https://github.com/ggml-org/llama.cpp/actions/workflows/build-self-hosted.yml it seems to be fluctuating between slow/fast without a clear point where it started happening.

Let's keep monitoring. These workflows are currently running on DGX Spark that I self-host. We are already looking for ways to migrate this to some more stable and scalable infrastructure, but don't have ETA atm.

0cc4m · 2026-08-20T07:51:43Z

Did it recently switch from T4 to the Spark? That would explain why it now passed some tests that were failing earlier due (likely to the Turing driver bug.

ggerganov · 2026-08-20T07:55:33Z

The DGX Spark has been running these and other jobs for about 3 months now (added on May 26). The T4 runners are also running - whoever runner picks of the job first will execute it.

jeffbolznv · 2026-08-20T13:47:02Z

@ggerganov The Nvidia-CM pipeline is timing out in test-backend-ops, I haven't seen that before. Is that expected? Is it possible to increase the timeout?

Maybe this change effects enough shaders that it get no benefit from the shader disk cache from previous runs? If so, a rerun might be enough to make it complete.

0cc4m · 2026-08-24T06:48:15Z

I reran it multiple times and it keeps failing, so I don't think that's it. I'll try to reproduce locally.

0cc4m · 2026-08-24T08:30:09Z

It consistently happens in the CI, but I cannot reproduce it locally. I get worst case 7 minutes on my DGX Spark, with caching just a little bit more than 3 minutes. I'm not sure what is going on there.

ggerganov · 2026-08-24T08:45:41Z

Here are the logs of one slow and one fast run:

slow
fast

One thing that draws attention is that in the slow run, the cmake builds at the start are quite a bit slower. F.ex:

# slow
2026-08-24T07:26:39.7092878Z real	5m8.856s
2026-08-24T07:26:39.7093100Z user	16m45.155s
2026-08-24T07:26:39.7093300Z sys	1m53.851s
2026-08-24T07:26:39.7095059Z + '[' -z ']'
2026-08-24T07:26:39.7098364Z + tee -a /home/ggml/results/llama.cpp/ctest_release-ctest.log

# fast
2026-08-24T05:27:37.5154950Z real	1m18.797s
2026-08-24T05:27:37.5155099Z user	3m29.091s
2026-08-24T05:27:37.5155197Z sys	0m47.416s
2026-08-24T05:27:37.5157269Z + '[' -z ']'
2026-08-24T05:27:37.5159526Z + tee -a /home/ggml/results/llama.cpp/ctest_release-ctest.log

ggerganov · 2026-08-24T08:46:39Z

It consistently happens in the CI, but I cannot reproduce it locally. I get worst case 7 minutes on my DGX Spark, with caching just a little bit more than 3 minutes. I'm not sure what is going on there.

Is this 7 minutes for the entire ci/run.sh script?

0cc4m · 2026-08-24T08:50:52Z

Just for test-backend-ops

0cc4m · 2026-08-24T10:19:31Z

The entire CI script went from 22:46 on master uncached to 12:09 cached, with the PR from 13:14 uncached to 7:44 cached, on my Spark. That isn't consistent with the CI either.

ggerganov · 2026-08-24T10:32:22Z

I just tried manually - on both my DGX Sparks, this command deadlocks/blocks with Vulkan on this branch:

GGML_VK_DISABLE_COOPMAT2=1 ./bin/test-backend-ops -o MUL_MAT

There is a suspicious error though at the start about incompatible driver:

ggml@spark-17ed:~/development/llama.cpp/build-vulkan$ GGML_VK_DISABLE_COOPMAT2=1 ./bin/test-backend-ops -o MUL_MAT
TU: error: ../src/freedreno/vulkan/tu_knl.cc:385: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA Tegra NVIDIA GB10 (NVIDIA) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 0 | matrix cores: KHR_coopmat
Testing 2 devices

Backend 1/2: Vulkan0
  Device description: NVIDIA Tegra NVIDIA GB10
  Device memory: 91929 MB (91923 MB free)

  MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK

Does this command complete on your end?
Any suggestions about what could be wrong causing this error?

Edit: the same command works fine on master. The error is still displayed.

0cc4m · 2026-08-24T10:54:32Z

Yes, it completes without issue. It's suspicious that your driver reports the device as NVIDIA Tegra NVIDIA GB10, mine shows NVIDIA GB10. What version are you on? vulkaninfo --summary shows it, for example for me it's:

GPU0:
        apiVersion         = 1.4.312
        driverVersion      = 580.159.3.0
        vendorID           = 0x10de
        deviceID           = 0x2e12
        deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
        deviceName         = NVIDIA GB10
        driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
        driverName         = NVIDIA
        driverInfo         = 580.159.03
        conformanceVersion = 1.4.1.3
        deviceUUID         = 89b1d82b-72d4-cfde-55b7-4adf19b0e8b3
        driverUUID         = 02d1943b-b572-5849-bc0a-9110d99e29a2

The driver warning is a mesa adreno Vulkan driver artifact, you can ignore it.

ggerganov · 2026-08-24T11:06:45Z

Here is spark 1 (the one from my last command) that shows "tegra":

Details

ggml@spark-17ed:~$ vulkaninfo --summary
'DISPLAY' environment variable not set... skipping surface info
TU: error: ../src/freedreno/vulkan/tu_knl.cc:385: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.335


Instance Extensions: count = 25
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1
VK_NV_display_stereo                   : extension revision 1

Instance Layers: count = 13
---------------------------
VK_LAYER_KHRONOS_profiles         Khronos Profiles layer                                                                                            1.4.335  version 1
VK_LAYER_KHRONOS_shader_object    Khronos Shader object layer                                                                                       1.4.335  version 1
VK_LAYER_KHRONOS_synchronization2 Khronos Synchronization2 layer                                                                                    1.4.335  version 1
VK_LAYER_KHRONOS_validation       Khronos Validation Layer                                                                                          1.4.335  version 1
VK_LAYER_LUNARG_api_dump          LunarG API dump layer                                                                                             1.4.335  version 2
VK_LAYER_LUNARG_crash_diagnostic  Crash Diagnostic Layer is a crash/hang debugging tool that helps determines GPU progress in a Vulkan application. 1.4.335  version 1
VK_LAYER_LUNARG_gfxreconstruct    GFXReconstruct Capture Layer Version 1.0.5                                                                        1.4.335  version 4194309
VK_LAYER_LUNARG_monitor           Execution Monitoring Layer                                                                                        1.4.335  version 1
VK_LAYER_LUNARG_screenshot        LunarG image capture layer                                                                                        1.4.335  version 1
VK_LAYER_MESA_device_select       Linux device selection layer                                                                                      1.4.303  version 1
VK_LAYER_MESA_overlay             Mesa Overlay layer                                                                                                1.4.303  version 1
VK_LAYER_NV_optimus               NVIDIA Optimus layer                                                                                              1.4.312  version 1
VK_LAYER_NV_present               NVIDIA GR2608 layer                                                                                               1.4.312  version 1

Devices:
========
GPU0:
	apiVersion         = 1.4.312
	driverVersion      = 580.95.5.0
	vendorID           = 0x10de
	deviceID           = 0x2e12
	deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
	deviceName         = NVIDIA Tegra NVIDIA GB10
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 580.95.05
	conformanceVersion = 1.4.1.3
	deviceUUID         = e13a03d3-eb3d-950a-7ca1-bfb87613a30c
	driverUUID         = b92269a1-b525-5615-ab8a-e2095ee37192
GPU1:
	apiVersion         = 1.4.305
	driverVersion      = 0.0.1
	vendorID           = 0x10005
	deviceID           = 0x0000
	deviceType         = PHYSICAL_DEVICE_TYPE_CPU
	deviceName         = llvmpipe (LLVM 20.1.2, 128 bits)
	driverID           = DRIVER_ID_MESA_LLVMPIPE
	driverName         = llvmpipe
	driverInfo         = Mesa 25.0.7-0ubuntu0.24.04.2 (LLVM 20.1.2)
	conformanceVersion = 1.3.1.1
	deviceUUID         = 6d657361-3235-2e30-2e37-2d3075627500
	driverUUID         = 6c6c766d-7069-7065-5555-494400000000

Here is the spark 2, which actually runs the CI - it does not show "tegra" for some reason:

Details

ggml@spark-a163:~$ vulkaninfo --summary
'DISPLAY' environment variable not set... skipping surface info
TU: error: ../src/freedreno/vulkan/tu_knl.cc:387: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
WARNING: [Loader Message] Code 0 : ICD for selected physical device does not export vkGetPhysicalDeviceDisplayPlanePropertiesKHR!
WARNING: [Loader Message] Code 0 : ICD for selected physical device does not export vkGetPhysicalDeviceDisplayPropertiesKHR!
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.341


Instance Extensions: count = 25
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1
VK_NV_display_stereo                   : extension revision 1

Instance Layers: count = 14
---------------------------
VK_LAYER_INTEL_nullhw             INTEL NULL HW                                                                                                     1.1.73   version 1
VK_LAYER_KHRONOS_profiles         Khronos Profiles layer                                                                                            1.4.341  version 1
VK_LAYER_KHRONOS_shader_object    Khronos Shader object layer                                                                                       1.4.341  version 1
VK_LAYER_KHRONOS_synchronization2 Khronos Synchronization2 layer                                                                                    1.4.341  version 1
VK_LAYER_KHRONOS_validation       Khronos Validation Layer                                                                                          1.4.341  version 1
VK_LAYER_LUNARG_api_dump          LunarG API dump layer                                                                                             1.4.341  version 2
VK_LAYER_LUNARG_crash_diagnostic  Crash Diagnostic Layer is a crash/hang debugging tool that helps determines GPU progress in a Vulkan application. 1.4.341  version 1
VK_LAYER_LUNARG_gfxreconstruct    GFXReconstruct Capture Layer Version 1.0.5                                                                        1.4.341  version 4194309
VK_LAYER_LUNARG_monitor           Execution Monitoring Layer                                                                                        1.4.341  version 1
VK_LAYER_LUNARG_screenshot        LunarG image capture layer                                                                                        1.4.341  version 1
VK_LAYER_MESA_device_select       Linux device selection layer                                                                                      1.4.303  version 1
VK_LAYER_MESA_overlay             Mesa Overlay layer                                                                                                1.4.303  version 1
VK_LAYER_NV_optimus               NVIDIA Optimus layer                                                                                              1.4.312  version 1
VK_LAYER_NV_present               NVIDIA GR2608 layer                                                                                               1.4.312  version 1

Devices:
========
GPU0:
	apiVersion         = 1.4.312
	driverVersion      = 580.126.9.0
	vendorID           = 0x10de
	deviceID           = 0x2e12
	deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
	deviceName         = NVIDIA GB10
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 580.126.09
	conformanceVersion = 1.4.1.3
	deviceUUID         = 032999c3-c6e9-ce30-47c7-b69dc4aff29e
	driverUUID         = 9756eed7-117e-56eb-8928-8e249b39bb59
GPU1:
	apiVersion         = 1.4.318
	driverVersion      = 25.2.8
	vendorID           = 0x10005
	deviceID           = 0x0000
	deviceType         = PHYSICAL_DEVICE_TYPE_CPU
	deviceName         = llvmpipe (LLVM 20.1.2, 128 bits)
	driverID           = DRIVER_ID_MESA_LLVMPIPE
	driverName         = llvmpipe
	driverInfo         = Mesa 25.2.8-0ubuntu0.24.04.1 (LLVM 20.1.2)
	conformanceVersion = 1.3.1.1
	deviceUUID         = 6d657361-3235-2e32-2e38-2d3075627500
	driverUUID         = 6c6c766d-7069-7065-5555-494400000000

The jobs also have this info at the start of the logs: https://github.com/ggml-org/llama.cpp/actions/runs/32224960472/job/97381646994?pr=25773#step:3:54. For example this job, runs on Runner name: 'ggml-dgx-spark-2-runner-4' - this is a docker container inside the spark 2.

0cc4m · 2026-08-24T11:43:38Z

Sorry, I made a mistake testing, my environment wasn't set up correctly and I didn't notice. I can actually reproduce it, I'll look into it.

0cc4m · 2026-08-26T05:20:19Z

I excluded LUT quants from the new unified shader, which seems to avoid whatever bug Nvidia was hitting with coopmat1. I'll clean up the code, then this should be ready for review.

vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types

0cc4m · 2026-09-09T08:34:28Z

This one is next to merge, please briefly check and reapprove @jeffbolznv

…ml-org#25773) * vulkan: use spec constant for mul mat type_a vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types * remove LUT quants from unified shader * clean up * restore coopmat2 q4_k/q5_k optimization * split out q4_k/q5_k cm2 shader to fix Ampere regression * revert iq shmem table renames * simplify cm2 code with single uint8_t buffer * fix fp4 extension use switch being overwritten by generic shader * clean up * adapt TQ1_0 changes * adapt ggml-org#27471 f16 Intel tuning changes

330 commits behind. Merging current master gives 13 conflicting files / 47 hunks, and ggml-org#25773 alone accounts for 15 of them -- that step is where the ROCmFPx types must be re-expressed in the new spec-constant matmul scheme. Four steps, benchmark each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ml-org#25773) * vulkan: use spec constant for mul mat type_a vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types * remove LUT quants from unified shader * clean up * restore coopmat2 q4_k/q5_k optimization * split out q4_k/q5_k cm2 shader to fix Ampere regression * revert iq shmem table renames * simplify cm2 code with single uint8_t buffer * fix fp4 extension use switch being overwritten by generic shader * clean up * adapt TQ1_0 changes * adapt ggml-org#27471 f16 Intel tuning changes

github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Jul 16, 2026

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from a619971 to 496b099 Compare July 17, 2026 07:18

jeffbolznv reviewed Jul 24, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_cm2.comp Outdated

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from cbf3468 to 001600c Compare July 28, 2026 13:09

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from 001600c to 197898e Compare August 19, 2026 06:48

This was referenced Aug 26, 2026

vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps #27726

Merged

Vulkan: wrong matmul results on Adreno (subgroupSize=128) — warptile WM exceeds BM (implicit WM <= BM invariant violated for subgroupSize > 64) #25734

Closed

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from df88488 to 19e4a29 Compare August 27, 2026 06:34

0cc4m mentioned this pull request Sep 8, 2026

vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 #27471

Merged

0cc4m added 11 commits September 9, 2026 08:23

remove LUT quants from unified shader

abebc83

clean up

e1f9066

restore coopmat2 q4_k/q5_k optimization

103f29e

split out q4_k/q5_k cm2 shader to fix Ampere regression

1004d80

revert iq shmem table renames

f14bab2

simplify cm2 code with single uint8_t buffer

0c35eae

fix fp4 extension use switch being overwritten by generic shader

8b43777

clean up

febd0e4

adapt TQ1_0 changes

537c220

adapt #27471 f16 Intel tuning changes

b1b70dd

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from b363e78 to b1b70dd Compare September 9, 2026 08:18

jeffbolznv approved these changes Sep 9, 2026

View reviewed changes

jeffbolznv mentioned this pull request Sep 9, 2026

vulkan: reduce stack usage in ggml_vk_load_shaders #28658

Closed

aldehir approved these changes Sep 9, 2026

View reviewed changes

0cc4m merged commit 91f6a6c into master Sep 9, 2026
27 of 30 checks passed

0cc4m deleted the 0cc4m/vulkan-mul-mm-refactor branch September 9, 2026 16:53

snailium mentioned this pull request Sep 10, 2026

[Vulkan] Deep-context decode collapses ~8x on Intel Arc Pro B70 (Xe2); RDNA3 does not #28721

Open

c4software mentioned this pull request Sep 13, 2026

vulkan : after the 2026-09-11 upstream sync (#47), batched mat-vec at 5..8 columns regresses on dense Qwen3.8-27B (pp8 79 -> 52 t/s) halo-box/strix-llama.cpp#51

Open

ggerganov mentioned this pull request Sep 14, 2026

ggml : bump version to 0.24.0 ggml-org/ggml#1627

Merged

BrewTestBot mentioned this pull request Sep 14, 2026

ggml 0.24.0 Homebrew/homebrew-core#310890

Merged

1 task done

dzannotti mentioned this pull request Sep 29, 2026

Merge halo-box/llama.cpp master into strix (2026-09-29) halo-box/strix-llama.cpp#111

Merged

srflaherty mentioned this pull request Oct 5, 2026

Eval bug: Vulkan prompt processing 16-19% slower for Q8_0 on RTX 5060 Ti (NV_coopmat2) since #25773 #30000

Closed

Conversation

0cc4m commented Jul 16, 2026

Overview

Requirements

Uh oh!

jeffbolznv commented Jul 16, 2026

Uh oh!

jeffbolznv commented Jul 17, 2026

Uh oh!

0cc4m commented Jul 24, 2026

Uh oh!

Uh oh!

jeffbolznv commented Jul 24, 2026

Uh oh!

0cc4m commented Jul 24, 2026

Uh oh!

0cc4m commented Aug 20, 2026

Uh oh!

ggerganov commented Aug 20, 2026

Uh oh!

0cc4m commented Aug 20, 2026

Uh oh!

ggerganov commented Aug 20, 2026

Uh oh!

jeffbolznv commented Aug 20, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 26, 2026

Uh oh!

0cc4m commented Sep 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ggerganov commented Aug 24, 2026 •

edited

Loading