vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 by fish-jiang · Pull Request #27471 · ggml-org/llama.cpp

fish-jiang · 2026-08-21T06:38:12Z

Overview

Add f16 B-type matmul pipelines for Intel coopmat1.
Warp tile size tuning for Intel coopmat1.
Scope note: this is split out of vulkan: GEMM/Group GEMM optimizations on Intel Xe (3/3, Xe-LPG Plus/Xe2/Xe3) #24407

Performance (Windows OS)

Xe1-ARL_H
Before:
C:\kernel\llama.cpp\build_base\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf

model	size	params	backend	ngl	fa	test	t/s
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	0	pp512	106.60 ± 0.16
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	1	pp512	103.85 ± 0.43
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	0	pp512	3409.77 ± 10.33
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	1	pp512	3453.43 ± 41.74
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	0	pp512	414.20 ± 1.12
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	1	pp512	427.43 ± 1.59
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	0	pp512	109.99 ± 0.71
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	1	pp512	111.86 ± 0.22
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	0	pp512	347.98 ± 4.65
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	1	pp512	353.19 ± 5.92
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp512	486.52 ± 1.38
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	pp512	514.71 ± 4.20

After:
C:\kernel\llama.cpp\build\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf

model	size	params	backend	ngl	fa	test	t/s
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	0	pp512	138.66 ± 0.46
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	1	pp512	133.81 ± 0.01
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	0	pp512	3539.43 ± 5.35
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	1	pp512	3542.68 ± 21.19
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	0	pp512	532.65 ± 2.49
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	1	pp512	545.46 ± 0.67
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	0	pp512	133.56 ± 0.86
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	1	pp512	133.21 ± 0.72
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	0	pp512	366.83 ± 6.08
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	1	pp512	365.20 ± 6.01
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp512	568.09 ± 1.56
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	pp512	603.93 ± 4.82

Xe3-PTL
Before:
C:\Users\dungeon\Desktop\fish\llama.cpp\build_ d59d\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\qwen3-8b-q4_k_m\qwen3-8b-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\qwen2.5-1.5b-instruct-q4_k_m\qwen2.5-1.5b-instruct-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\Llama-3.1-8B-Instruct-Q4_K_M\Llama-3.1-8B-Instruct-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.8-27B-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf

model	size	params	backend	ngl	fa	test	t/s
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	0	pp512	1011.55 ± 29.38
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	1	pp512	1022.33 ± 29.91
qwen2 1.5B Q4_K - Medium	1.04 GiB	1.78 B	Vulkan	99	0	pp512	4418.79 ± 581.94
qwen2 1.5B Q4_K - Medium	1.04 GiB	1.78 B	Vulkan	99	1	pp512	4231.56 ± 563.52
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	99	0	pp512	1015.82 ± 8.45
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	99	1	pp512	1105.96 ± 2.56
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	0	pp512	284.39 ± 2.62
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	1	pp512	263.66 ± 0.69
qwen35moe 35B.A3B Q4_K - Medium	21.10 GiB	35.51 B	Vulkan	99	0	pp512	841.70 ± 7.98
qwen35moe 35B.A3B Q4_K - Medium	21.10 GiB	35.51 B	Vulkan	99	1	pp512	791.32 ± 10.30
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp512	1163.01 ± 13.35
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	pp512	1276.05 ± 1.22

After:
C:\Users\dungeon\Desktop\fish\llama.cpp\build\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\qwen3-8b-q4_k_m\qwen3-8b-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\qwen2.5-1.5b-instruct-q4_k_m\qwen2.5-1.5b-instruct-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\Llama-3.1-8B-Instruct-Q4_K_M\Llama-3.1-8B-Instruct-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.8-27B-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics i gfx-driver-ci-comp_igc-34261 DCH RI (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	fa	test	t/s
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	0	pp512	1139.91 ± 25.88
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	1	pp512	1149.18 ± 79.59
qwen2 1.5B Q4_K - Medium	1.04 GiB	1.78 B	Vulkan	99	0	pp512	5352.63 ± 193.44
qwen2 1.5B Q4_K - Medium	1.04 GiB	1.78 B	Vulkan	99	1	pp512	5210.39 ± 89.54
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	99	0	pp512	1270.88 ± 14.97
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	99	1	pp512	1291.23 ± 29.36
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	0	pp512	358.42 ± 0.86
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	1	pp512	342.91 ± 1.17
qwen35moe 35B.A3B Q4_K - Medium	21.10 GiB	35.51 B	Vulkan	99	0	pp512	887.73 ± 10.83
qwen35moe 35B.A3B Q4_K - Medium	21.10 GiB	35.51 B	Vulkan	99	1	pp512	887.54 ± 3.34
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp512	1218.73 ± 30.89
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	pp512	1337.71 ± 29.63

Xe2-B70
Before:
C:\kernel\llama.cpp\build_base\bin\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf

model	size	params	backend	ngl	fa	test	t/s
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	0	pp512	679.62 ± 0.48
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	1	pp512	604.42 ± 1.61
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	0	pp512	21110.67 ± 180.71
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	1	pp512	17368.51 ± 140.55
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	0	pp512	2852.25 ± 59.26
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	1	pp512	2707.06 ± 25.70
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	0	pp512	758.01 ± 1.26
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	1	pp512	748.29 ± 0.77
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	0	pp512	2179.01 ± 18.09
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	1	pp512	2136.83 ± 15.25
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp512	3085.12 ± 21.90
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	pp512	3108.67 ± 35.17

After:
C:\kernel\llama.cpp\build\bin\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf

model	size	params	backend	ngl	fa	test	t/s
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	0	pp512	907.72 ± 6.05
gemma4 31B Q4_K - Medium	17.05 GiB	30.70 B	Vulkan	99	1	pp512	774.29 ± 2.08
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	0	pp512	21530.34 ± 227.02
qwen3 0.6B Q4_K - Medium	456.11 MiB	751.63 M	Vulkan	99	1	pp512	17533.69 ± 101.30
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	0	pp512	3669.75 ± 255.18
qwen3 8B Q4_K - Medium	4.68 GiB	8.19 B	Vulkan	99	1	pp512	3481.77 ± 55.19
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	0	pp512	995.25 ± 1.99
qwen35 27B Q4_K - Medium	15.92 GiB	27.32 B	Vulkan	99	1	pp512	969.67 ± 1.97
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	0	pp512	2440.35 ± 29.64
qwen35moe 35B.A3B Q4_K - Medium	20.60 GiB	34.66 B	Vulkan	99	1	pp512	2375.47 ± 21.98
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp512	3345.54 ± 46.36
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	pp512	3378.68 ± 63.72

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, used claude code, then lots of manual review/tweaking.

jeffbolznv

Haven't done a full review, just some initial comments.

jeffbolznv · 2026-08-21T13:22:55Z

@@ -9208,6 +9308,10 @@ static void ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, vk_context& sub
    const bool x_non_contig = (ctx->device->coopmat2 && src0->type == GGML_TYPE_F32) ||
                              !ggml_vk_dim01_contiguous(src0);
    const bool y_non_contig = (ctx->device->coopmat2 && src1->type == GGML_TYPE_F32) ||
+                              // Intel coopmat1: force f32->f16 conversion so the f16-B-type pipeline is used.


Maybe we should do more testing and try to enable this across all devices? Reducing the load bandwidth for the B matrix seems like it should generally be a win. I've also thought about potentially fusing the f32->f16 conversion into the previous node to eliminate any overhead. But even without that, this is a win on NV and at least some Intel, so it seems likely it's a win everywhere.

Yes, testing on more platforms would be good, but I don't have a non-Intel GPU on hand right now. May try to test later.
Fusing the f32->f16 conversion into the previous node is a great idea — it could reduce the overhead of that conversion.

jeffbolznv · 2026-08-21T13:23:20Z

@@ -7908,6 +7993,19 @@ static vk_matmul_pipeline ggml_vk_get_mul_mat_mat_id_pipeline(ggml_backend_vk_co
        case GGML_TYPE_MXFP4:
        case GGML_TYPE_NVFP4:
        case GGML_TYPE_TQ2_0:
+            if (src1_type == GGML_TYPE_F16 && !ctx->device->coopmat2) {


Can this be merged into the logic at the end of the function? I think it's just a different selection of mmp.

Thanks, updated

newbeelearn · 2026-08-24T09:44:15Z

this will also be on windows only for intel arl_h right? since coopmat is not enabled on linux due to driver issue?

0cc4m · 2026-08-26T14:12:01Z

I tested it on AMD and Nvidia GPUs:

Dense models (pp512 t/s)

GPU	Model	Baseline	f16 B	Change
RTX 3090	Llama 3.1 8B Q3_K_S	2873.1	2883.5	+0.4%
	Llama 3.1 8B Q4_K_S	3600.7	3747.8	+4.1%
	Llama 3.1 8B Q8_0	3370.7	3492.4	+3.6%
GB10	Llama 3.1 8B Q3_K_S	1809.3	1872.2	+3.5%
	Llama 3.1 8B Q6_K	1624.0	1650.2	+1.6%
R9700	Llama 3 8B Q3_K_S	3358.0	3392.4	+1.0%
	Llama 3 8B Q4_K_S	4432.6	4520.4	+2.0%
	Qwen3 8B Q8_0	4515.5	4686.7	+3.8%

tg128 unaffected

MoE models (pp512 and tg128 t/s)

GPU	Model	Test	Baseline	f16 B	Change
RTX 3090	Qwen3.6 35B-A3B Q2_K	pp512	2473.1	2298.3	-7.1%
		tg128	146.0	147.4	+0.9%
GB10	Qwen3.6 35B-A3B Q4_K_M	pp512	1639.1	1439.4	-12.2%
		tg128	64.1	64.2	+0.2%
R9700	Qwen3.6 35B-A3B Q4_0	pp512	3845.9	3899.0	+1.4%
		tg128	133.7	124.9	-6.6%

For MoE there's probably a missing size threshold where the savings are too small for the overhead, but in general it can help on other vendors too.

fish-jiang · 2026-08-31T08:22:03Z

this will also be on windows only for intel arl_h right? since coopmat is not enabled on linux due to driver issue?

Yes — it just follows from coopmat_support.

fish-jiang · 2026-08-31T08:31:36Z

@0cc4m, Thanks for testing this on AMD and NVIDIA GPUs!
How about this commit: ce1a1b7? Dense mode has f16 B enabled for all vendors; for MoE it's currently Intel-only. A size threshold for MoE seems hard to pin down and may be device-dependent.

I ran a quick local test on an AMD iGPU (890M) — both dense and MoE show gains. Small sample, so take with a grain of salt (iGPU, more prone to power/thermal noise than a discrete GPU):

Before:
C:\Users\AE\Desktop\fish\llama.cpp\build_base\bin\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\AE\Downloads\models\Qwen3-8B-Q8_0.gguf,C:\Users\AE\Downloads\models\Meta-Llama-3.1-8B-Instruct-Q4_K_S.gguf,C:\Users\AE\Downloads\models\Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) 890M Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	fa	test	t/s
qwen3 8B Q8_0	8.11 GiB	8.19 B	Vulkan	99	0	pp512	101.07 ± 6.11
qwen3 8B Q8_0	8.11 GiB	8.19 B	Vulkan	99	1	pp512	101.19 ± 4.09
llama 8B Q4_K - Small	4.36 GiB	8.03 B	Vulkan	99	0	pp512	155.55 ± 40.04
llama 8B Q4_K - Small	4.36 GiB	8.03 B	Vulkan	99	1	pp512	139.45 ± 7.00
qwen35moe 35B.A3B Q2_K - Medium	11.44 GiB	34.66 B	Vulkan	99	0	pp512	159.33 ± 20.42
qwen35moe 35B.A3B Q2_K - Medium	11.44 GiB	34.66 B	Vulkan	99	1	pp512	174.51 ± 8.79

After:

model	size	params	backend	ngl	fa	test	t/s
qwen3 8B Q8_0	8.11 GiB	8.19 B	Vulkan	99	0	pp512	110.03 ± 6.58
qwen3 8B Q8_0	8.11 GiB	8.19 B	Vulkan	99	1	pp512	124.06 ± 10.86
llama 8B Q4_K - Small	4.36 GiB	8.03 B	Vulkan	99	0	pp512	172.26 ± 3.15
llama 8B Q4_K - Small	4.36 GiB	8.03 B	Vulkan	99	1	pp512	192.63 ± 12.07
qwen35moe 35B.A3B Q2_K - Medium	11.44 GiB	34.66 B	Vulkan	99	0	pp512	192.67 ± 12.93
qwen35moe 35B.A3B Q2_K - Medium	11.44 GiB	34.66 B	Vulkan	99	1	pp512	190.72 ± 10.43

BTW, this PR theoretically doesn't touch decode. So the R9700 tg128 -6.6% seems unlikely to come from this change — could it be a power/frequency variance in that run?

fish-jiang · 2026-09-03T09:25:17Z

@0cc4m Do you have any further guidance or comments to help move the PR forward?

… Intel coopmat1

…oE stays Intel-only)

… to Intel, fix mmp selection

…5773) * vulkan: use spec constant for mul mat type_a vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types * remove LUT quants from unified shader * clean up * restore coopmat2 q4_k/q5_k optimization * split out q4_k/q5_k cm2 shader to fix Ampere regression * revert iq shmem table renames * simplify cm2 code with single uint8_t buffer * fix fp4 extension use switch being overwritten by generic shader * clean up * adapt TQ1_0 changes * adapt #27471 f16 Intel tuning changes

… Intel coopmat1 (ggml-org#27471) * vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 * simplify mmp selection in mul_mat_id per review comment * vulkan: enable f16 B-type pipeline for dense matmul on all vendors (MoE stays Intel-only) * vulkan: add missing ocp_fp4 branches, restrict required_subgroup_size to Intel, fix mmp selection * refine mmp selection in mul_mat_id * add f16B pipeline selection just like q8_1 * update f16B pipeline selection in dense function

…ml-org#25773) * vulkan: use spec constant for mul mat type_a vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types * remove LUT quants from unified shader * clean up * restore coopmat2 q4_k/q5_k optimization * split out q4_k/q5_k cm2 shader to fix Ampere regression * revert iq shmem table renames * simplify cm2 code with single uint8_t buffer * fix fp4 extension use switch being overwritten by generic shader * clean up * adapt TQ1_0 changes * adapt ggml-org#27471 f16 Intel tuning changes

… Intel coopmat1 (ggml-org#27471) * vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 * simplify mmp selection in mul_mat_id per review comment * vulkan: enable f16 B-type pipeline for dense matmul on all vendors (MoE stays Intel-only) * vulkan: add missing ocp_fp4 branches, restrict required_subgroup_size to Intel, fix mmp selection * refine mmp selection in mul_mat_id * add f16B pipeline selection just like q8_1 * update f16B pipeline selection in dense function

…ml-org#25773) * vulkan: use spec constant for mul mat type_a vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types * remove LUT quants from unified shader * clean up * restore coopmat2 q4_k/q5_k optimization * split out q4_k/q5_k cm2 shader to fix Ampere regression * revert iq shmem table renames * simplify cm2 code with single uint8_t buffer * fix fp4 extension use switch being overwritten by generic shader * clean up * adapt TQ1_0 changes * adapt ggml-org#27471 f16 Intel tuning changes

… Intel coopmat1 (ggml-org#27471) * vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 * simplify mmp selection in mul_mat_id per review comment * vulkan: enable f16 B-type pipeline for dense matmul on all vendors (MoE stays Intel-only) * vulkan: add missing ocp_fp4 branches, restrict required_subgroup_size to Intel, fix mmp selection * refine mmp selection in mul_mat_id * add f16B pipeline selection just like q8_1 * update f16B pipeline selection in dense function

…ml-org#25773) * vulkan: use spec constant for mul mat type_a vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types * remove LUT quants from unified shader * clean up * restore coopmat2 q4_k/q5_k optimization * split out q4_k/q5_k cm2 shader to fix Ampere regression * revert iq shmem table renames * simplify cm2 code with single uint8_t buffer * fix fp4 extension use switch being overwritten by generic shader * clean up * adapt TQ1_0 changes * adapt ggml-org#27471 f16 Intel tuning changes

… Intel coopmat1 (ggml-org#27471) * vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 * simplify mmp selection in mul_mat_id per review comment * vulkan: enable f16 B-type pipeline for dense matmul on all vendors (MoE stays Intel-only) * vulkan: add missing ocp_fp4 branches, restrict required_subgroup_size to Intel, fix mmp selection * refine mmp selection in mul_mat_id * add f16B pipeline selection just like q8_1 * update f16B pipeline selection in dense function

…ml-org#25773) * vulkan: use spec constant for mul mat type_a vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types * remove LUT quants from unified shader * clean up * restore coopmat2 q4_k/q5_k optimization * split out q4_k/q5_k cm2 shader to fix Ampere regression * revert iq shmem table renames * simplify cm2 code with single uint8_t buffer * fix fp4 extension use switch being overwritten by generic shader * clean up * adapt TQ1_0 changes * adapt ggml-org#27471 f16 Intel tuning changes

fish-jiang requested a review from a team as a code owner August 21, 2026 06:38

fish-jiang marked this pull request as draft August 21, 2026 06:38

github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 21, 2026

jeffbolznv reviewed Aug 21, 2026

View reviewed changes

github-actions Bot mentioned this pull request Aug 24, 2026

FreeToken ROCm Watch — notification channel Castoff995/FreeToken-rocm-test#1

Draft

fish-jiang force-pushed the intel/xe-warptile-f16 branch 2 times, most recently from b44211b to ce1a1b7 Compare August 31, 2026 08:13

fish-jiang closed this Aug 31, 2026

fish-jiang reopened this Aug 31, 2026

fish-jiang marked this pull request as ready for review September 2, 2026 01:50

0cc4m reviewed Sep 4, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

0cc4m reviewed Sep 4, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

0cc4m reviewed Sep 8, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

fish-jiang added 5 commits September 8, 2026 15:31

vulkan: add f16 B-type matmul pipelines and warp tile size tuning for…

0ddb19e

… Intel coopmat1

simplify mmp selection in mul_mat_id per review comment

4255022

vulkan: enable f16 B-type pipeline for dense matmul on all vendors (M…

a039a1d

…oE stays Intel-only)

vulkan: add missing ocp_fp4 branches, restrict required_subgroup_size…

ba1b943

… to Intel, fix mmp selection

refine mmp selection in mul_mat_id

28b7bb8

fish-jiang force-pushed the intel/xe-warptile-f16 branch from d7a1d16 to 28b7bb8 Compare September 8, 2026 07:44

add f16B pipeline selection just like q8_1

1b36eb9

0cc4m reviewed Sep 8, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

update f16B pipeline selection in dense function

a53ae45

0cc4m approved these changes Sep 8, 2026

View reviewed changes

0cc4m added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 9, 2026

ggerganov merged commit 1945e09 into ggml-org:master Sep 9, 2026
26 of 30 checks passed

0cc4m added a commit that referenced this pull request Sep 9, 2026

adapt #27471 f16 Intel tuning changes

b1b70dd

snailium mentioned this pull request Sep 10, 2026

[Vulkan] Deep-context decode collapses ~8x on Intel Arc Pro B70 (Xe2); RDNA3 does not #28721

Open

ggerganov mentioned this pull request Sep 14, 2026

ggml : bump version to 0.24.0 ggml-org/ggml#1627

Merged

BrewTestBot mentioned this pull request Sep 14, 2026

ggml 0.24.0 Homebrew/homebrew-core#310890

Merged

1 task done

fish-jiang mentioned this pull request Sep 23, 2026

vulkan: GEMM/Group GEMM optimizations on Intel Xe (3/3, Xe-LPG Plus/Xe2/Xe3) #24407

Closed

Conversation

fish-jiang commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Performance (Windows OS)

Requirements

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

jeffbolznv Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

fish-jiang Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

jeffbolznv Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

fish-jiang Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

newbeelearn commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 26, 2026

Dense models (pp512 t/s)

MoE models (pp512 and tg128 t/s)

Uh oh!

fish-jiang commented Aug 31, 2026

Uh oh!

fish-jiang commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fish-jiang commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fish-jiang commented Aug 21, 2026 •

edited

Loading

fish-jiang commented Aug 31, 2026 •

edited

Loading