sycl: accelerate GLM MLA prefill with MKL flash attention by anantshri · Pull Request #29171 · ggml-org/llama.cpp

anantshri · 2026-09-20T07:40:26Z

Overview

route GLM-4.7 Flash's DKQ=576, DV=512, GQA-20 F16 MLA prompt shape through the existing SYCL MKL flash-attention pipeline
correctly handle V as a narrower, strided view of the K cache
keep the exception narrowly restricted to the validated GLM shape
add an exact backend test for the newly supported layout

Additional information

Problem

The normal SYCL MKL flash-attention gate requires matching K and V head dimensions and limits the head dimension to 512.

GLM-4.7 Flash uses:

Q head width: 576
K head width: 576
V head width: 512
Q heads: 20
KV heads: 1
K/V type: F16

Because DKQ != DV and DKQ > 512, GLM prefill falls back to the generic TILE kernel. That fallback scales poorly with context length and causes severe long-context prefill degradation.

The existing MKL pipeline already parameterizes the KQ and VKQ operations with separate dimensions, so it can support this shape once dispatch and V-cache layout handling are corrected.

Implementation

The new dispatch exception is deliberately narrow. It admits only:

Q = 576
K = 576
V = 512
GQA = 20
F16 K and V

All existing prompt, mask, bias, softcap, batch, graph, and cache restrictions remain in force. Other unequal K/V shapes continue to use their existing fallback paths.

GLM stores V as a 512-element sub-view of a 576-element K-cache row. Its logical row is 1,024 bytes, while its physical row stride remains 1,152 bytes.

The implementation therefore:

selects the strided F16 descriptor when a KV row has padding
accepts the unusual V stride only when V is a real view sharing K's row stride
aliases K and V dequantization buffers only when their logical dimensions match
keeps unrelated attention shapes on their existing paths

An exact backend test covers the 576/512, GQA-20, strided-V configuration.

Performance

Intel Arc Pro B70, IntelLLVM 2026.1, oneDNN enabled, F16 KV, -ngl 999 -b 4096 -ub 1024 -fa on, three repetitions.

This table compares clean master with this standalone patch only.

Prompt processing

Prompt	Master	Patched	Improvement
512	1,283.35 tok/s	1,284.97 tok/s	+0.13%
8,192	429.28 tok/s	1,293.63 tok/s	3.01x, +201.35%
64,000	67.071 tok/s	378.145 tok/s	5.64x, +463.80%

The improvement grows with context length because the patch removes GLM's quadratic TILE fallback and routes the supported MLA shape through MKL flash attention.

Context-loaded generation

Generation of 256 tokens after preloading the stated KV-cache depth:

Depth	Master	Patched	Change
512	45.029 tok/s	45.089 tok/s	+0.13%
8,192	32.596 tok/s	32.636 tok/s	+0.12%
64,000	11.766 tok/s	11.778 tok/s	+0.10%

Generation remains effectively unchanged, as expected for a prompt-processing dispatch change.

Validation

applies cleanly to current upstream master b23efaa2
Release IntelLLVM 2026.1 SYCL build succeeds with oneDNN, F16, graphs, and Level Zero support
exact GLM MLA backend case passes: 1/1
debug output confirms MKL dispatch with D=576, DV=512, 20 Q heads, and one KV head
complete SYCL0 FLASH_ATTN_EXT suite: 4,044/4,045 passed
the sole failure is the existing Q8_0 permuted-layout case, reproduced on the unmodified baseline
git diff --check passes

Scope

This is a standalone three-file change:

ggml/src/ggml-sycl/fattn.cpp
ggml/src/ggml-sycl/fattn-mkl.cpp
tests/test-backend-ops.cpp

It does not depend on the IQ3 optimization, the MMVQ decode fusions, PR #28918, PR #28931, or any other B70 campaign patch.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes

This specific PR and associated fixes are created using GPT 5.6 Sol

ggml-gh-bot · 2026-09-20T07:45:04Z

Hi @anantshri, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 4 open PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

arthw

It's good job!

GLM-4.7-Flash-Q4_K_M.gguf on B60:

Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
pp8192	0	681.32	671.02	-1.51%
pp8192	1	218.31	619.75	183.89%
tg128	0	40.77	41.00	0.56%
tg128	1	40.00	40.07	0.18%

Thank you!

ggerganov · 2026-09-23T05:20:26Z

+    // GLM-4.7 Flash: exact MLA shape that dispatches to MKL FA for prompt batches.
+    test_cases.emplace_back(new test_flash_attn_ext(576, 512, 1, {20, 1}, 1024, 32, true, false, 0, 0,
+        GGML_PREC_F32, GGML_TYPE_F16, GGML_TYPE_F16, {0, 1, 2, 3}, true, true));


Prefer not to add this shape because it is very slow on some Vulkan jobs: #28842

Thanks for the headsup. I will try to see what i can do about this.

I was experimenting on GLM only and i think i might have found more performance improvement point will circle back on this one in next 24 hours or so.

anantshri · 2026-09-23T19:37:13Z

rebased on latest master. gave me a chance to optimize this a bit further.
Removed the test case as suggested by @ggerganov
Added 1 more commit to get some better stats.

Measured on an Intel Arc Pro B70 with the dispatch from the previous
commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on:

pp8192 1583.9 -> 1657.1 tok/s (+4.6%)
pp64000 610.0 -> 684.5 tok/s (+12.2%)
pp131072 ~354 -> 402.1 tok/s (+13.5%)

@arthw @ggerganov for your review and consideration.

GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape because the normal MKL flash-attention gate requires matching K/V widths and caps the head dimension at 512, so prompt processing falls back to the substantially slower TILE kernel. Admit only the validated 576/576/512, GQA-20 F16 shape to the existing MKL pipeline. Keep all other mismatched K/V shapes on their current fallback paths. Handle GLM's V cache as a narrower strided view of K rows. Select the strided F16 descriptor when row stride is padded, and alias K/V dequantization buffers only when their logical widths match. Restrict the stride exception to a real V view sharing K's row stride. Add the exact 576/512, GQA-20 prompt-path backend test. On an Intel Arc Pro B70 at master e613ef2, pp8192 improves from 432.80 to 1292.29 tok/s (2.99x, +198.6%). tg256 remains unchanged within noise at 45.67 versus 45.65 tok/s. The exact MLA test passes and debug output confirms MKL dispatch.

Keep the QK GEMM output in F16 instead of F32. The online softmax still converts each score to F32 for its max, exponent, and sum, so the per-element math is unchanged apart from score rounding, and the F32 matrix was being written only to be consumed as F16 probabilities. The F32 score matrix is the largest flash-attention intermediate on this path; storing it as F16 halves its size and traffic. This builds on the coalesced softmax loads from 1aa2954, which read each score row cooperatively, so the smaller dtype pays off. Measured on an Intel Arc Pro B70 with the dispatch from the previous commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on: pp8192 1583.9 -> 1657.1 tok/s (+4.6%) pp64000 610.0 -> 684.5 tok/s (+12.2%) pp131072 ~354 -> 402.1 tok/s (+13.5%) tg256 at 8k context is unchanged (32.64), and the FLASH_ATTN_EXT suite shows no new failures. The exact GLM MLA backend cases pass against CPU. Adjust the ~354 baseline figure if you prefer citing only measured pairs (the 131k dispatch-only point came from the equivalent maintained build). Optionally add Assisted-by: <tool name> per the contribution guidelines since AI contributed to the change.

This reverts commit 265f974.

anantshri · 2026-09-29T12:55:51Z

rebased to upstream master.
removed the last change of f32-> f16 as there was a chance of overflow. (thanks @arthw for pointing. )

arthw · 2026-09-30T14:40:33Z

@anantshri

On Arc770, I didn't see the perf increase.
I think the original perf increase is due to the change from fp32 to fp16.
Without it, the current code can't bring more benefit.

Could you confirm it?

Here is my test result:

GGUF	Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
Qwen3.5-0.8B-MTP-Q4_K_M.gguf	pp512	0	5108.22	5071.71	-0.71%
Qwen3.5-0.8B-MTP-Q4_K_M.gguf	pp512	1	5218.08	5228.70	0.20%
Qwen3.5-0.8B-MTP-Q4_K_M.gguf	pp8192	0	5515.25	5515.98	0.01%
Qwen3.5-0.8B-MTP-Q4_K_M.gguf	pp8192	1	5914.71	5924.40	0.16%
Qwen3.5-0.8B-MTP-Q4_K_M.gguf	tg128	0	55.02	58.79	6.85%
Qwen3.5-0.8B-MTP-Q4_K_M.gguf	tg128	1	58.03	59.82	3.08%

GGUF	Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
gpt-oss-20b-mxfp4.gguf	pp512	0	787.78	787.78	0.00%
gpt-oss-20b-mxfp4.gguf	pp512	1	753.02	754.50	0.20%
gpt-oss-20b-mxfp4.gguf	pp8192	0	855.20	854.85	-0.04%
gpt-oss-20b-mxfp4.gguf	pp8192	1	675.67	675.54	-0.02%
gpt-oss-20b-mxfp4.gguf	tg128	0	20.38	20.42	0.20%
gpt-oss-20b-mxfp4.gguf	tg128	1	23.86	23.89	0.13%

Thank you!

anantshri · 2026-09-30T14:52:43Z

@arthw you will only see perf gain for GLM 4.7

route GLM-4.7 Flash's DKQ=576, DV=512, GQA-20 F16 MLA prompt shape through the existing SYCL MKL flash-attention pipeline

your initial test was for GLM 4.7 #29171 (review)

as prefil size increases the benefit is amplified.

arthw · 2026-10-02T16:02:01Z

@anantshri

On B60, I tested with the latest code.
No performance increasing.

Could you double check?

GGUF	Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
GLM-4.7-Flash-Q4_K_M.gguf	pp512	0	781.18	781.01	-0.02%
GLM-4.7-Flash-Q4_K_M.gguf	pp512	1	577.19	577.68	0.08%
GLM-4.7-Flash-Q4_K_M.gguf	tg128	0	40.90	40.88	-0.05%
GLM-4.7-Flash-Q4_K_M.gguf	tg128	1	39.97	39.95	-0.05%

anantshri · 2026-10-02T17:47:45Z

@arthw can you try for larger sizes

the performance increment is at the larger context as i showed here #29171

i am running a comparision but mainline is pathetically slow on 64k or more pp so its taking time will post the values once i get them

pp 512 main 1287.370738 [1288.78, 1285.97]
pp 512 patched 1289.715368 [1291.23, 1288.2]
pp 8192 main 432.382653 [433.634, 431.132]
pp 8192 patched 1585.877664 [1588.54, 1583.22]

edit more entries i will add here

pp32000 main 130.030881 [130.031]
pp32000 patched 936.351538 [936.352]
pp64000 main 66.921487 [66.9215]
pp64000 patched 610.239738 [610.24]

As you can see the gains become visible the movement you start going bigger on pp size.

Edit with final updates

Context tokens	Mainline pp	Patched pp	pp improvement	Mainline tg	Patched tg	tg change
512	1,287.37	1,289.72	+0.18%	45.5334	45.4962	−0.082%
8,192	432.38	1,585.88	+266.78% (3.67×)	32.8157	32.8176	+0.006%
32,000	130.03	936.35	+620.10% (7.20×)	18.5321	18.5321	<+0.001%
64,000	66.92	610.24	+811.87% (9.12×)	11.8176	11.8297	+0.102%
130,000	33.30	353.06	+960.27% (10.60×)	6.7838	6.7882	+0.065%

Would request to please check at higher sizes. you will find the improvements.

sorry for the delay in posting all the details but wanted to run everything and show exact improvements. at top end of the size spectrum you can see massive gains.

…cl-b70-combined-20261001

arthw

It's good job!

On B60:

GGUF	Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
GLM-4.7-Flash-Q4_K_M.gguf	pp8192	0	602.81	595.73	-1.17%
GLM-4.7-Flash-Q4_K_M.gguf	pp8192	1	204.50	602.64	194.69%
GLM-4.7-Flash-Q4_K_M.gguf	tg128	0	40.79	40.97	0.44%
GLM-4.7-Flash-Q4_K_M.gguf	tg128	1	39.95	40.09	0.35%

Thank you!

…9171) * sycl: accelerate GLM MLA prefill with MKL flash attention GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape because the normal MKL flash-attention gate requires matching K/V widths and caps the head dimension at 512, so prompt processing falls back to the substantially slower TILE kernel. Admit only the validated 576/576/512, GQA-20 F16 shape to the existing MKL pipeline. Keep all other mismatched K/V shapes on their current fallback paths. Handle GLM's V cache as a narrower strided view of K rows. Select the strided F16 descriptor when row stride is padded, and alias K/V dequantization buffers only when their logical widths match. Restrict the stride exception to a real V view sharing K's row stride. Add the exact 576/512, GQA-20 prompt-path backend test. On an Intel Arc Pro B70 at master e613ef2, pp8192 improves from 432.80 to 1292.29 tok/s (2.99x, +198.6%). tg256 remains unchanged within noise at 45.67 versus 45.65 tok/s. The exact MLA test passes and debug output confirms MKL dispatch. * sycl: store MKL flash attention scores in F16 Keep the QK GEMM output in F16 instead of F32. The online softmax still converts each score to F32 for its max, exponent, and sum, so the per-element math is unchanged apart from score rounding, and the F32 matrix was being written only to be consumed as F16 probabilities. The F32 score matrix is the largest flash-attention intermediate on this path; storing it as F16 halves its size and traffic. This builds on the coalesced softmax loads from 1aa2954, which read each score row cooperatively, so the smaller dtype pays off. Measured on an Intel Arc Pro B70 with the dispatch from the previous commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on: pp8192 1583.9 -> 1657.1 tok/s (+4.6%) pp64000 610.0 -> 684.5 tok/s (+12.2%) pp131072 ~354 -> 402.1 tok/s (+13.5%) tg256 at 8k context is unchanged (32.64), and the FLASH_ATTN_EXT suite shows no new failures. The exact GLM MLA backend cases pass against CPU. Adjust the ~354 baseline figure if you prefer citing only measured pairs (the 131k dispatch-only point came from the equivalent maintained build). Optionally add Assisted-by: <tool name> per the contribution guidelines since AI contributed to the change. * Revert "sycl: store MKL flash attention scores in F16" This reverts commit 265f974. (cherry picked from commit d2a79e6)

…9171) * sycl: accelerate GLM MLA prefill with MKL flash attention GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape because the normal MKL flash-attention gate requires matching K/V widths and caps the head dimension at 512, so prompt processing falls back to the substantially slower TILE kernel. Admit only the validated 576/576/512, GQA-20 F16 shape to the existing MKL pipeline. Keep all other mismatched K/V shapes on their current fallback paths. Handle GLM's V cache as a narrower strided view of K rows. Select the strided F16 descriptor when row stride is padded, and alias K/V dequantization buffers only when their logical widths match. Restrict the stride exception to a real V view sharing K's row stride. Add the exact 576/512, GQA-20 prompt-path backend test. On an Intel Arc Pro B70 at master e613ef2, pp8192 improves from 432.80 to 1292.29 tok/s (2.99x, +198.6%). tg256 remains unchanged within noise at 45.67 versus 45.65 tok/s. The exact MLA test passes and debug output confirms MKL dispatch. * sycl: store MKL flash attention scores in F16 Keep the QK GEMM output in F16 instead of F32. The online softmax still converts each score to F32 for its max, exponent, and sum, so the per-element math is unchanged apart from score rounding, and the F32 matrix was being written only to be consumed as F16 probabilities. The F32 score matrix is the largest flash-attention intermediate on this path; storing it as F16 halves its size and traffic. This builds on the coalesced softmax loads from 1aa2954, which read each score row cooperatively, so the smaller dtype pays off. Measured on an Intel Arc Pro B70 with the dispatch from the previous commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on: pp8192 1583.9 -> 1657.1 tok/s (+4.6%) pp64000 610.0 -> 684.5 tok/s (+12.2%) pp131072 ~354 -> 402.1 tok/s (+13.5%) tg256 at 8k context is unchanged (32.64), and the FLASH_ATTN_EXT suite shows no new failures. The exact GLM MLA backend cases pass against CPU. Adjust the ~354 baseline figure if you prefer citing only measured pairs (the 131k dispatch-only point came from the equivalent maintained build). Optionally add Assisted-by: <tool name> per the contribution guidelines since AI contributed to the change. * Revert "sycl: store MKL flash attention scores in F16" This reverts commit 265f974.

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Sep 20, 2026

anantshri mentioned this pull request Sep 20, 2026

Feature Request: Performance Improvements on SYCL #28990

Open

4 tasks done

anantshri marked this pull request as ready for review September 21, 2026 08:36

anantshri requested review from a team and ggerganov as code owners September 21, 2026 08:36

arthw approved these changes Sep 21, 2026

View reviewed changes

arthw added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 22, 2026

ggerganov reviewed Sep 23, 2026

View reviewed changes

ggerganov removed the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 23, 2026

anantshri force-pushed the sycl-glm-mla-mkl-fa branch from 3942532 to 265f974 Compare September 23, 2026 19:35

arthw reviewed Sep 29, 2026

View reviewed changes

Comment thread ggml/src/ggml-sycl/fattn-mkl.cpp Outdated

anantshri added 3 commits September 29, 2026 17:49

Revert "sycl: store MKL flash attention scores in F16"

88e6449

This reverts commit 265f974.

anantshri force-pushed the sycl-glm-mla-mkl-fa branch from 265f974 to 88e6449 Compare September 29, 2026 12:53

arthw added merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. and removed merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. labels Oct 2, 2026

anantshri added a commit to anantshri/llama.cpp that referenced this pull request Oct 4, 2026

Merge updated PR ggml-org#29171 head (88e6449, review fixups) into sy…

e1a4b8d

…cl-b70-combined-20261001

arthw approved these changes Oct 6, 2026

View reviewed changes

arthw added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Oct 6, 2026

ggerganov merged commit d2a79e6 into ggml-org:master Oct 7, 2026
18 checks passed

ggerganov mentioned this pull request Oct 15, 2026

ggml : bump version to 0.27.0 ggml-org/ggml#1659

Merged

BrewTestBot mentioned this pull request Oct 15, 2026

ggml 0.27.0 Homebrew/homebrew-core#318733

Merged

1 task done

Navigation Menu

Repository navigation

sycl: accelerate GLM MLA prefill with MKL flash attention - #29171

sycl: accelerate GLM MLA prefill with MKL flash attention#29171
ggerganov merged 3 commits into
ggml-org:masterfrom
anantshri:sycl-glm-mla-mkl-fa

anantshri commented Sep 20, 2026

Uh oh!

ggml-gh-bot Bot commented Sep 20, 2026

Uh oh!

arthw left a comment

Uh oh!

ggerganov Sep 23, 2026

Uh oh!

anantshri Sep 23, 2026

Uh oh!

anantshri commented Sep 23, 2026

Uh oh!

Uh oh!

anantshri commented Sep 29, 2026

Uh oh!

arthw commented Sep 30, 2026

Uh oh!

anantshri commented Sep 30, 2026

Uh oh!

arthw commented Oct 2, 2026

Uh oh!

anantshri commented Oct 2, 2026 •

edited

Loading

Uh oh!

arthw left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

anantshri commented Sep 20, 2026

Overview

Additional information

Problem

Implementation

Performance

Prompt processing

Context-loaded generation

Validation

Scope

Requirements

Uh oh!

ggml-gh-bot Bot commented Sep 20, 2026

Uh oh!

arthw left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

anantshri Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

anantshri commented Sep 23, 2026

Uh oh!

Uh oh!

anantshri commented Sep 29, 2026

Uh oh!

arthw commented Sep 30, 2026

Uh oh!

anantshri commented Sep 30, 2026

Uh oh!

arthw commented Oct 2, 2026

Uh oh!

anantshri commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arthw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anantshri commented Oct 2, 2026 •

edited

Loading