Skip to content

CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 - #24546

Merged
pwilkin merged 7 commits into
ggml-org:masterfrom
ravel7524:mmqx-rdna3-routed-moe-tiling
Sep 7, 2026
Merged

CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3#24546
pwilkin merged 7 commits into
ggml-org:masterfrom
ravel7524:mmqx-rdna3-routed-moe-tiling

Conversation

@ravel7524

@ravel7524 ravel7524 commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

This PR improves routed MoE prefill performance on RDNA3 by changing how llama.cpp chooses MMQ N-tile sizes. Instead of using the worst case expert width, it estimates the typical routed expert width and uses that when it is smaller than the RDNA3 maximum tile width. The launch grid still uses the original worst case width, so output coverage and correctness are preserved. The change is limited to host side tile selection in mul_mat_q_case and does not modify structs, call sites, launch parameters, or device kernels. Benchmarks on a Radeon Pro W7800/gfx1100 with ROCm 7.2.3 show large prefill gains in many routed MoE cases, while decode and non RDNA3 architectures are mostly unchanged.

If maintainers or other contributors have access to NVIDIA, CDNA, RDNA2, or RDNA4 hardware, the natural follow up would be to test the same routed width picker under the corresponding MMQ/launch policy and enable it per architecture where it is beneficial.

Benchmark Summary

Hardware/software:

  • AMD Radeon Pro W7800, gfx1100 / RDNA3
  • ROCm 7.2.3
  • HIP backend, -ngl 99
  • GGML_CUDA_FA_ALL_QUANTS=ON
  • -fa 1 -ctk q8_0 -ctv f16 -sm none -t 4 --poll 0 -r 5

The improvement depends on the physical ubatch/chunk width that reaches routed MoE MMQ, not total context length:

win when:          ncols_typical < mmq_x_max
crossover_ubatch: 128 * n_experts / n_expert_used

So long prompts can still benefit when they are processed in smaller ubatch chunks.

Benchmark sweep
model / quant experts ubatch pp512 pp1024 pp2048 pp4096 pp8192 tg128
Qwen3.6-35B-A3B Q6_K 256 2048 +48.46% +22.15% +7.57% n/a n/a -0.67%
Qwen3-30B-A3B Q6_K 128 512 +35.50% +33.79% +32.15% n/a n/a -1.11%
Qwen3-30B-A3B Q6_K 128 2048 +36.62% +13.90% -0.75% n/a n/a -0.89%
Qwen3-30B-A3B Q4_K_M 128 512 +33.68% +31.54% +29.81% +27.70% +23.86% +0.35%
Qwen3-30B-A3B Q4_K_M 128 2048 +33.77% +14.00% -0.08% -0.06% -0.01% -0.20%
Qwen3.6-35B-A3B Q3_K_M 256 512 +23.48% +22.37% +22.73% +22.91% +21.18% -0.08%
Qwen3.6-35B-A3B Q3_K_M 256 2048 +22.93% +6.99% +4.02% +3.92% +3.69% +0.15%
Gemma-4-26B-A4B Q4_K_M 128 512 +29.53% +28.52% +26.65% +25.46% +23.19% +0.04%
Gemma-4-26B-A4B Q4_K_M 128 2048 +29.87% +13.16% +0.03% +0.05% +0.04% -0.09%
Gemma-4-26B-A4B Q6_K 128 512 +25.05% +24.05% +23.21% +22.51% +20.59% +0.08%
Gemma-4-26B-A4B Q6_K 128 2048 +25.53% +11.75% +0.11% -0.04% -0.00% +0.50%

The 128 expert models show the expected crossover behavior at ubatch 2048: pp2048 and larger are neutral because the typical width has reached mmq_x_max = 128. The 256-expert Qwen3.6 shape remains below crossover and still improves through pp8192.

Requirements

  • Yes, I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, AI was used to help review the change and summarize benchmark results. I reviewed the final diff and can explain all code changes.

@github-actions github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Jun 12, 2026
@ravel7524
ravel7524 marked this pull request as ready for review June 12, 2026 21:04
@ravel7524
ravel7524 requested a review from a team as a code owner June 12, 2026 21:04
@ravel7524

ravel7524 commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

Here is an additional sweep extending the prior pp8192 table upward. Values are MMQX vs base throughput deltas. tg128 is included as the decode control.

Hardware/software: AMD Radeon Pro W7800 (gfx1100), ROCm 7.2.3, HIP backend, -ngl 99, GGML_CUDA_FA_ALL_QUANTS=ON, -fa 1 -ctk q8_0 -ctv f16 -sm none -t 4 --poll 0.

model / quant experts ubatch pp8192 pp16384 pp32768 pp65536 tg128 largest clean
Qwen3-30B-A3B Q4_K_M 128 512 1661.44 -> 2049.97 (+23.39%) 1349.44 -> 1599.57 (+18.54%) 990.08 -> 1118.48 (+12.97%) 633.79 -> 682.82 (+7.74%) 133.61 -> 134.08 (+0.35%) pp65536
Qwen3-30B-A3B Q4_K_M 128 1024 2043.18 -> 2243.77 (+9.82%) 1602.36 -> 1726.49 (+7.75%) 1121.08 -> 1179.00 (+5.17%) 691.16 -> 711.71 (+2.97%) 130.59 -> 131.20 (+0.46%) pp65536
Qwen3-30B-A3B Q4_K_M 128 2048 2265.11 -> 2259.45 (-0.25%) 1730.70 -> 1731.02 (+0.02%) 1178.53 -> 1177.80 (-0.06%) 718.44 -> 718.40 (-0.00%) 130.73 -> 130.73 (+0.00%) pp65536
Qwen3-30B-A3B Q4_K_M 128 4096 2312.65 -> 2308.85 (-0.16%) 1758.76 -> 1757.65 (-0.06%) 1192.18 -> 1192.16 (-0.00%) 720.98 -> 720.85 (-0.02%) 129.29 -> 129.89 (+0.46%) pp65536
Qwen3-30B-A3B Q4_K_M 128 8192 2087.19 -> 2150.76 (+3.16%) 1675.31 -> 1674.92 (-0.02%) 1154.52 -> 1154.02 (-0.04%) 705.68 -> 705.59 (-0.01%) 128.90 -> 129.74 (+0.65%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 512 1849.80 -> 2279.65 (+23.24%) 1662.29 -> 2004.19 (+20.57%) 1401.96 -> 1623.81 (+15.83%) 1054.25 -> 1171.73 (+11.14%) 85.08 -> 85.02 (-0.06%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 1024 2278.15 -> 2502.82 (+9.86%) 1990.69 -> 2164.03 (+8.71%) 1617.03 -> 1722.56 (+6.53%) 1169.98 -> 1224.99 (+4.70%) 78.59 -> 84.29 (+7.84%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 2048 2474.42 -> 2467.24 (-0.29%) 2124.07 -> 2124.74 (+0.03%) 1702.03 -> 1702.09 (+0.00%) 1213.12 -> 1211.99 (-0.09%) 78.47 -> 84.25 (+7.98%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 4096 2448.56 -> 2442.26 (-0.26%) 2070.54 -> 2069.72 (-0.04%) 1668.67 -> 1668.29 (-0.02%) 1193.49 -> 1195.54 (+0.17%) 78.54 -> 84.19 (+7.79%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 8192 2122.56 -> 2118.94 (-0.17%) 1854.58 -> 1854.11 (-0.03%) 1510.94 -> 1511.74 (+0.05%) 1110.49 -> 1108.28 (-0.20%) 77.39 -> 84.41 (+9.99%) pp65536
Gemma-4-26B-A4B Q6_K 128 512 1605.25 -> 1935.92 (+20.60%) 1460.33 -> 1733.39 (+18.70%) 1267.12 -> 1444.12 (+13.97%) 983.37 -> 1086.60 (+10.50%) 88.47 -> 88.33 (-0.15%) pp65536
Gemma-4-26B-A4B Q6_K 128 1024 2046.42 -> 2238.64 (+9.39%) 1811.34 -> 1966.32 (+8.56%) 1503.42 -> 1603.36 (+6.65%) 1117.89 -> 1171.41 (+4.79%) 86.96 -> 87.59 (+0.73%) pp65536
Gemma-4-26B-A4B Q6_K 128 2048 2283.96 -> 2275.69 (-0.36%) 1981.77 -> 1982.07 (+0.01%) 1617.79 -> 1616.75 (-0.06%) 1180.25 -> 1180.33 (+0.01%) 87.59 -> 87.60 (+0.01%) pp65536
Gemma-4-26B-A4B Q6_K 128 4096 2301.54 -> 2293.49 (-0.35%) 1963.89 -> 1961.98 (-0.10%) 1608.37 -> 1608.08 (-0.02%) 1175.85 -> 1175.53 (-0.03%) 87.41 -> 87.65 (+0.27%) pp65536
Gemma-4-26B-A4B Q6_K 128 8192 2028.64 -> 2020.28 (-0.41%) 1780.78 -> 1784.42 (+0.21%) 1470.22 -> 1468.89 (-0.09%) 1089.65 -> 1088.48 (-0.11%) 87.63 -> 87.72 (+0.10%) pp65536

Across the tested 128 expert models, the gain persists at long context for smaller ubatches, and the larger ubatches are effectively neutral rather than regressing. I can do additional long context benchmarking for 256 expert models

@github-actions github-actions Bot added the CUDA Related to the CUDA backend label Jun 19, 2026
@ravel7524

ravel7524 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

@jiachengjason would you be able to benchmark this PR on your RDNA4 GPU when you have time? Since you tested #23685 on RDNA4, having the same kind of validation here would be very helpful.

If you know anyone who could also test on RDNA2 or CDNA, that would be great as well. No pressure.

@ravel7524

ravel7524 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@IMbackK when you have time, could you sanity check whether this RDNA3 routed MoE MMQ selector direction looks reasonable? The PR is scoped to gfx1100/RDNA3 and uses measured W7800 results; I mainly want to know if the shape is acceptable or if you would prefer a different architecture gating/config style.

@GZGavinZhao

GZGavinZhao commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Do you expect this to work well on RDNA3.5 / Strix Halo as well? Happy to test!

@ravel7524

ravel7524 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Yes, the PR’s RDNA3 predicate also includes RDNA3.5 / Strix Halo, so the routed-width selector should be active there. I expect the same general direction, although the performance magnitude may differ from gfx1100. Testing would be very helpful, ideally the same base vs PR prefill and decode comparison used in the PR.

@IMbackK IMbackK self-assigned this Jul 7, 2026
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Jul 17, 2026
@justinappler @claude
strix-halo: test routed-MoE ncols picker vs static J=48 (PR ggml-org#…
b47bb31 
…24546)

Upstream PR ggml-org#24546 (ravel7524, open) sizes routed-MoE MMQ N-tiles from the
typical expert width instead of the worst case. That is the same problem the
static J_max=48 cap from Findings ggml-org#8/ggml-org#9 solves, approached from the other end:
the cap clamps the search ceiling, the picker fixes the search objective. This
commit swaps the cap for the picker so the two can be A/B'd on gfx1151.

The PR does not apply as written - it patches mmq_x/mmq_x_max/
mmq_get_granularity_host in mul_mat_q_case, all deleted by PR ggml-org#24127, the same
refactor that forced the Finding ggml-org#5 -> ggml-org#9 re-port. Re-authored onto
mul_mat_q_switch_J's config-table search and scoped to RDNA3.5: gfx1151 is the
only arch we can measure, and narrowing the predicate keeps the A/B
single-variable. Upstream's predicate already covers RDNA3.5 via
GGML_CUDA_CC_IS_RDNA3, and the author confirmed on 2026-07-04 that Strix Halo
should benefit, but nobody has ever tested it there.

On the routed-MoE path ncols_max is ne12 - the worst case of one expert
receiving every token - so the search never reaches ntiles==1 and simply runs
to whatever ceiling it is given. The picker instead tiles from
ncols_dst/nchannels_x, the width a typical expert covers. launch_mul_mat_q
still builds the grid from ncols_max, so worst-case coverage is unchanged.

  - mmq.cuh: replace the RDNA3.5 MoE J_max=48 clamp with the typical-width
    picker; J_max is now a uniform 128 and the search terminates on its own.
  - mmq-table-check.cpp: pick_J mirrors the selection loop, so update it in
    lockstep or the check silently becomes a rubber stamp. Adds the routed-MoE
    arg shape from mmq.cu and a crossover case.

For Qwen 3.6 35B-A3B (256 experts, 8 active) at the production ub=2048 the
typical width is 2048*8/256 = 64, so this is a 48 -> 64 change at our operating
point - narrow enough that it may land in noise. Verified host-side without
ROCm: MoE J=64, dense J=128, no aborts across all 21 types.

Above ub=4096 the typical width reaches 128, the picker disengages and the
search runs to J=128 where the cap held 48. Findings ggml-org#5/ggml-org#8 only ever measured
the typical=64 regime, so that is untested on this chip; production ub=2048
cannot reach it. See strix-halo/mmq-moe-ncols-picker.md for the bench plan and
the keep/revert criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Jul 17, 2026
@justinappler @claude
strix-halo: revert routed-MoE ncols picker, flat vs static J=48
4781fb9 
Benched b47bb31 against 05e837f on gfx1151 (ROCm 7.14.0, canonical Qwen 3.6
35B-A3B Q4_K_XL matrix). The picker is not distinguishable from the static cap
at the production operating point:

  pp512  @ d=0      1428.13 -> 1388.93  (-2.7%)
  pp512  @ d=2048   1299.39 -> 1316.26  (+1.3%)
  pp512  @ d=8192   1135.42 -> 1142.23  (+0.6%)
  pp512  @ d=16384   971.25 ->  977.36  (+0.6%)

Correctness was clean first (790/790 MUL_MAT_ID, 1134/1134 MUL_MAT), so the
grid does still cover ncols_max while the tile is sized from the typical width.

The tg128 control fell ~1.6% at every depth. This change cannot move tg -
decode goes through MMVQ where ncols_dst==1, so the picker never runs - which
makes that a session drift between two builds measured on different days, the
same contamination Finding ggml-org#9 hit. The conclusion survives it either way: raw,
pp is -2.7% to +1.3%; calibrated against the drift, -1.2% to +2.9%. Every depth
is inside the host's ~2% noise floor, so the effect is smaller than this rig can
resolve, and the d=0 -2.7% is no more real than the d=2048 +1.3%.

So: at ub=2048, J=48 and J=64 are the same speed. The doc predicted this before
the numbers existed ("narrow enough that it may well land in noise"), which is
the only reason that reading is credible rather than post-hoc. Reverting to the
static cap because it is the validated status quo, not because it won.

Worth reporting on ggml-org#24546: the picker is neutral on RDNA3.5,
on the exact model and ubatch its sweep claims +7.57% for on gfx1100. The PR is
gated on GGML_CUDA_CC_IS_RDNA3, which covers both. gfx1151's table is I=64/
nthreads=128 vs gfx1100's I=128/nthreads=256 - our tiles are already half-width,
so there is much less worst-case over-sizing left to recover. That is the data
ravel7524 asked for on 2026-07-03.

Doc kept with the numbers and the untested ub>=4096 crossover regime recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Jul 22, 2026
@justinappler @claude
strix-halo: test routed-MoE ncols picker vs static J=48 (PR ggml-org#…
cd35b46 
…24546)

Upstream PR ggml-org#24546 (ravel7524, open) sizes routed-MoE MMQ N-tiles from the
typical expert width instead of the worst case. That is the same problem the
static J_max=48 cap from Findings ggml-org#8/ggml-org#9 solves, approached from the other end:
the cap clamps the search ceiling, the picker fixes the search objective. This
commit swaps the cap for the picker so the two can be A/B'd on gfx1151.

The PR does not apply as written - it patches mmq_x/mmq_x_max/
mmq_get_granularity_host in mul_mat_q_case, all deleted by PR ggml-org#24127, the same
refactor that forced the Finding ggml-org#5 -> ggml-org#9 re-port. Re-authored onto
mul_mat_q_switch_J's config-table search and scoped to RDNA3.5: gfx1151 is the
only arch we can measure, and narrowing the predicate keeps the A/B
single-variable. Upstream's predicate already covers RDNA3.5 via
GGML_CUDA_CC_IS_RDNA3, and the author confirmed on 2026-07-04 that Strix Halo
should benefit, but nobody has ever tested it there.

On the routed-MoE path ncols_max is ne12 - the worst case of one expert
receiving every token - so the search never reaches ntiles==1 and simply runs
to whatever ceiling it is given. The picker instead tiles from
ncols_dst/nchannels_x, the width a typical expert covers. launch_mul_mat_q
still builds the grid from ncols_max, so worst-case coverage is unchanged.

  - mmq.cuh: replace the RDNA3.5 MoE J_max=48 clamp with the typical-width
    picker; J_max is now a uniform 128 and the search terminates on its own.
  - mmq-table-check.cpp: pick_J mirrors the selection loop, so update it in
    lockstep or the check silently becomes a rubber stamp. Adds the routed-MoE
    arg shape from mmq.cu and a crossover case.

For Qwen 3.6 35B-A3B (256 experts, 8 active) at the production ub=2048 the
typical width is 2048*8/256 = 64, so this is a 48 -> 64 change at our operating
point - narrow enough that it may land in noise. Verified host-side without
ROCm: MoE J=64, dense J=128, no aborts across all 21 types.

Above ub=4096 the typical width reaches 128, the picker disengages and the
search runs to J=128 where the cap held 48. Findings ggml-org#5/ggml-org#8 only ever measured
the typical=64 regime, so that is untested on this chip; production ub=2048
cannot reach it. See strix-halo/mmq-moe-ncols-picker.md for the bench plan and
the keep/revert criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Jul 22, 2026
@justinappler @claude
strix-halo: revert routed-MoE ncols picker, flat vs static J=48
7ce0188 
Benched b47bb31 against 05e837f on gfx1151 (ROCm 7.14.0, canonical Qwen 3.6
35B-A3B Q4_K_XL matrix). The picker is not distinguishable from the static cap
at the production operating point:

  pp512  @ d=0      1428.13 -> 1388.93  (-2.7%)
  pp512  @ d=2048   1299.39 -> 1316.26  (+1.3%)
  pp512  @ d=8192   1135.42 -> 1142.23  (+0.6%)
  pp512  @ d=16384   971.25 ->  977.36  (+0.6%)

Correctness was clean first (790/790 MUL_MAT_ID, 1134/1134 MUL_MAT), so the
grid does still cover ncols_max while the tile is sized from the typical width.

The tg128 control fell ~1.6% at every depth. This change cannot move tg -
decode goes through MMVQ where ncols_dst==1, so the picker never runs - which
makes that a session drift between two builds measured on different days, the
same contamination Finding ggml-org#9 hit. The conclusion survives it either way: raw,
pp is -2.7% to +1.3%; calibrated against the drift, -1.2% to +2.9%. Every depth
is inside the host's ~2% noise floor, so the effect is smaller than this rig can
resolve, and the d=0 -2.7% is no more real than the d=2048 +1.3%.

So: at ub=2048, J=48 and J=64 are the same speed. The doc predicted this before
the numbers existed ("narrow enough that it may well land in noise"), which is
the only reason that reading is credible rather than post-hoc. Reverting to the
static cap because it is the validated status quo, not because it won.

Worth reporting on ggml-org#24546: the picker is neutral on RDNA3.5,
on the exact model and ubatch its sweep claims +7.57% for on gfx1100. The PR is
gated on GGML_CUDA_CC_IS_RDNA3, which covers both. gfx1151's table is I=64/
nthreads=128 vs gfx1100's I=128/nthreads=256 - our tiles are already half-width,
so there is much less worst-case over-sizing left to recover. That is the data
ravel7524 asked for on 2026-07-03.

Doc kept with the numbers and the untested ub>=4096 crossover regime recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Aug 3, 2026
@justinappler @claude
strix-halo: test routed-MoE ncols picker vs static J=48 (PR ggml-org#…
d12f0e2 
…24546)

Upstream PR ggml-org#24546 (ravel7524, open) sizes routed-MoE MMQ N-tiles from the
typical expert width instead of the worst case. That is the same problem the
static J_max=48 cap from Findings ggml-org#8/ggml-org#9 solves, approached from the other end:
the cap clamps the search ceiling, the picker fixes the search objective. This
commit swaps the cap for the picker so the two can be A/B'd on gfx1151.

The PR does not apply as written - it patches mmq_x/mmq_x_max/
mmq_get_granularity_host in mul_mat_q_case, all deleted by PR ggml-org#24127, the same
refactor that forced the Finding ggml-org#5 -> ggml-org#9 re-port. Re-authored onto
mul_mat_q_switch_J's config-table search and scoped to RDNA3.5: gfx1151 is the
only arch we can measure, and narrowing the predicate keeps the A/B
single-variable. Upstream's predicate already covers RDNA3.5 via
GGML_CUDA_CC_IS_RDNA3, and the author confirmed on 2026-07-04 that Strix Halo
should benefit, but nobody has ever tested it there.

On the routed-MoE path ncols_max is ne12 - the worst case of one expert
receiving every token - so the search never reaches ntiles==1 and simply runs
to whatever ceiling it is given. The picker instead tiles from
ncols_dst/nchannels_x, the width a typical expert covers. launch_mul_mat_q
still builds the grid from ncols_max, so worst-case coverage is unchanged.

  - mmq.cuh: replace the RDNA3.5 MoE J_max=48 clamp with the typical-width
    picker; J_max is now a uniform 128 and the search terminates on its own.
  - mmq-table-check.cpp: pick_J mirrors the selection loop, so update it in
    lockstep or the check silently becomes a rubber stamp. Adds the routed-MoE
    arg shape from mmq.cu and a crossover case.

For Qwen 3.6 35B-A3B (256 experts, 8 active) at the production ub=2048 the
typical width is 2048*8/256 = 64, so this is a 48 -> 64 change at our operating
point - narrow enough that it may land in noise. Verified host-side without
ROCm: MoE J=64, dense J=128, no aborts across all 21 types.

Above ub=4096 the typical width reaches 128, the picker disengages and the
search runs to J=128 where the cap held 48. Findings ggml-org#5/ggml-org#8 only ever measured
the typical=64 regime, so that is untested on this chip; production ub=2048
cannot reach it. See strix-halo/mmq-moe-ncols-picker.md for the bench plan and
the keep/revert criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Aug 3, 2026
@justinappler @claude
strix-halo: revert routed-MoE ncols picker, flat vs static J=48
c9da8a0 
Benched b47bb31 against 05e837f on gfx1151 (ROCm 7.14.0, canonical Qwen 3.6
35B-A3B Q4_K_XL matrix). The picker is not distinguishable from the static cap
at the production operating point:

  pp512  @ d=0      1428.13 -> 1388.93  (-2.7%)
  pp512  @ d=2048   1299.39 -> 1316.26  (+1.3%)
  pp512  @ d=8192   1135.42 -> 1142.23  (+0.6%)
  pp512  @ d=16384   971.25 ->  977.36  (+0.6%)

Correctness was clean first (790/790 MUL_MAT_ID, 1134/1134 MUL_MAT), so the
grid does still cover ncols_max while the tile is sized from the typical width.

The tg128 control fell ~1.6% at every depth. This change cannot move tg -
decode goes through MMVQ where ncols_dst==1, so the picker never runs - which
makes that a session drift between two builds measured on different days, the
same contamination Finding ggml-org#9 hit. The conclusion survives it either way: raw,
pp is -2.7% to +1.3%; calibrated against the drift, -1.2% to +2.9%. Every depth
is inside the host's ~2% noise floor, so the effect is smaller than this rig can
resolve, and the d=0 -2.7% is no more real than the d=2048 +1.3%.

So: at ub=2048, J=48 and J=64 are the same speed. The doc predicted this before
the numbers existed ("narrow enough that it may well land in noise"), which is
the only reason that reading is credible rather than post-hoc. Reverting to the
static cap because it is the validated status quo, not because it won.

Worth reporting on ggml-org#24546: the picker is neutral on RDNA3.5,
on the exact model and ubatch its sweep claims +7.57% for on gfx1100. The PR is
gated on GGML_CUDA_CC_IS_RDNA3, which covers both. gfx1151's table is I=64/
nthreads=128 vs gfx1100's I=128/nthreads=256 - our tiles are already half-width,
so there is much less worst-case over-sizing left to recover. That is the data
ravel7524 asked for on 2026-07-03.

Doc kept with the numbers and the untested ub>=4096 crossover regime recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@IMbackK

IMbackK commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

I think this is a reasonable idea, but it would have to be updated for the changes in #24127

@ravel7524
ravel7524 force-pushed the mmqx-rdna3-routed-moe-tiling branch from 9ab2b07 to 3087cea Compare August 5, 2026 18:48
@ravel7524

ravel7524 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

I adapted the implementation to the MMQ configuration changes from #24127 and I provisionally enabled the selector for NVIDIA Volta and newer so those architectures can be benchmarked.

IMbackK
IMbackK requested changes Aug 5, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh Outdated
@IMbackK

IMbackK commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

I tested it on CDNA 1 and it seams that its not entirely a win in all cases.

Model Test Profile b10288 t/s PR t/s Delta (%)
qwen35moe 122B.A10B Q4_K - Medium pp32 244.15 ± 2.58 291.14 ± 1.43 +19.25%
pp512 649.31 ± 3.81 733.93 ± 3.05 +13.03%
pp2048 1128.57 ± 7.95 1245.77 ± 2.55 +10.39%
tg32 52.26 ± 0.78 52.69 ± 0.87 +0.82%
deepseek2 30B.A3B Q8_0 pp32 402.87 ± 3.37 441.00 ± 8.24 +9.46%
pp512 1508.40 ± 5.74 1494.33 ± 4.37 -0.93%
pp2048 1327.65 ± 2.52 1308.98 ± 2.13 -1.41%
tg32 84.63 ± 1.21 86.51 ± 1.28 +2.22%
deepseek2 30B.A3B Q8_0 pp32 @ d16384 212.65 ± 1.43 223.26 ± 1.42 +4.99%
pp512 @ d16384 334.95 ± 0.46 323.71 ± 28.41 -3.36%
pp2048 @ d16384 315.96 ± 9.35 315.71 ± 9.81 -0.08%
tg32 @ d16384 56.34 ± 0.64 57.04 ± 0.63 +1.24%
gpt-oss 120B MXFP4 MoE pp32 @ d16384 327.72 ± 5.80 369.25 ± 3.87 +12.67%
pp512 @ d16384 912.97 ± 5.61 831.88 ± 0.86 -8.88%
pp2048 @ d16384 1257.41 ± 178.26 1183.48 ± 174.00 -5.88%
tg32 @ d16384 99.70 ± 0.73 99.47 ± 0.86 -0.23%

IMbackK
IMbackK previously approved these changes Aug 6, 2026
View reviewed changes

@IMbackK IMbackK left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont bother cleaning this up if its not uniformly performance positive at least on rdna3

Comment thread ggml/src/ggml-cuda/mmq.cuh Outdated
@IMbackK
IMbackK self-requested a review August 6, 2026 19:15
@IMbackK
IMbackK dismissed their stale review August 6, 2026 19:16

missclick

@ravel7524
ravel7524 force-pushed the mmqx-rdna3-routed-moe-tiling branch from 9c9bcc8 to 7d8bc65 Compare August 30, 2026 12:30
@ravel7524

ravel7524 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Merged master and the conflict resolved.
Apologies for the churn in between, a bad push briefly put ~350 unrelated commits on this branch. That's been corrected; it's back to the original commits plus the merge, 10 files, +36/−12.
One thing on the results: 0b5be7e4a (#26284) retuned the RDNA3 CASE ladder earlier today, which is the table this PR builds on. My benchmarks predate it, so I'm re-running the A/B on gfx1100 against the new baseline and will post updated numbers before this is worth a review pass.

@ggerganov ggerganov removed the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 30, 2026
@ggerganov

ggerganov commented Aug 30, 2026

Copy link
Copy Markdown
Member

Ok, I will remove the "merge ready" label for now. Let us know when your A/B tests are complete.

@CISC CISC removed documentation Improvements or additions to documentation model Model specific build Compilation issues testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs Vulkan Issues specific to the Vulkan backend devops improvements to build systems and github actions server labels Aug 30, 2026
@ravel7524

ravel7524 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author
model / quant experts ubatch pp32 pp512 pp2048 pp8192 tg128
Qwen3-30B-A3B Q4_K_M 128 512 733.20 → 863.53 (+17.78%) 2289.62 → 3189.41 (+39.30%) 2197.17 → 2965.48 (+34.97%) 1819.55 → 2303.36 (+26.59%) 122.44 → 122.48 (+0.03%)
Qwen3-30B-A3B Q4_K_M 128 1024 733.06 → 860.32 (+17.36%) 2283.84 → 3182.35 (+39.34%) 2911.50 → 3308.42 (+13.63%) 2265.44 → 2506.82 (+10.65%) 122.89 → 122.29 (-0.49%)
Qwen3-30B-A3B Q4_K_M 128 2048 736.29 → 861.27 (+16.97%) 2280.25 → 3183.38 (+39.61%) 3321.97 → 3320.29 (-0.05%) 2528.55 → 2530.69 (+0.08%) 122.36 → 122.29 (-0.06%)
Qwen3-30B-A3B Q4_K_M 128 4096 733.61 → 860.68 (+17.32%) 2280.76 → 3186.56 (+39.71%) 3322.12 → 3320.10 (-0.06%) 2581.54 → 2572.63 (-0.35%) 122.36 → 123.23 (+0.71%)
Qwen3-30B-A3B Q4_K_M 128 8192 737.28 → 861.10 (+16.79%) 2283.13 → 3182.73 (+39.40%) 3325.78 → 3316.32 (-0.28%) 2391.97 → 2408.30 (+0.68%) 122.42 → 122.55 (+0.11%)
Gemma-4-26B-A4B Q4_K_M 128 512 797.88 → 890.61 (+11.62%) 2459.39 → 3233.34 (+31.47%) 2279.49 → 2932.43 (+28.64%) 2045.13 → 2560.17 (+25.18%) 81.47 → 81.64 (+0.21%)
Gemma-4-26B-A4B Q4_K_M 128 1024 797.64 → 890.94 (+11.70%) 2458.11 → 3226.82 (+31.27%) 2943.36 → 3260.88 (+10.79%) 2525.81 → 2762.04 (+9.35%) 81.54 → 81.62 (+0.10%)
Gemma-4-26B-A4B Q4_K_M 128 2048 797.25 → 889.69 (+11.59%) 2456.71 → 3228.90 (+31.43%) 3330.54 → 3324.15 (-0.19%) 2723.18 → 2701.49 (-0.80%) 81.39 → 81.52 (+0.16%)
Gemma-4-26B-A4B Q4_K_M 128 4096 795.16 → 890.82 (+12.03%) 2457.95 → 3229.32 (+31.38%) 3328.93 → 3325.16 (-0.11%) 2662.87 → 2664.16 (+0.05%) 81.54 → 81.56 (+0.02%)
Gemma-4-26B-A4B Q4_K_M 128 8192 795.25 → 887.73 (+11.63%) 2455.83 → 3230.34 (+31.54%) 3331.74 → 3327.79 (-0.12%) 2300.03 → 2298.01 (-0.09%) 81.47 → 81.48 (+0.01%)
Gemma-4-26B-A4B Q6_K 128 512 623.94 → 780.25 (+25.05%) 2005.69 → 2642.57 (+31.75%) 1880.54 → 2425.13 (+28.96%) 1741.74 → 2168.59 (+24.51%) 79.73 → 79.96 (+0.29%)
Gemma-4-26B-A4B Q6_K 128 1024 623.48 → 776.50 (+24.54%) 2000.23 → 2630.37 (+31.50%) 2499.55 → 2832.09 (+13.30%) 2189.62 → 2446.66 (+11.74%) 79.80 → 79.90 (+0.13%)
Gemma-4-26B-A4B Q6_K 128 2048 623.42 → 774.26 (+24.20%) 1997.23 → 2630.83 (+31.72%) 2921.14 → 2915.42 (-0.20%) 2449.62 → 2433.62 (-0.65%) 79.73 → 79.85 (+0.15%)
Gemma-4-26B-A4B Q6_K 128 4096 625.23 → 774.59 (+23.89%) 1996.45 → 2626.36 (+31.55%) 2921.40 → 2913.61 (-0.27%) 2459.81 → 2439.36 (-0.83%) 79.80 → 79.87 (+0.09%)
Gemma-4-26B-A4B Q6_K 128 8192 621.83 → 773.96 (+24.46%) 1999.66 → 2629.99 (+31.52%) 2917.04 → 2912.77 (-0.15%) 2149.42 → 2161.37 (+0.56%) 79.68 → 79.95 (+0.34%)
Qwen3.6-35B-A3B Q3_K_M 256 512 642.02 → 724.77 (+12.89%) 2137.51 → 2631.25 (+23.10%) 2075.64 → 2548.21 (+22.77%) 1857.17 → 2282.59 (+22.91%) 97.24 → 97.30 (+0.06%)
Qwen3.6-35B-A3B Q3_K_M 256 1024 641.49 → 721.41 (+12.46%) 2133.10 → 2626.16 (+23.11%) 2736.59 → 3064.64 (+11.99%) 2422.21 → 2692.86 (+11.17%) 97.24 → 97.36 (+0.12%)
Qwen3.6-35B-A3B Q3_K_M 256 2048 643.06 → 721.57 (+12.21%) 2132.36 → 2632.68 (+23.46%) 3195.90 → 3406.64 (+6.59%) 2820.19 → 2974.40 (+5.47%) 97.39 → 97.24 (-0.15%)
Qwen3.6-35B-A3B Q3_K_M 256 4096 641.20 → 722.80 (+12.73%) 2137.60 → 2630.12 (+23.04%) 3201.18 → 3411.91 (+6.58%) 3091.48 → 3075.36 (-0.52%) 97.28 → 97.47 (+0.20%)
Qwen3.6-35B-A3B Q3_K_M 256 8192 643.08 → 723.63 (+12.53%) 2134.86 → 2633.19 (+23.34%) 3206.19 → 3417.45 (+6.59%) 3046.17 → 3027.84 (-0.60%) 97.52 → 97.32 (-0.21%)
GPT-OSS-20B MXFP4 32 512 670.76 → 1062.08 (+58.34%) 3027.83 → 3670.29 (+21.22%) 2990.12 → 3567.64 (+19.31%) 2750.83 → 3213.06 (+16.80%) 151.38 → 151.72 (+0.22%)
GPT-OSS-20B MXFP4 32 1024 673.09 → 1066.60 (+58.46%) 3030.37 → 3667.32 (+21.02%) 3711.92 → 3708.66 (-0.09%) 3326.50 → 3308.92 (-0.53%) 151.30 → 151.54 (+0.16%)
GPT-OSS-20B MXFP4 32 2048 672.65 → 1064.39 (+58.24%) 3028.62 → 3667.55 (+21.10%) 4081.59 → 4085.46 (+0.09%) — 151.12 → 151.51 (+0.26%)
GPT-OSS-20B MXFP4 32 4096 672.28 → 1066.36 (+58.62%) 3030.76 → 3668.58 (+21.04%) 4080.02 → 4078.49 (-0.04%) — 151.37 → 151.75 (+0.25%)
GPT-OSS-20B MXFP4 32 8192 672.23 → 1064.85 (+58.41%) 3026.77 → 3673.09 (+21.35%) 4084.70 → 4081.69 (-0.07%) — 150.97 → 151.28 (+0.21%)

No regressions larger than 0.83% were observed. Results at and beyond the expected ubatch crossover were effectively neutral, while smaller ubatches showed consistent gains; tg128 also remained neutral.

@Dev-next-gen

This comment was marked as spam.

Sign in to view
@ravel7524

ravel7524 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Sorry @ggerganov, forgot to ping you, benchmark results are positive and the additional comment confirms the results

@Casten-Wang

Casten-Wang commented Sep 1, 2026

Copy link
Copy Markdown

I ran an independent Windows/gfx1151 backport experiment based on the typical-expert-width MMQ configuration idea in this PR.

Environment: Windows 11, AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151, 20 CUs), ROCm 7.1, driver 32.0.31021.5001, Qwen3.6-35B-A3B-UD-Q4_K_M, full GPU offload, and mmap=off.

Important limitation: this was implemented in an older MMQ scanner, so it is not a test of the current PR HEAD and was not rebased after PR #26284.

Operator correctness after making the loader, kernel and write-back wave counts consistent:

  • Q4_K: 153/153 passed
  • Q5_K: 23/23 passed
  • Q6_K: 23/23 passed

Operator median improvements for typical expert width 16:

  • Q4_K uniform routing: +21.66%
  • Q4_K skewed routing: +15.41%
  • Q5_K uniform routing: +7.45%
  • Q5_K skewed routing: +13.79%
  • width-64 negative controls remained within approximately ±0.33%

Q4_K_M model-level Prefill at p2048 / ubatch=512 / batch=4096 / threads=4, 12 samples per version:

  • baseline median: 1630.17 token/s
  • candidate median: 1667.29 token/s
  • change: +2.28%

The broad configuration initially regressed ubatch=1024 by -1.74% because it selected an unvalidated x32/y64 configuration. Restricting the typical-width override to the validated x16 case removed that regression:

  • ubatch=1024: +0.12%
  • ubatch=2048: -0.07%
  • Decode, 128 tokens: -0.20%

The final scoped candidate passed Q4_K 107/107, Q5_K 23/23, and Q6_K 23/23 checks.

These results support testing a narrowly scoped gfx1151/rdna3-5 configuration, especially for typical width 16. A fresh A/B against the current PR HEAD and current master is still required before treating this as direct validation.

@Casten-Wang

Casten-Wang commented Sep 2, 2026

Copy link
Copy Markdown

Tested on Windows 11, ROCm 7.1, Radeon 8060S (gfx1151).

  • Q4_K MUL_MAT_ID correctness: 73/73 passed.
  • Model: Qwen3.6-35B-A3B Q4_K_M
    • Prefill (input processing): +0.16% / +0.64% / -0.07% at ubatch = 512 / 1024 / 2048.
    • Decode (output generation): +0.02%, essentially flat.

One thing worth flagging: in the current source, gfx1151 sets use_typical_moe_ncols = false, so this PR's core typical-expert-width strategy isn't actually taken on gfx1151. That likely explains why I see no measurable change here — the target optimization path isn't enabled for this arch. No correctness regressions either way.

IMbackK
IMbackK approved these changes Sep 2, 2026
View reviewed changes

@IMbackK IMbackK left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is still beneficial against and when rebated on latest master.

Details
GPU Model Microbatch size Test t/s master t/s mmqx-rdna3-routed-moe-tiling Speedup
RX 7900 XTX gemma4 26B.A4B Q6_K 8 pp2048 430.12 430.35 1.00
RX 7900 XTX gemma4 26B.A4B Q6_K 64 pp2048 1058.57 1553.66 1.47
RX 7900 XTX gemma4 26B.A4B Q6_K 512 pp2048 2502.63 3197.90 1.28
RX 7900 XTX gemma4 26B.A4B Q6_K 1024 pp2048 3316.65 3773.25 1.14
RX 7900 XTX gpt-oss 20B MXFP4 MoE 8 pp2048 636.17 635.05 1.00
RX 7900 XTX gpt-oss 20B MXFP4 MoE 64 pp2048 1594.12 2116.75 1.33
RX 7900 XTX gpt-oss 20B MXFP4 MoE 512 pp2048 3846.41 4696.47 1.22
RX 7900 XTX gpt-oss 20B MXFP4 MoE 1024 pp2048 4809.61 4831.44 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 8 pp2048 402.41 402.97 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 64 pp2048 2451.46 2449.18 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 512 pp2048 6072.19 6058.35 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 1024 pp2048 8385.86 8395.05 1.00
RX 7900 XTX lfm2moe 8B.A1B Q4_0 8 pp2048 1413.81 1417.80 1.00
RX 7900 XTX lfm2moe 8B.A1B Q4_0 64 pp2048 3433.56 4559.60 1.33
RX 7900 XTX lfm2moe 8B.A1B Q4_0 512 pp2048 9144.60 10884.34 1.19
RX 7900 XTX lfm2moe 8B.A1B Q4_0 1024 pp2048 11583.59 11661.76 1.01
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 8 pp2048 1003.92 1012.50 1.01
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 64 pp2048 3185.46 4461.54 1.40
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 512 pp2048 8429.84 10221.60 1.21
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 1024 pp2048 10737.57 10826.09 1.01
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 8 pp2048 985.99 989.12 1.00
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 64 pp2048 3128.37 4400.83 1.41
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 512 pp2048 8249.88 9973.57 1.21
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 1024 pp2048 10564.36 10640.39 1.01
RX 7900 XTX lfm2moe 8B.A1B Q6_K 8 pp2048 852.31 860.02 1.01
RX 7900 XTX lfm2moe 8B.A1B Q6_K 64 pp2048 2257.27 3378.81 1.50
RX 7900 XTX lfm2moe 8B.A1B Q6_K 512 pp2048 5863.58 7124.40 1.22
RX 7900 XTX lfm2moe 8B.A1B Q6_K 1024 pp2048 7587.75 7671.78 1.01
RX 7900 XTX lfm2moe 8B.A1B Q8_0 8 pp2048 1020.85 1024.86 1.00
RX 7900 XTX lfm2moe 8B.A1B Q8_0 64 pp2048 3461.44 4352.49 1.26
RX 7900 XTX lfm2moe 8B.A1B Q8_0 512 pp2048 9130.16 11053.51 1.21
RX 7900 XTX lfm2moe 8B.A1B Q8_0 1024 pp2048 11759.62 11835.76 1.01
RX 7900 XTX qwen35 27B Q5_K_M 8 pp2048 98.13 98.63 1.01
RX 7900 XTX qwen35 27B Q5_K_M 64 pp2048 557.11 558.72 1.00
RX 7900 XTX qwen35 27B Q5_K_M 512 pp2048 758.67 762.83 1.01
RX 7900 XTX qwen35 27B Q5_K_M 1024 pp2048 776.35 778.11 1.00
Backend GGML op Op parameters TFLOPS master TFLOPS mmqx-rdna3-routed-moe-tiling Speedup
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.85 0.89 1.04
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 5.07 5.13 1.01
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.49 6.39 0.98
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.64 1.76 1.07
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 0.91 0.83 0.91
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 7.07 7.13 1.01
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 2.94 2.83 0.96
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 0.80 0.83 1.04
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 0.87 0.89 1.02
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.07 6.66 1.10
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 8.04 8.04 1.00
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 2.78 2.71 0.98
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 0.88 0.94 1.07
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 14.21 14.20 1.00
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 4.77 4.73 0.99
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 0.85 0.85 1.01
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.87 0.89 1.03
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 0.76 0.75 0.98
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 1.01 0.95 0.94
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 0.33 0.35 1.04
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 0.46 0.48 1.04
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 1.78 1.81 1.02
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 0.55 0.58 1.06
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 0.49 0.47 0.97
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 0.89 0.88 0.99
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 1.51 1.49 0.99
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 2.68 2.67 1.00
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 1.05 1.02 0.97
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 0.51 0.61 1.20
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 4.32 4.14 0.96
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 1.27 1.27 1.00
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 0.51 0.51 1.01
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.87 0.89 1.03
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 2.58 7.93 3.07
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 4.87 11.11 2.28
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.79 2.40 1.34
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 1.65 1.74 1.06
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 9.06 15.63 1.73
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 2.20 3.87 1.76
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 1.08 1.07 0.99
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.32 1.32 1.00
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 4.99 11.13 2.23
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 9.48 16.40 1.73
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.12 3.75 1.20
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 1.65 1.70 1.03
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 18.59 21.45 1.15
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 4.20 7.23 1.72
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 1.23 1.16 0.95
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=1,k=2880 3.20 3.20 1.00
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=4,k=2880 4.86 4.86 1.00
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=512,k=2880 24.47 31.41 1.28
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=8,k=2880 4.43 4.61 1.04
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 1.35 1.41 1.04
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 3.39 11.61 3.43
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.25 16.16 2.59
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 2.15 3.79 1.76
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 4.87 5.07 1.04
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 12.03 17.68 1.47
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 3.09 6.46 2.09
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 6.12 6.23 1.02
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.44 1.46 1.01
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.71 17.51 2.61
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 12.84 21.70 1.69
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.90 6.66 1.71
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 5.34 5.18 0.97
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 24.57 28.93 1.18
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 6.06 12.65 2.09
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 6.47 6.59 1.02
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.91 0.96 1.05
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 3.30 12.38 3.76
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.11 16.18 2.65
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 2.82 4.10 1.45
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 2.64 2.72 1.03
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 11.77 23.02 1.96
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 3.09 6.80 2.20
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 2.02 1.85 0.92
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.84 1.86 1.01
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.58 19.32 2.94
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 12.44 26.08 2.10
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 5.09 7.06 1.39
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 2.75 2.83 1.03
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 23.96 29.67 1.24
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 6.11 13.81 2.26
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 2.41 2.63 1.09
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 1.31 1.34 1.03
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 2.02 8.37 4.14
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 3.79 11.79 3.11
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.78 2.48 1.39
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 1.94 2.03 1.04
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 7.47 15.10 2.02
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 1.89 4.47 2.37
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 1.18 1.34 1.13
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.26 1.38 1.10
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 3.98 11.99 3.01
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 7.42 16.34 2.20
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.03 4.21 1.39
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 2.15 2.13 0.99
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 14.78 18.03 1.22
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 3.60 8.56 2.38
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 1.59 1.53 0.96
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 1.28 1.33 1.04
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 3.43 9.96 2.90
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.40 16.13 2.52
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.96 3.14 1.60
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 2.82 2.87 1.02
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 12.16 17.79 1.46
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 3.23 5.47 1.69
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 2.72 2.58 0.95
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.35 1.36 1.01
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.66 16.99 2.55
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 12.76 19.17 1.50
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.42 5.27 1.54
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 3.89 3.90 1.00
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 24.59 31.39 1.28
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 6.17 10.11 1.64
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 4.23 4.85 1.15

@JohannesGaessler

JohannesGaessler commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Let me be frank: I am not happy with the process by which this PR was merged. I was not aware of its existence until I saw the changes to my local repository and would have appreciated to at least be asked for input. As I said off-site, while I am currently not reading all of my Github notifications I am still available when contacted via other channels by maintainers.

@CISC

CISC commented Sep 7, 2026

Copy link
Copy Markdown
Member

Where did merge ready come from? According to this thread it was last removed by @ggerganov.

@pwilkin

pwilkin commented Sep 7, 2026

Copy link
Copy Markdown
Member

@JohannesGaessler @CISC

I think @IMbackK readded it, I know it showed as "Merge ready" on the filter. Maybe some bug?

Maybe we come back to designating a subgroup of maintainers who are fine to get pinged and get the second approvals that way? Today I did a run over the "merge ready" PRs and some of them were sitting there for quite a few days - and then there's a situation like this one. I feel like the mechanism isn't working as intended.

If you feel this is premature, we can revert this.

JohannesGaessler
JohannesGaessler reviewed Sep 7, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh
if (config_max.use_typical_moe_ncols) {
// Use the typical expert width only for tile selection.
// The launch grid still uses args.ncols_max.
const int64_t ncols_typical = (args.ncols_dst + args.nchannels_x - 1) / args.nchannels_x;

@JohannesGaessler JohannesGaessler Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way this is being calculated is logically incorrect for >1 expert used. The average number of output columns is ncols_dst * n_expert_used / nchannels_x. Fundamentally you have a tradeoff between wasted work with large tiles and excessive overhead/bad arithmetic intensity with small tiles. Even if the logic here is wrong this can still be a good target value simply based on empirical findings.

@JohannesGaessler

JohannesGaessler commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

I don't care whether it is a revert or a followup but the code should definitely not stay like this because it touches the wrong parts of the kernel for what it does. Instead of in the kernel configurations, the logic for the value to pick a concrete tile size aginst should be 100% on the host side. Just extend mmq_args with a value like ncols_opt that is intended to be used as the value to optimize the tile sizes against. For dense models, simply use ncols_dst. For MoE models, pick a value < ncols_max where it can be empirically found to be faster. The entire logic for the determination of ncols_opt should be in ggml_cuda_mul_mat_q.

This was referenced Sep 7, 2026
Merged
Merged
@IMbackK IMbackK mentioned this pull request Sep 10, 2026
Open
@raufaser raufaser mentioned this pull request Sep 10, 2026
Merged
@ggerganov ggerganov mentioned this pull request Sep 14, 2026
Merged
@BrewTestBot BrewTestBot mentioned this pull request Sep 14, 2026
Merged
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JohannesGaessler JohannesGaessler JohannesGaessler left review comments
@pwilkin pwilkin pwilkin approved these changes
@IMbackK IMbackK IMbackK approved these changes
@ggerganov ggerganov Awaiting requested review from ggerganov

Assignees

@IMbackK IMbackK

@JohannesGaessler JohannesGaessler

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@ravel7524 @GZGavinZhao @IMbackK @frizikk @ggerganov @Dev-next-gen @Casten-Wang @JohannesGaessler @CISC @pwilkin