[Xarbirus]

Overview

Fixes #30579.
For Mellum it sets GGML_PREC_F32 on src1 for ffn_moe_down, which enables the src1 rescale in the Metal mul_mm_id. The input to Mellum's ffn_moe_down reaches ~2.9e5, while the F16 limit is 65504.
Reuses the mechanism from #29029 (follow-up of #26223), same as for Mistral4.

Additional information

Only Metal (Apple M2 Max) and CPU tested.

With GGML_PREC_F32 set CUDA, Vulkan, WebGPU, Hexagon, and OpenCL (with Adreno MoE kernels) return false from supports_op for MUL_MAT_ID. It means that ffn_moe_down in all layers of Mellum will run on the CPU. Same trade-off as for Mistral4 (#29029).

Perplexity

Prompt and command as in the issue (-c 2048 -b 2048, -ngl 99 for Metal, -ngl 0 for CPU). Base = master df18bee11, fix = b418747f5.

Quant	Metal base	Metal fix	CPU (base == fix)
BF16	2.2062 2.1612 2.3576	2.2062 2.1612 2.3576	2.2054 2.1612 2.3583
Q8_0	2.1771 2.1360 nan	2.1771 2.1360 2.3170	2.1839 2.1340 2.3094
Q6_K	2.2696 2.2252 nan	2.2696 2.2252 2.4302	2.2640 2.2225 2.4359
Q4_K_M	2.2387 2.2290 nan	2.2387 2.2290 2.5032	2.2416 2.2324 2.5094
MXFP4_MOE	2.2803 2.2543 2.4710	2.2803 2.2543 2.4710	2.2739 2.2469 2.4555

• Chunks without overflow are bit-identical to base
• BF16 / MXFP4_MOE identical with and without the fix (bfloat staging, power-of-two rescale is exact)
• CPU identical for base and fix

Performance (Metal)

llama-bench -ngl 99 -p 512,2048 -n 128 -r 5. ABBA order (base, fix, fix, base), 90 s pause before each pass. Values are t/s, two passes per variant; diff is best fix vs best base.

Quant	Test	Base	Fix	Diff
BF16	pp512	1190.7 / 1446.9	1446.1 / 1444.1	-0.1%
BF16	pp2048	1350.4 / 1384.0	1381.6 / 1379.2	-0.2%
BF16	tg128	66.9 / 67.5	67.3 / 67.4	-0.1%
Q8_0	pp512	1605.8 / 1617.5	1615.9 / 1609.7	-0.1%
Q8_0	pp2048	1533.6 / 1514.5	1533.3 / 1531.5	-0.0%
Q8_0	tg128	91.6 / 90.3	90.6 / 91.4	-0.2%
Q6_K	pp512	1453.8 / 1461.9	1449.1 / 1453.6	-0.6%
Q6_K	pp2048	1390.3 / 1395.9	1384.1 / 1385.4	-0.7%
Q6_K	tg128	95.3 / 94.9	94.7 / 94.7	-0.6%
Q4_K_M	pp512	1446.6 / 1460.8	1458.6 / 1453.9	-0.1%
Q4_K_M	pp2048	1376.5 / 1384.6	1388.7 / 1385.9	+0.3%
Q4_K_M	tg128	106.8 / 111.3	109.7 / 110.2	-1.0%
MXFP4_MOE	pp512	1225.4 / 1229.8	1223.7 / 897.1	-0.5%
MXFP4_MOE	pp2048	1176.9 / 1182.6	1163.5 / 1038.4	-1.6%
MXFP4_MOE	tg128	124.1 / 124.0	123.1 / 126.5	+1.9%

Tests

• test-backend-ops test -o MUL_MAT_ID -b MTL0 : 883/883 passed on MTL0 vs CPU (including 6 cases with src1 above F16 and GGML_PREC_F32 (amax=100000; q8_0 and q4_K; n = 16 / 32 / 64))

• test-llama-archs -a '^mellum$' : 4 tests passed

• Full local CI (GG_BUILD_METAL=1 bash ./ci/run.sh <results> <mnt>), all steps passed

  • ctest_debug 51/51, ctest_release 52/52, ctest_with_model_debug / _release 5/5 each
  • test_backend_ops (all ops, Metal vs CPU): 17604/17604
  • test_llama_archs_models / _tensor_split, embd_bge_small, rerank_tiny, test_scripts, qwen3_0_6b: passed

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, research, benchmark, tests

[Xarbirus]

Multiple open PRs from a new contributor

Is this still relevant, given that I already have a few contributions?

[CISC]

Multiple open PRs from a new contributor

Is this still relevant, given that I already have a few contributions?

No, though you may hit a hard limit at some point.

[taronaeo]

I think we should block the merge until tomorrow as @/ggerganov is preparing the next release. See: #30588 (comment)

[Xarbirus]

@CISC @taronaeo it seems we can continue

[taronaeo]

I think its OK to merge now, given that there have been pushes already. CI failures unrelated to PR. Merging.

[Green-Sky]

This pr destroyed performance on CUDA, see #30704 .