[leehack]

Replaces patch 0006 with a faster overflow-safe F16 GEMM. The first 0006 (#177) accumulated every llamafile F16 tile in F32, which cut F16-weight prefill on Apple CPUs to 0.44× of v0.8.0-3. This version gets back to 0.91× (FA off) and 0.94× (FA on). Finite inputs still never overflow.

Refs #174 (keep open). Upstream: ggml-org/llama.cpp#31734.

Design

The FP16 tinyBLAS<8, float16x8_t, ...> tile keeps its F16 FMAs and upstream order, with two changes:

1. Blocked widening. Each lane sums at most 64 products (512 K elements) in F16. That sum is then added into a per-output F32 accumulator: FMLAL/FMLAL2 by 1.0 with FP16 FML, or F16→F32 conversion plus an F32 add otherwise.
2. Detect and redo. An F16 overflow leaves inf/NaN, and that value survives every later F16 and F32 add. After the tile is stored, any nonfinite output causes the whole tile to be recomputed by the F32 tile from the first 0006 (tinyBLAS<4, float32x4_t, float32x4_t, ggml_fp16_t, ggml_fp16_t, float>, called through a friend declaration). The recomputed results overwrite the tile.

Patch: one sgemm.cpp hunk adds gemm_bloc_f16 and a dispatch from gemm_bloc when D is float16x8_t. A second hunk is the unchanged k >= 0 marker guard. Both apply strictly to the new pin v0.9.0 (cd959bae, from #175) and to every LLAMADART_UPSTREAMS row (v0.7.0 620d2225, v0.8.0 8642541b, v0.9.0).

Correctness contract

	First 0006 (F32 tile)	This 0006
Finite F16 inputs, any magnitude	never nonfinite	never nonfinite (an overflowing tile is redone in F32)
Tile that overflowed in F16, or had a nonfinite input	—	bit-identical to the first 0006
Tile with no overflow, k ≤ 512	F32-accurate	bit-identical to the unpatched upstream tile (checked by output hash at k = 64, 136, 512)
Tile with no overflow, k > 512	F32-accurate	F16 rounding within each 512-element block; blocks summed in F32. More accurate than upstream (max error/Σ|p| at k = 4864: 9.6e-5 vs 3.5e-4); F32 tile 8e-8
Results below the F16 subnormal range	kept	rounded to zero, as upstream
Every tile overflowing (worst case)	2.5× unpatched	3.6× unpatched (1.4× the F32 tile)

No magnitude bound is needed for safety: the block length only sets precision and how often tiles are redone. An over-read past k that reaches only NaN padding would be hidden by the redo. The attention case catches an over-read that reaches the next row.

Choosing the design

M4 Max, GGML_NATIVE=OFF, F16 GEMM 4096×4096×512, time against the unpatched tile (median of 2):

Candidate	1 thread	8 threads
First 0006 (F32 tile)	+163%	+153%
Blocked, 16 products/lane, 4×6 tile	+40%	+37%
Blocked, 16, split into 4×3 tiles so F32 accumulators stay in registers	+22%	+21% (the narrower tile is slower even without flushes)
Blocked, 32	+20–24%	+20%
Blocked, 64 (chosen), FMLAL / conversion flush	+13% / +14%	+10% / +13%
Peeling or fully unrolling the block	no change	no change

FMLAL and FMLA.8H have the same throughput on M4: about 3.4 per cycle in a microbenchmark. A tile that widens on every FMA can therefore get at most half the F16 tile's speed. The earlier tinyBLAS<8, float32x4x2_t> attempt was slower still (+270%) because it spilled. So the speed comes from F16 FMAs, and safety comes from detecting overflow. End to end, pp512 was 0.79 with 16 products per lane, 0.87 with 32 and 0.92 with 64 (FA off). 64 keeps k ≤ 512 bit-identical to upstream and gives the conversion path (linux_armv8.2_2) the most headroom.

Performance

M4 Max, CPU only, 8 threads, Qwen2.5-0.5B-Instruct, F16 KV. Runtimes: shipped v0.8.0-3 macOS dylib, the 0006 dylib built from a073d08, and this head (macos-arm64-full preset). 3 interleaved rounds; each ratio is the median of the per-round ratios (thermal drift moved absolute numbers by about 10%):

Model	FA	Test	v0.8.0-3 t/s	First 0006	This PR
F16	off	pp512	1866	0.44	0.91 (0.87–0.94)
F16	on	pp512	1743	0.46	0.94 (0.93–0.95)
F16	off / on	tg128	202 / 204	0.99 / 0.96	1.00 / 0.98
F16	off / on	tg64@d1024	177 / 187	1.02 / 0.93	1.02 / 0.96
Q8_0	off / on	pp512	2442 / 2627	0.92 / 1.00	0.98 / 1.01
Q8_0	off / on	tg128, tg64@d1024	268–355	0.99–1.02	1.01–1.03

An earlier run of the same fast path, which differed only in an unoptimized redo, gave F16 pp512 at 0.92 / 0.94. Decode never reaches the tile (n < 8), so tg differences are noise. All 216 result rows reported 0 nonfinite logits.

Qwen3-TTS cancel latency (core validation tts pack, --backend cpu, Mac, q174/harness/mac_tts.sh), all samples pooled:

Runtime	Runs (samples)	Median	p90	Max
v0.8.0-3	4 (36)	14.6 ms	31.8 ms	48.1 ms
First 0006	1 (9)	64.8 ms (4.4×)	75.3 ms	78.6 ms
This PR	3 (27)	16.3 ms (1.12×)	39.6 ms (1.25×)	49.0 ms (1.02×)

Every pack exited 0 within the 500 ms budget. Neither the 4.4× of the first 0006 nor this PR's ratio has been measured on linux_armv8.2_2 hardware. That variant runs the conversion flush, which costs +12–14% GEMM time on M4 in a armv8.2-a+fp16+dotprod build.

Validation (code at 6f419890; head 6c9ad9c8 merges main at b1ce76c8 (#175, v0.9.0 pin))

The merges were clean. #175's manual-dispatch wording in docs/native174_f16_dot_accumulation.md is kept. Benchmarks below were run on the v0.8.0 pin; the 0006 hunks and their context are identical on v0.9.0.

Check	Result
llamadart_f16_dot_accumulation_test, default Apple flags (FMLAL, 306 fmlal in test binary)	PASS
Same, GGML_CPU_ARM_ARCH=armv8.2-a+fp16+dotprod (0 fmlal)	PASS
Same two builds on the new pin v0.9.0 (cd959bae) after merging #175	PASS / PASS
Auditor reproducers batch.cpp (1–32 columns) and attn.cpp (ordinary + flash, 1–32 queries), both builds, v0.8.0 and v0.9.0	0 nonfinite
apply --strict on the v0.9.0 pin; 0006 applies to the v0.7.0, v0.8.0 and v0.9.0 rows	OK
CI on head 6c9ad9c8	46/46 SUCCESS, including macos-exit-teardown release and ASan for v0.7.0, v0.8.0 and v0.9.0 (FMLAL and conversion F16 test)
CI on previous head 40b526e7 (before #175)	34/34 SUCCESS
python3 -m unittest discover -s tests -p 'test_*.py'	448 OK (1 skipped) at 6f419890; 446 OK (1 skipped) at 6c9ad9c8

New test cases for the bound:

• gemm-block-maximum: every lane of every block sums exactly to 65504.
• gemm-block-overflow: every lane sums to 65536, so every tile is redone.
• gemm-block-exact: sums are exact in F16 within one block but not across two. Widths are 8, 504, 512, 520 and 2056, so tails and a longer block are covered.
• gemm-tile: an overflow covers rows 5–6 × columns 8–9 of the 4×6 tile at (4, 6). Exact integers are compared with tolerance 0.
• Weight rows keep their NaN padding.

Mutants, each built and run on both paths. Every mutant that applies to a path fails at least one check there:

Mutant	Failing checks (FMLAL / conversion)
No dispatch (0005 only)	16 / 16
No block flush (whole k in F16)	1 / 1 (gemm-block-exact)
128-product blocks	1 / 1
Tail block dropped	202 / 202
No redo	14 / 14
Finiteness checked on the tile origin only	4 / 4 (gemm-tile)
Redo at row 0 / column 0	14 / 14 each
Fast path ignores jj	96 / 96
Flush drops the high half (FMLAL / conversion variant)	176 / 176
Block end not clamped to k (over-read)	1 / 1 (attention)

tests/test_llama_cpp_patches.py now uses the dispatch hunk's context, D Cv[RN][RM] = {};, to check exact-source application. The old context, tinyBLAS<8, float16x8_t, is no longer in the patch, and the test fails with it.

Not covered / gates

• The Fix ARM F16 dot accumulation overflow #177 publication gates still apply: run the test on every published CPU module, measure throughput on Android devices (Android builds no llamafile, so 0006 does not reach it), and measure an F16-weight model plus TTS cancel latency on linux_armv8.2_2.
• Zero-length and strided query tensors are not covered, as before.

Sequencing

Branched from a073d08. main was merged in at #178 (f3961b7f) and at #175 (b1ce76c8), with no force-push. The change touches only 0006, the patch README, its tests and docs/native174*. It must land before the manual v0.9.0 release dispatch. Independent audit: ACCEPT at 6c9ad9c8 against b1ce76c8, with no blocking findings. The auditor built six configurations (FMLAL, conversion, ASan+UBSan, GCC 16 conversion, unpatched, first 0006). Results:

• Bit-exact fuzz against a scalar emulation of the kernel: about 26M outputs, 0 nonfinite and 0 mismatches.
• Guard-page over-read probe: no fault.
• Strict apply on v0.7.0, v0.8.0 and v0.9.0.
• Measured perf matches: pp512 0.92 / 0.94 of unpatched.

Non-blocking test gaps go to a follow-up:

• gemm-tile never overflows the tile's last row or column, so a finiteness check that skips them survives.
• Block length is pinned from above only at 67+ products.