[devin-ai-integration]

Summary

Phase 2 of the AVX2 / AVX-VNNI repack project (Phase 1c design brief
landed in PR #10).

This PR adds a stand-alone microbenchmark harness at
tools/microbench/q1_g128_repack.cpp that:

1. Synthesises N deterministic Q1_0_g128 weight blocks and matching
   Q8_0 activations from a fixed RNG seed, so every run produces
   identical inputs.
2. Implements the four kernel variants from the Phase-1c brief
   (scalar reference, existing single-row AVX2, proposed 4x4 GEMV
   AVX2 [Kernel A], proposed 4x4 GEMV AVX-VNNI [Kernel B]) inside a
   single self-contained translation unit. Nothing in ggml-cpu/ is
   touched; that's Phase 3.
3. Runs a parity check: every kernel must produce FP32 output that
   matches the scalar baseline within 1e-3 relative tolerance per
   row. Failed parity exits before timing.
4. Times each kernel via clock_gettime(CLOCK_MONOTONIC_RAW) over a
   warm-up + steady-state loop, reports median ns / 4 rows so all
   kernels can be compared apples-to-apples (the GEMV does 4 rows in
   one call; the single-row kernels are looped 4×).

Why a separate harness, not a tests/ test?

The harness is a CLI you can iterate on in seconds — change a few
intrinsics, rebuild only microbench-q1-g128-repack, run, see
numbers. Once the kernel intrinsics are validated against a parity
check + show real perf, they get ported into
ggml/src/ggml-cpu/arch/x86/repack.cpp (Phase 3) and the dispatcher

• tensor_traits plumbing follows (Phase 4).

Headline numbers (committed in tools/microbench/baseline-zen3-epyc-7763.txt)

EPYC 7763 / Zen 3 (AVX2 + FMA + F16C, no AVX-512), nb=13312
(~1.7M weights ≈ Bonsai-1.7B-shaped), iters=256, single-threaded:

Kernel	ns / 4 rows	ns / row	vs baseline
scalar reference	34 476 683	8 619 171	0.013×
vec_dot_avx2 (existing)	451 293	112 823	1.00×
gemv_4x4_avx2 (NEW Kernel A)	351 807	87 952	1.28×

Parity passes — every kernel produces bit-identical FP32 output
within FP16 quantisation noise.

Important: actual gain is less than the design brief predicted

The Phase-1c brief estimated 2.5× over the existing AVX2 path on
Zen 3. The measured gain is 1.28×. Root cause: the per-row bit
expansion (broadcast + shuffle + and + cmpeq) is the single most
expensive part of the inner loop, and each row has different
qbits, so this work cannot be shared across rows in this layout.
The savings come only from sharing qy and sum_qy_v across the 4
rows, which is real but smaller than estimated.

This is exactly what Phase 2 was for — the harness produces the data
that gates Phase 3. The 1.28× number changes the conversation:

• Real-world projection on the demo VM: 23 t/s prefill → ~28–30 t/s.
  That's a ~30% improvement, not the 50–60 t/s the brief targeted.
• The Q3_K_M alternative (already documented in cv-guard PR docs(avx2-repack-design): drop-unpack experiment write-up (kernel reverted) #15)
  delivers 41 t/s with zero kernel work.
• AVX-VNNI is the real prize — Kernel B collapses the per-row dot
  to a single vpdpbusd_avx and is expected to push throughput
  close to the AVX-512 fast-path. We can't measure that on Zen 3;
  it needs an Alder Lake P-core or Zen 4 host.

The README in tools/microbench/README.md walks through this
finding in detail and proposes a revised Phase-3-go/no-go criterion.

Build & run

cmake -S . -B build -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF
cmake --build build -j --target microbench-q1-g128-repack

./build/bin/microbench-q1-g128-repack --blocks 13312 --iters 256

For AVX-VNNI hosts:

cmake -S . -B build -DLLAMA_MICROBENCH_AVX_VNNI=ON

Review & Testing Checklist for Human

This is a measurement PR, not a kernel-change PR. The right thing
to look at is the methodology + the actual numbers, not the
intrinsics. Marking yellow because the numbers contradict the brief
and we should re-decide the Phase-3 plan before writing kernel code
in ggml-cpu/.

• Confirm the block_q1_0_g128x4 packed layout in
  q1_g128_repack.cpp:97 matches the design brief's
  "[d0 d1 d2 d3] [r0_sb0 r1_sb0 r2_sb0 r3_sb0] …" memo (Phase-1c §"Quant byte interleave granularity").
• Sanity check the AVX2 kernel at q1_g128_repack.cpp:343
  against the Phase-1c pseudocode in docs/avx2-repack-design/03-design-brief.md
  §GEMV-A. The harness matches the post-fix vectorised version
  (no horizontal sums in the inner loop, 2*partial − sum_qy
  in __m256i, hsum_float_8 only at row close).
• Run the harness on a different x86 host (any AVX2 host) and
  confirm the parity check still passes and the speedup ratio is
  in the same ballpark. The deterministic seed means the FP outputs
  must be byte-identical across runs of the same --seed.

Test Plan

cmake --build build -j --target microbench-q1-g128-repack
./build/bin/microbench-q1-g128-repack --blocks 4096 --iters 256
# parity OK, gemv_4x4_avx2 ~1.24× faster than vec_dot_avx2

./build/bin/microbench-q1-g128-repack --blocks 13312 --iters 256
# parity OK, ~1.28× faster (committed at tools/microbench/baseline-zen3-epyc-7763.txt)

No tests in tests/ are added because the kernel still lives in
the harness, not in ggml-cpu. Phase 3 will add a real
test-quantize-fns-style parity test against the
tensor_traits dispatcher.

Decision needed before Phase 3

Three options based on the new data:

1. Ship the 1.28× win as Phase 3 (port Kernel A into
   ggml-cpu/arch/x86/repack.cpp + dispatcher). Modest but real,
   transparent to upstream model artifacts.
2. Pivot to AVX-VNNI only, document Zen 3 as "use Q3_K_M" and
   target the win on Alder Lake / Zen 4 hosts where Kernel B
   should approach AVX-512 perf.
3. Keep iterating in this harness before integrating: try
   sum_qy precomputation in block_q8_0_sum, or 8-row
   interleave, or shared bit expansion via 64-bit shuffle pairs.
   Probably gets us to 1.5×; whether that's worth another week
   depends on whether (1) is a hard "no".

I have no preference between (1) and (2); (3) is honest engineering
but the diminishing returns are real.

Link to Devin session: https://uney.devinenterprise.com/sessions/576d78b01f9b4389b375033162f7b7b4
Requested by: @kennguy3n

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring