[devin-ai-integration]

Make sure to read the contributing guidelines before submitting a PR

Summary

Adds AVX-VNNI variants of the 4-row GEMV and true 4×4 GEMM kernels that PR #13 introduced for AVX2-only hosts. Same block_q1_0_g128x4 repack format (no on-disk format change); the dispatcher in ggml_gemv_q1_0_g128_4x4_q8_0 and ggml_gemm_q1_0_g128_4x4_q8_0 prefers Kernel B over Kernel A whenever __AVXVNNI__ is defined.

This is Phase 7 of the multi-phase Q1_0_g128 AVX2 repack project (Phase 1 = PR #10, Phase 2 = PR #11, Phase 3 = PR #12 + #13, Phase 5 = PR #14, drop-unpack experiment = PR #15). It lands the AVX-VNNI Kernel B that was drafted in the Phase 2 microbench (PR #11) and deferred at the end of the Phase 3 follow-up plan.

Inner-loop op-count change

Per (r_a, r_w) per sub-block, the inner body collapses from:

const __m256i masked  = _mm256_blendv_epi8(zero, qy, mask_w[r_w]);
const __m256i partial = _mm256_madd_epi16(_mm256_maddubs_epi16(ones_b, masked), ones_w);
const __m256i row_dot = _mm256_sub_epi32(_mm256_slli_epi32(partial, 1), sum_qy_v);

(5 vector ops + 2 shared per qy load for sum_qy_v) into:

const __m256i signed_qy = _mm256_blendv_epi8(neg_qy, qy, mask_w[r_w]);
const __m256i int_acc   = _mm256_dpbusd_avx_epi32(zero, ones_b, signed_qy);

(2 vector ops + 1 shared per qy load for neg_qy).

Per-sub-block GEMM inner-loop op count: 22 → 9 ops (Kernel A → Kernel B). Register-pressure profile (16 __m256 accumulators) and memory-traffic profile (one block_q1_0_g128x4 load per y-group, one block_q8_0x4 unpack) are both unchanged.

Validation

Default Zen 3 / AVX2-only build (host)

No code change for AVX2-only callers: Kernel B is omitted at compile time when __AVXVNNI__ is undefined, and the dispatcher falls back to Kernel A from PR #13.

$ cd build && ctest -j4
100% tests passed, 0 tests failed out of 50

tests/test-q1-g128-repack-parity passes nr=1 / nr=4 / nr=8 with diff = 0.000000e+00 on every element — same as before this PR.

-DGGML_AVX_VNNI=ON build under Intel SDE 9.48 -adl

$ cmake -B build-vnni -G Ninja -DCMAKE_BUILD_TYPE=Release \
      -DGGML_NATIVE=OFF \
      -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON \
      -DGGML_BMI2=ON -DGGML_AVX_VNNI=ON \
      -DLLAMA_BUILD_TESTS=ON
$ cmake --build build-vnni -j --target test-q1-g128-repack-parity llama-cli
$ sde64 -adl -- ./build-vnni/bin/test-q1-g128-repack-parity
PASS: GEMV (nr=1) + GEMM (nr=4) + GEMM (nr=8) all match single-row vec_dot within FP16 tolerance

Full SDE output committed at docs/avx2-repack-design/parity-alderlake-sde-kernel-b.txt with the build/run recipe in the file header.

End-to-end smoke under SDE

$ sde64 -adl -- ./build-vnni/bin/llama-cli -m Bonsai-1.7B.gguf \
      -p "The capital of France is" -n 8 -t 4 --no-warmup
The capital of France is **Paris**
[ Prompt: 1.3 t/s | Generation: 1.2 t/s ]

Token output is correct under SDE. The 1.3 t/s is SDE's dynamic-translation overhead, not a real measurement (SDE translates vpdpbusd_avx into host-native sequences and timing under emulation is meaningless).

Disassembly cross-check

$ objdump -d build-vnni/bin/libggml-cpu.so | grep -c vpdpbusd
506

Both GEMV + GEMM 4-rows variants are inlined into the public dispatchers by GCC 11.4 at -O3. No vpdpbusd instructions in the default Zen 3 build (kernel correctly excluded at compile time).

What this change does NOT include

• Real-hardware perf measurement. The demo VM is AMD EPYC 7763 (Zen 3, no native AVX-VNNI). Whoever owns Alder Lake / Sapphire Rapids / Zen 4 can bench against the PR ggml-cpu(repack): true 4x4 GEMM for Q1_0_g128 — 1.22× prefill over PR #12 #13 Zen 3 baseline of 30.87 ± 0.15 t/s pp512 and commit the result next to phase3-end-to-end-zen3-epyc-7763.txt. The PR description for the Phase-2 microbench (PR tools/microbench: Phase 2 harness for Q1_0_g128 AVX2/AVX-VNNI repack #11) projected ~70 t/s pp512 on Alder Lake; that number is unverified on real hardware and may overstate. The doc at docs/avx2-repack-design/06-avx-vnni-kernel-b.md covers the methodology.
• AVX-512 VNNI variant. Targets __AVX512VNNI__ rather than __AVXVNNI__. AVX-512 hosts already use the existing single-row vec_dot AVX-512 path; a 4×4 AVX-512-VNNI kernel would be a follow-up. Note the cross-arch validation (PR docs(avx2-repack-design): Phase 5 cross-arch validation of PR #13 #14) flagged a pre-existing GCC 11.4 build break in Q2_0 AVX-512 unrelated to this work.
• Runtime CPU detection in the dispatcher. Currently compile-time only via __AVXVNNI__. Phase 4 (the on-disk repack converter + --repack-q1-g128-avx2 flag) will likely add runtime detection alongside.

Files changed (this PR's net diff)

• ggml/src/ggml-cpu/arch/x86/repack.cpp — Kernel B GEMV + GEMM 4-rows + dispatcher branch (+234 LOC, gated #if defined(__AVXVNNI__)).
• docs/avx2-repack-design/06-avx-vnni-kernel-b.md — design + op-count table + bench recipe + projected uplift discussion (+204 LOC, new).
• docs/avx2-repack-design/parity-alderlake-sde-kernel-b.txt — verbatim SDE Alder Lake parity output (+42 LOC, new).

Review & Testing Checklist for Human

• Read docs/avx2-repack-design/06-avx-vnni-kernel-b.md. Sanity-check the inner-loop op-count diff (5 ops → 2 ops per (r_a, r_w) per sub-block).
• Verify the default Zen 3 / AVX2-only build is unchanged: cmake --build build && ./build/bin/test-q1-g128-repack-parity should still produce diff = 0 on every element. objdump -d build/bin/libggml-cpu.so | grep -c vpdpbusd should be 0.
• Optional: rebuild with -DGGML_AVX_VNNI=ON and re-run sde64 -adl -- ./build-vnni/bin/test-q1-g128-repack-parity to reproduce the parity output committed in this PR.
• Optional: if you have an Alder Lake / Zen 4 host, run llama-bench -t 4 -p 256,512,1024 -n 32 -r 3 and commit the numbers as phase7-end-to-end-<host>.txt next to the Phase 3 numbers.

Notes

• AI authorship disclosure: per AGENTS.md / CONTRIBUTING.md (with the prism private-fork preamble added in PR docs: clarify AI policy for prism private fork (preamble note in AGENTS.md / CONTRIBUTING.md) #16), this is the prism private fork. The kernel port from PR tools/microbench: Phase 2 harness for Q1_0_g128 AVX2/AVX-VNNI repack #11's Phase 2 microbench draft into arch/x86/repack.cpp, the dispatcher wiring, and the design doc are AI-assisted under explicit authorisation from @kennguy3n. Parity validation under SDE was run on the live build artefact; the captured output is committed verbatim. Not intended for upstream ggml-org/llama.cpp.
• This closes the AVX-VNNI Kernel B item from the post-PR-ggml-cpu(repack): true 4x4 GEMM for Q1_0_g128 — 1.22× prefill over PR #12 #13 follow-up list. Remaining queued item: Phase 4 (full dispatcher + on-disk repack converter + --repack-q1-g128-avx2 flag in llama-quantize).

Link to Devin session: https://uney.devinenterprise.com/sessions/576d78b01f9b4389b375033162f7b7b4

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.