[devin-ai-integration]

Make sure to read the contributing guidelines before submitting a PR

Summary

Phase 4 of the Q1_0_g128 AVX2 repack project is doc-only. The runtime dispatcher is already infrastructure-complete via the existing GGML_CPU_ALL_VARIANTS + GGML_BACKEND_DL machinery. The on-disk repack converter is intentionally not added because the equivalent upstream pattern (GGML_TYPE_Q4_0_4_4 / _4_8 / _8_8) was deprecated and removed.

This is the closeout for the multi-phase AVX2 repack work. After this PR, all originally planned phases are either delivered or intentionally re-scoped with documented rationale, and there is no further AVX2-repack work in the queue.

What I expected to do vs. what I found

Expected (from prior session plan): port a runtime dispatcher into cpu.c, register a GGML_TYPE_Q1_0_G128_AVX2 enum, plumb a --repack-q1-g128-avx2 flag in llama-quantize. Estimated 2-3 days.

Actual investigation result:

• The runtime dispatcher already exists. ggml/src/CMakeLists.txt:307-385 registers per-ISA backend variants (ggml-cpu-haswell.so, ggml-cpu-alderlake.so, ggml-cpu-skylakex.so, ggml-cpu-zen4.so, ggml-cpu-sapphirerapids.so, etc.), each compiled with a different ISA flag set. The runtime backend loader inspects the host CPU at process start and picks the most-capable variant the host supports.
• The kernel-level compile-time gating I added in PR ggml-cpu(repack): true 4x4 GEMM for Q1_0_g128 — 1.22× prefill over PR #12 #13 (__AVX2__) and PR ggml-cpu(repack): AVX-VNNI Kernel B for Q1_0_g128 4×4 repack (SDE-validated, no real-hardware perf) #17 (__AVXVNNI__) drops cleanly into this system: each variant gets exactly the right kernel for its ISA, no source-level changes required.
• The runtime gate in repack.cpp:3702-3712 (avx2 && !avx512) registers the 4×4 repack only on AVX2-without-AVX-512 hosts, which correctly prefers the existing single-row AVX-512 vec_dot path on AVX-512 hosts. No code change needed there either.
• The on-disk repack converter pattern (a new GGML_TYPE_* enum + --repack-* flag) was deprecated upstream in PR #10446 for distribution-complexity / maintenance reasons. Adding it on prism purely to mirror the original Phase 4 plan would re-introduce costs upstream already paid down. Decision Add GGML_TYPE_Q2_0 ternary 2-bit quantization (block size 128) #1 of 03-design-brief.md already said "no on-disk GGUF format changes".

So Phase 4's actual work is verify + document, not code.

Verification (this VM, AMD EPYC 7763 / Zen 3)

$ cmake -B build-variants -G Ninja -DCMAKE_BUILD_TYPE=Release \
      -DGGML_CPU_ALL_VARIANTS=ON -DGGML_BACKEND_DL=ON \
      -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=ON
$ cmake --build build-variants -j --target ggml-cpu-haswell ggml-cpu-alderlake
$ objdump -d build-variants/bin/libggml-cpu-haswell.so   | grep -c vpdpbusd
0
$ objdump -d build-variants/bin/libggml-cpu-alderlake.so | grep -c vpdpbusd
506

• haswell variant (AVX2 + FMA + F16C + BMI2) gets Kernel A only — 0 vpdpbusd instructions, matches expected (no __AVXVNNI__ defined in this variant's compile environment).
• alderlake variant (AVX2 + FMA + F16C + BMI2 + AVX_VNNI) gets Kernel B — 506 vpdpbusd instructions inlined into the public dispatchers, matches the SDE-validated PR ggml-cpu(repack): AVX-VNNI Kernel B for Q1_0_g128 4×4 repack (SDE-validated, no real-hardware perf) #17 disassembly.

Per-variant kernel routing table (from the doc):

Variant	__AVXVNNI__	__AVX512F__	Kernel selected at runtime
haswell	undefined	undefined	Kernel A (PR #13)
alderlake	defined	undefined	Kernel B (PR #17)
skylakex / cascadelake / icelake / zen4 / sapphirerapids	undefined	defined	Existing single-row AVX-512 vec_dot

Note on the GCC 11.4 / _mm512_setr_epi8 issue

A full GGML_CPU_ALL_VARIANTS=ON build hits a pre-existing GCC 11.4 break in arch/x86/quants.c:682 (_mm512_setr_epi8 not declared in <immintrin.h> for GCC < 12) blocking the AVX-512 variants. This is unrelated to the repack work — it predates Phase 1 and was already flagged in PR #14's 04-cross-arch-validation.md. The verification above is constrained to the haswell + alderlake variants where this PR's referenced kernels live. Fixing the AVX-512 build break is a separate follow-up tracked in PR #14's doc.

Files changed

• docs/avx2-repack-design/07-phase4-dispatcher.md — new (+253 LOC). Contains:
  • The verification recipe + numbers (above).
  • The full per-variant routing table.
  • The "why no on-disk converter" rationale + upstream-deprecation reference.
  • A status table closing out all 7+1 phases of the AVX2 repack project.
  • A list of optional follow-ups (real-hardware Alder Lake / Zen 4 perf measurement; AVX-512 VNNI 4×4 variant; the _mm512_setr_epi8 fix).

What this closes

After this PR merges, the multi-phase AVX2 repack project is complete as scoped by 03-design-brief.md:

Phase	Status	PR	Net merged
1	Design brief	#10	✓
2	Microbench	#11	✓
3	4×4 GEMV+GEMM	#12	✓
3.5	True 4×4 GEMM	#13	✓
4	Dispatcher	this PR	doc-only
5	Cross-arch	#14	✓
6	cv-guard bench	cv-guard #17	✓
7	AVX-VNNI Kernel B	#17	✓
+	Drop-unpack	#15 (doc-only)	✓

Review & Testing Checklist for Human

• Read docs/avx2-repack-design/07-phase4-dispatcher.md. Sanity-check the per-variant routing table.
• Sanity-check the rationale for not adding the on-disk converter. If you want it added anyway as a prism-specific path, push back and I'll write it as a separate PR — but I'd recommend reading the upstream PR #10446 deprecation discussion first.
• Optional: reproduce the verification on your machine: cmake -B build-variants -G Ninja -DGGML_CPU_ALL_VARIANTS=ON -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF && cmake --build build-variants -j --target ggml-cpu-haswell ggml-cpu-alderlake && objdump -d build-variants/bin/libggml-cpu-{haswell,alderlake}.so | grep -c vpdpbusd.

Notes

• AI authorship disclosure: per AGENTS.md / CONTRIBUTING.md preamble for the prism private fork, this design document was prepared with AI assistance under explicit authorisation from @kennguy3n. Verification numbers (vpdpbusd-count disassembly check) were captured from live build artefacts in this session. Not intended for upstream ggml-org/llama.cpp.

Link to Devin session: https://uney.devinenterprise.com/sessions/576d78b01f9b4389b375033162f7b7b4

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring