Repository navigation
docs(avx2-repack-design): Phase 5 cross-arch validation of PR #13 - #14
Merged
kennguy3n merged 1 commit intoMay 1, 2026
Merged
Conversation
Validates the Q1_0_g128 4x4 AVX2 runtime repack across four configurations on AMD EPYC 7763 / GCC 11.4: 1. x86 AVX2 (PR #13 default): 50/50 ctest pass, parity diff = 0 (already established in PR #13). 2. x86 no-AVX no-AVX2 build: parity test passes, diffs <= 1e-6 FP32 rounding noise. 3. x86 AVX-512 build: pre-existing breakage in quants.c:682 from prism commit 96c3636 (Q2_0 AVX-512 path uses _mm512_setr_epi8 which requires GCC 12+). Independent of the AVX2 repack work; filed as an open issue. 4. AArch64 NEON cross-build + qemu execution: parity test passes, diffs <= 1.2e-6 FP32 rounding noise. Generic fallback in repack.cpp is exercised; arch-fallback.h aliases the *_generic symbols to the actual functions on aarch64. Conclusion: PR #13's runtime repack is safe across all checked architectures. The dispatcher gate (avx2 && !avx512) keeps the runtime repack opt-in only on the targeted hosts; non-x86 / AVX-512 / no-AVX2 hosts fall back to the existing single-row vec_dot path. AI-authored: this doc was drafted with Devin assistance. The work runs on the prism (private) fork, not upstream llama.cpp.
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This was referenced May 1, 2026
kennguy3n
added a commit
that referenced
this pull request
May 1, 2026
…ispatcher * docs(avx2-repack-design): Phase 4 dispatcher closeout (no new code) Phase 4 was originally scoped to add three pieces: 1. Full dispatcher (runtime ISA selection) 2. On-disk repack converter (--repack-q1-g128-avx2 flag in llama-quantize, new GGML_TYPE_Q1_0_G128_AVX2 enum) 3. Cross-arch fallback testing Item (3) was completed in PR #14. This doc closes out (1) and (2). (1) is already infrastructure-complete via GGML_CPU_ALL_VARIANTS=ON + GGML_BACKEND_DL=ON. The build produces per-ISA shared libraries (ggml-cpu-haswell.so, ggml-cpu-alderlake.so, ggml-cpu-skylakex.so, etc.) and the runtime backend loader picks the most capable variant the host CPU supports. Verified via disassembly: $ objdump -d build-variants/bin/libggml-cpu-haswell.so | grep -c vpdpbusd 0 $ objdump -d build-variants/bin/libggml-cpu-alderlake.so | grep -c vpdpbusd 506 Kernel A from PR #13 lands in the haswell variant (no vpdpbusd); Kernel B from PR #17 lands in the alderlake variant (506 vpdpbusd inlined into the public dispatchers). The runtime gate at repack.cpp:3702 (avx2 && !avx512) routes correctly per variant. (2) is intentionally not implemented. The equivalent upstream pattern (GGML_TYPE_Q4_0_4_4 / _4_8 / _8_8) was deprecated and removed upstream because: (a) on-disk size is identical to the unrepacked format -- no storage win; (b) the runtime repack is ~1 ms per tensor, negligible vs model load; (c) each enum entry forces every backend (cuda, metal, vk, ...) to add a code path for a CPU-only optimisation. Adopting the deprecated pattern on prism purely to mirror the original Phase 4 plan would re-introduce the costs upstream already paid down. Decision #1 of 03-design-brief.md ("no on-disk GGUF format changes") was right at the start. This closes the AVX2 repack project. Status table at the bottom of the doc summarises all 7+1 phases / PRs. Note: a full GGML_CPU_ALL_VARIANTS build hits a pre-existing GCC 11.4 break in arch/x86/quants.c:682 (_mm512_setr_epi8 not declared in <immintrin.h> for GCC < 12) that blocks the AVX-512 variants (skylakex / cascadelake / icelake / cannonlake / cooperlake / zen4 / sapphirerapids). This is unrelated to the repack work and already tracked in 04-cross-arch-validation.md. The verification above was constrained to the haswell + alderlake variants (where my work lands); the rest of the dispatcher infrastructure is upstream code that has been in production for years. * docs(avx2-repack-design): fix variant→kernel mapping in recommended-build paragraph Devin Review caught that lines 201-205 of 07-phase4-dispatcher.md contradicted the routing table earlier in the same doc (lines 58-65 and 86-94). Skylake-X has AVX-512, so it routes to the existing single-row AVX-512 vec_dot, not Kernel A. Zen 4 also has AVX-512, so it also routes to vec_dot, not Kernel B. Corrected mapping: Kernel A → Zen 3 (haswell variant) Kernel B → Alder Lake (alderlake variant) vec_dot → Skylake-X / Cascade Lake / Zen 4 / Sapphire Rapids This matches the per-variant routing table at the top of the doc and the analysis at lines 86-94. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make sure to read the contributing guidelines before submitting a PR
Summary
Docs-only PR. Validates PR #13's
Q1_0_g1284×4 AVX2 runtime repack across four build configurations on AMD EPYC 7763 / GCC 11.4. New file:docs/avx2-repack-design/04-cross-arch-validation.md.Configurations tested
repack.cppOpen issue: AVX-512 build broken on GCC 11.4
The AVX-512 build fails with:
This is in your prism commit
96c3636c(Q2_0 AVX-512 vec_dot, April 30 — pre-dates this AVX2 repack project)._mm512_setr_epi8was only added in GCC 12; the demo VM has GCC 11.4. Independent of the runtime-repack work — none of the new repack code references_mm512_setr_epi8.Fix candidates (out of scope for this PR):
_mm512_setr_epi8call site with a_mm512_loadu_si512from a 64-bytestatic const uint8_t shuf[64]array.The dispatcher gate is independent of this issue: an AVX-512 host (when the build succeeds elsewhere) hits
repack.cpp:3708, which returnsnullptrforQ1_0_g128, keeping the existing single-rowvec_dotpath active.Conclusion
PR #13's runtime repack is safe across all checked architectures. Generic fallback (used outside the AVX2-no-AVX-512 gate) produces FP32 results bit-equivalent to the AVX2 fast path within FP16 tolerance.
AI authorship disclosure
This doc was drafted with Devin assistance. The work runs on the prism (private) fork — not upstream llama.cpp. Per the user's prior session sign-off, AI-assisted implementation is permitted on the private fork.
Link to Devin session: https://uney.devinenterprise.com/sessions/576d78b01f9b4389b375033162f7b7b4
Requested by: @kennguy3n