[devin-ai-integration]

Summary

Phase 1 (research / design) of a multi-phase project to give Q1_0_g128 a meaningful AVX2 throughput uplift on hosts that lack AVX-512 (Zen 3, Skylake-X-non-VNNI, etc.) without changing the on-disk GGUF format and without touching the existing AVX-512 / CUDA / Metal / Vulkan / NEON fast paths.

This PR adds only three design documents under docs/avx2-repack-design/. No source code is changed in this PR. It is a sign-off PR — once the design is approved, follow-up PRs implement the microbenchmark harness (Phase 2), the kernel intrinsics (Phase 3), and the dispatcher / repack plumbing (Phase 4).

Why

On the cv-guard demo VM (AMD EPYC 7763, Zen 3, AVX2 only), Bonsai-1.7B.gguf (Q1_0_g128, 248 MB) currently prefills at 23 t/s, giving a 38 s P50 per decide() call. Investigation showed the AVX2 path is already enabled and is correctly executing 256-bit AVX2 instructions — the perf gap vs AVX-512 is real and fundamental to the existing AVX2 inner loop (~10 ops per 32 weights vs ~3 ops on AVX-512 VNNI). See docs/avx2-repack-design/02-q1-layout.md for the instruction-count breakdown.

Mainline ggml already solves this exact pattern for Q4_0 via Q4_0_4_4 / 4_8 / 8_8 runtime repacking: weights are physically re-laid-out at tensor-allocation time so a single SIMD load services 4 or 8 output rows in parallel. The on-disk GGUF format stays identical; only the in-memory layout changes. This PR proposes mirroring that exact mechanism for Q1_0_g128.

What's in this PR

• docs/avx2-repack-design/01-survey.md — annotated walkthrough of the existing mainline Q4_0 repack pattern (block templates, make_block_* helpers, repack converters, GEMV entry points, dispatcher, backend buffer registration), with file:line references into this fork. The new Q1_0_g128 repack will mirror this template exactly.
• docs/avx2-repack-design/02-q1-layout.md — inventory of the prism block_q1_0_g128 layout (18 B per block: f16 delta + 16 packed weight bits), and an instruction-count breakdown of the existing AVX-512 fast path vs the AVX2 fallback. Lists four optimisation levers ranked by ease and ROI.
• docs/avx2-repack-design/03-design-brief.md — concrete proposal: block_q1_0_g128x4 layout (4 deltas + 4 × 16 packed bytes interleaved at 4-byte granularity = 72 B), AVX2-only and AVX-VNNI GEMV kernels, dispatcher gate, test plan, microbenchmark plan, 6-phase implementation schedule, and a yes/no sign-off checklist at the bottom.

What this PR is not

• No C/C++ kernel code (Phase 3).
• No ggml_type enum changes, no ggml.h ABI changes, no on-disk GGUF format changes — ever. The repack happens at tensor-allocation time only.
• No microbenchmark harness yet (Phase 2 — explicitly deferred until the design is signed off).
• No changes to Bonsai-1.7B.gguf or any other model artifact.
• No changes to the existing AVX-512 / CUDA / Metal / Vulkan / NEON / scalar paths.

Expected outcome (once the full project lands)

Path	Effort	Realistic prefill (Zen 3)	Memory cap
Status quo Q1_0_g128 AVX2	done	23 t/s	248 MB
Repacked Q1_0_g128 AVX2	~5–7 days	~50–60 t/s (est.)	248 MB
Repacked Q1_0_g128 AVX-VNNI	~5–7 days	~70 t/s on Alder Lake / Zen 4	248 MB
Existing AVX-512 fast path	unchanged	~70 t/s	248 MB

Pre-commit fix (precursor commit on this branch)

The pre-commit config pinned flake8-no-print as an additional dependency for flake8 7.0.0, but flake8-no-print 0.1.x hard-pins flake8==4.0.1 — an unresolvable dependency conflict that prevented pre-commit from installing the hook environment, so every git commit on this repo was blocked. This branch includes a one-line precursor commit dropping that additional dependency. flake8 7.0.0 still runs the default pyflakes/pycodestyle checks. If we want a no-print rule back later, we can re-introduce it via a flake8 plugin compatible with flake8 ≥ 7.

Sign-off

Per the checklist in 03-design-brief.md, this design is gated on yes/no answers to:

1. Layout block_q1_0_g128x4 (4 deltas + 4 × 16 quant bytes, 4-byte interleave, 72 B total) — OK?
2. sum(qy) computed on the fly inside the kernel (not stored in the activation block) — OK?
3. Two kernel branches (AVX2-only + AVX-VNNI), same packed layout, dispatched on capability — OK?
4. AVX-512 hosts use the existing fast path (no GEMV repack on AVX-512) — OK?
5. No on-disk GGUF format changes, no ggml_type enum changes, no ggml.h ABI changes — OK?
6. Test plan (parity vs existing vec_dot + e2e log-prob diff + generic fallback) — OK?
7. Microbenchmark harness (Phase 2) lands first before any new kernel is written — OK?

A "yes" on all 7 boxes triggers Phase 2 (microbench harness PR).

Link to Devin session: https://uney.devinenterprise.com/sessions/576d78b01f9b4389b375033162f7b7b4
Requested by: @kennguy3n

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring