Repository navigation
Conversation
The activation is pre-permuted once per token into the 2-bit codes' plane order (ActQ::qp/psum/pscale), so rows need no unpack interleave: four planes summed in int16, one madd, the -1 code offset subtracted exactly in the integers, one FMA per two blocks per token. 2.3-2.8x faster per core than the b256 kernel, closer to exact (rms rel 1.0e-7 vs 2.7e-7), width-invariant. STRATA_Q2_LEGACY=1 selects the previous kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Down rows are only 10 blocks, so the per-row reduction was a large share; four rows now share one hadd tree. The single-row reduction uses the same tree, so a row's bits do not depend on how rows are split into tasks. Down rows +6-7% per core. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Measured on a Haswell-EP Xeon with slow DDR3, where the PR said end-to-end was not measured on such a machine. PC: Xeon E5-2673 v3 (12 cores, AVX2, no AVX-512), 3 x 32 GB DDR3-1866 LRDIMM (the first 64 GB interleaved over two Kernel alone (a small harness calling
On this CPU the current kernel is compute-bound from two tokens per expert on (L3 and DRAM give the same rate); with Numerics: against a double-precision evaluation of the same formula (d · Σ(code−1)·q·scale, random blocks and End to end (server, 4 prompts: Chinese essay, code edit, translation, explanation; greedy, 512 tokens, thinking
+4.3%; all 8 prompt pairs faster (essay 47.0/47.0 → 47.9/47.5, code edit 48.5/48.1 → 50.5/50.6, translation |
Il kernel Q2_0 AVX2 di default spacchetta i codici con 8 operazioni invece di ~17 (stessi vettori lo/hi byte per byte), moltiplica d per la scala in vettore e lavora 4 righe insieme sui down corti: ogni operazione in virgola mobile e' la stessa, nello stesso ordine. STRATA_Q2_LEGACY=1 tiene il kernel precedente. Differenza dalla PR upstream Niko1221#706, che fonde le scale di due blocchi e cambia l'arrotondamento (KL 0,0008-0,0015 a posizionamento fisso). Verifica: q2_exact_parity 504 casi, 570.634 uscite identiche bit per bit al kernel precedente; pool_quant_parity 64 casi sul GGUF reale; teacher forcing a posizionamento fisso identico bit per bit su code ed explain. Un thread: 1,11-1,45x; pool reale (7 worker): down Q2_0 -4%, gate/up Q2_0 -8%, tutto Q2_0 -17% per strato. A 131k nel rumore (la CPU calcola ~2 esperti per strato). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…T1=1); the default keeps 0.1.39's rounding Items 1-2 of #863 (gathered decode per core, AVX-VNNI rows) are bitwise and stay. Item 3 changes a lone token's IQ3_S rounding on Intel hybrid CPUs, so it needs the switch. The #706 bit-plane path stays in front of the VNNI/plain rows. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
In 0.1.40, opt-in: STRATA_Q2_BITPLANE=1 selects the bit-plane kernel (off by default, the activation image is only built when it is on). New q2_bitplane_parity test: against the legacy kernel within tolerance, and bitwise for a token alone vs in a window vs split row ranges. |
…T1=1); the default keeps 0.1.39's rounding Items 1-2 of Niko1221#863 (gathered decode per core, AVX-VNNI rows) are bitwise and stay. Item 3 changes a lone token's IQ3_S rounding on Intel hybrid CPUs, so it needs the switch. The Niko1221#706 bit-plane path stays in front of the VNNI/plain rows.
…T1=1); the default keeps 0.1.39's rounding Items 1-2 of Niko1221#863 (gathered decode per core, AVX-VNNI rows) are bitwise and stay. Item 3 changes a lone token's IQ3_S rounding on Intel hybrid CPUs, so it needs the switch. The Niko1221#706 bit-plane path stays in front of the VNNI/plain rows.
3 participants
A new AVX2 row kernel for Q2_0 experts on CPUs without AVX-512, the default path in
q2_0_gguf_rows_multi_avx2.Single thread, weights in L2: 2.3–2.8× faster at every token count from 1 to 4.