Repository navigation

turboquant_kv_cache_updated_v11 - #33

Merged

InfernalDread merged 171 commits into

InfernalDread:turboquant_kv_cache_updated_v11from

Addy-ad:addyad-latest

Apr 22, 2026

Merged

turboquant_kv_cache_updated_v11#33
InfernalDread merged 171 commits into
InfernalDread:turboquant_kv_cache_updated_v11from
Addy-ad:addyad-latest

Commits on Mar 26, 2026
- WIP: add TurboQuant KV cache types (turbo3, turbo4)
  
  TheTom
  and
  claude
  committed
  9b041de
- feat: Metal kernels for TurboQuant KV cache (turbo3, turbo4) #21
  
  TheTom
  and
  claude
  committed
  9f3771a
- feat: full TurboQuant with rotation matrices in Metal kernels #21
  
  TheTom
  and
  claude
  committed
  70a313e
- feat: inline rotation matrices in Metal shader + C round-trip test #21
  
  TheTom
  and
  claude
  committed
  dcd15a1
- fix: remove thread static from Metal dequantize, fix stale code #23
  
  TheTom
  and
  claude
  committed
  793f157
- feat: replace dense 128x128 matvec with Fast Walsh-Hadamard rotation #26
  
  TheTom
  and
  claude
  committed
  d4ee5b4
- docs: detailed speed investigation plan for TurboQuant Metal shader #23
  
  TheTom
  and
  claude
  committed
  8997c00
- docs: log simd_broadcast attempt — no speed improvement #23
  
  TheTom
  and
  claude
  committed
  283441c
- docs: log threadgroup attempt — no speed improvement, rethinking #23
  
  TheTom
  and
  claude
  committed
  aede1bb
- docs: CRITICAL — dequant is NOT the bottleneck, no-op still 2.4 tok/s #23
  
  TheTom
  and
  claude
  committed
  1f1f8f2
- fix: inline turbo-wht.h — was causing CPU fallback, not Metal! #23
  
  TheTom
  and
  claude
  committed
  b73d683
- docs: real Metal benchmarks after #include fix — 8× gap not 35× #23
  
  TheTom
  and
  claude
  committed
  b8410b3
- docs: final investigation summary + upstream tracking #23 #27
  
  TheTom
  and
  claude
  committed
  0c9bada
- docs: upstream competitive intel — pre-rotate-queries is the key #28
  
  TheTom
  and
  claude
  committed
  456574c
- docs: speed ceiling test — 49 tok/s without dequant rotation (4.6× gain) #23
  
  TheTom
  and
  claude
  committed
  290732b
- docs: pre-rotate-queries implementation plan + speed ceiling 49 tok/s
  
  TheTom
  and
  claude
  committed
  9549963
- feat: pre-rotate-queries optimization — 51.4 tok/s (5× speedup) #23
  
  TheTom
  and
  claude
  committed
  d27a4a5
- docs: final investigation summary — 2.4 → 51.4 tok/s journey complete #23
  
  TheTom
  and
  claude
  committed
  1659689
- feat: MSE-only mode — drop QJL, all 3 bits to PolarQuant #23
  
  TheTom
  and
  claude
  committed
  f9841e0
- docs: Change 2 not needed — Q rotation overhead is negligible
  
  TheTom
  and
  claude
  committed
  18f4241
- docs: block size is the bottleneck — q4_0 at block 32 = 100% of q8_0
  
  TheTom
  and
  claude
  committed
  a1230c8
- feat: block size 32 — 77.7 tok/s MoE (91% of q8_0), 17.0 Qwopus (97%) 🎉
  
  TheTom
  and
  claude
  committed
  aed7a94
- fix: TURBO_D=128 independent of QK_TURBO3, file turbo4 bugs #29
  
  TheTom
  and
  claude
  committed
  29786b2
- docs: final investigation log — 77.7 tok/s, 91% of q8_0
  
  TheTom
  and
  claude
  committed
  c3a7afd
- CRITICAL: turbo3 perplexity is 165.6 vs q8_0 6.1 — quality broken #30
  
  TheTom
  and
  claude
  committed
  93c3e63
- CRITICAL: found TWO root causes for PPL=165 #30
  
  TheTom
  and
  claude
  committed
  753b872
- docs: bisect confirms block size innocent, rotation access is the bug #31
  
  TheTom
  and
  claude
  committed
  3d9bcc7
- fix: restore inverse rotation in dequant — PPL 6.19 (1.2% of q8_0) #31 #30
  
  TheTom
  and
  claude
  committed
  cf6270c
- docs: perplexity 6.194 confirmed — 1.4% of q8_0 #30
  
  TheTom
  and
  claude
  committed
  d9b9725
- docs: complete quality benchmark summary + lessons learned #30
  
  TheTom
  and
  claude
  committed
  ded3e94
- perf: fp16 WHT dequant + SIMD cooperative dequant — 45% speedup
  
  TheTom
  and
  claude
  committed
  810b4b2
- chore: move turboquant docs to turboquant_plus repo
  
  TheTom
  and
  claude
  committed
  b097f15
- perf: vectorized half4 WHT butterfly — 31% speedup (1074 → 1411 tok/s)
  
  TheTom
  and
  claude
  committed
  ea35e51
- perf: pre-packed half4 sign arrays — minor speedup (1411 → 1424 tok/s)
  
  TheTom
  and
  claude
  committed
  01fc3cd
- perf: graph-side WHT rotation — 2095 tok/s (0.78x q8_0, was 0.53x)
  
  TheTom
  and
  claude
  committed
  c76b717
- perf: block-32 + graph WHT — 2747 tok/s (1.02x q8_0!!!)
  
  TheTom
  and
  claude
  committed
  316f88f
- feat: layer-adaptive KV cache — q8_0 quality with 80% turbo3 compression
  
  TheTom
  and
  claude
  committed
  ccd1232
- fix: address Codex review on layer-adaptive — thread safety + underflow guard
  
  TheTom
  and
  claude
  committed
  63c8d6a
- wip: context scaling fix — skip unnecessary ggml_cont + 32x32 rotation data
  
  TheTom
  and
  claude
  committed
  48e46bb
- experiment: group-32 rotation FAILED — PPL 7.06 (target 6.19)
  
  TheTom
  and
  claude
  committed
  99489db
- feat: add GGML_OP_TURBO_WHT — custom O(d log d) Walsh-Hadamard Transform
  
  TheTom
  and
  claude
  committed
  8bf235b
- perf: optimized turbo3 dequant — eliminates context scaling regression
  
  TheTom
  and
  claude
  committed
  2157f04
- ci: quality+speed gate script — PPL + context scaling check before push
  
  TheTom
  and
  claude
  committed
  bc8ae28
- perf: fp16 centroid LUT — decode +6-14% at long context (#33)
  
  TheTom
  and
  claude
  committed
  7dd0af1
- perf: float norm broadcast in vec dequant — decode +2-3% over fp16 LUT
  
  TheTom
  and
  claude
  committed
  abc6e88
- fix: add turbo3/turbo4 cache types to llama-bench arg parser
  
  TheTom
  and
  claude
  committed
  05412b3
- experiment: split 2x4-entry constant LUT for M1 decode fix
  
  TheTom
  and
  claude
  committed
  a9ef409
- fix: Metal shader comment accuracy per Codex review
  
  TheTom
  and
  claude
  committed
  9087f91
- cleanup: remove stray diagnostic output files
  TheTom
  committed
  99da38b
- feat: turbo3 norm correction — PPL 6.211 → 6.176 (free quality win)
  
  committed
  02268fc
- fix: auto-enable flash attention for turbo cache types + fix ggml context overflow
  
  TheTom
  and
  claude
  committed
  929b8ba
- experiment: register centroid LUT tested — register spill on Metal
  
  committed
  5811aa5
- feat: CUDA port of TurboQuant3 KV cache compression (RTX 5090 / SM 12.0)
  
  signalnine
  and
  claude
  committed
  b2a5a88
- perf: enable MMA/TILE flash attention for turbo3 — 0.97x q8_0 prefill
  
  signalnine
  and
  claude
  committed
  eb9a589
Commits on Mar 27, 2026
- perf: parallel k_set_rows_turbo3 + optimise KQ/V dequant — +31% decode (71→94 t/s, 98.5% of F16)
  
  signalnine
  and
  claude
  committed
  8b36e47
- experiment: batched byte extraction + explicit bit field pre-extract
  
  committed
  9f23354
- experiment: profiling modes for turbo3 decode bottleneck isolation
  
  TheTom
  and
  claude
  committed
  4b0918e
- fix: turbo4 SET_ROWS corruption, tail-block truncation, constant coupling (Issue #29)
  
  seanrasch
  and
  claude
  committed
  65ed372
- experiment: 4-entry magnitude LUT + branchless sign (XOR trick)
  TheTom
  committed
  830d76b
- experiment: force non-vec FA path for turbo3 (nl=2 vs nl=8)
  
  TheTom
  and
  claude
  committed
  d602c8e
- fix: stack overflow in turbo4 CPU init — 64KB array on worker thread stack
  
  seanrasch
  and
  claude
  committed
  80430e3
- experiment: zero-LUT select chain — 2-level ternary, no constant memory
  TheTom
  committed
  1406691
- feat: auto-detect hardware, use 4-mag LUT on pre-M5 (+38-45% decode)
  
  committed
  3dfd54c
- experiment: 2-pair half2 LUT — only 2 constant addresses per lookup
  TheTom
  committed
  78fac6c
- experiment: deferred norm multiply (batch float4 * norm at end)
  TheTom
  committed
  edfff21
- revert to proven 4-mag + per-element norm (deferred norm was slower)
  
  TheTom
  and
  claude
  committed
  f29b8bb
- experiment: named-register centroid×norm — 4 constant reads upfront, zero divergence, ternary select
  TheTom
  committed
  199d619
- revert to 4-mag LUT (proven best), document all findings
  TheTom
  committed
  39ba0c0
- experiment: inline block processing — bypass template dequant in FA inner loop
  TheTom
  committed
  bc41397
- experiment: inline block WORSE on M2 (-10-15%), reverted to 4-mag
  
  TheTom
  and
  claude
  committed
  ac9e3a7
- experiment: FULLY BRANCHLESS FMA decode — zero ternary, zero memory, zero branch
  TheTom
  committed
  f0c4c79
- final: 12 approaches tested, 4-mag LUT is the hardware limit
  
  TheTom
  and
  claude
  committed
  6063cf4
- experiment: SIMD SHUFFLE magnitude select — cross-lane LUT replacement
  TheTom
  committed
  8888205
- experiment: simd_shuffle 14.7 at 8K — close to 4-mag (15.1) but not better
  
  TheTom
  and
  claude
  committed
  d01a000
- experiment: fused block dot — per-centroid Q accumulation, 4 constant reads total
  TheTom
  committed
  fd9538c
- experiment: fused block dot 8.1 at 8K — worst result, 64 comparisons kill it
  
  TheTom
  and
  claude
  committed
  e9d06b0
- experiment: 4-mag helps M5 at 16K (+2.4%) but hurts at 32K (-7.3%)
  
  TheTom
  and
  claude
  committed
  61a03d9
- experiment: M5 LUT cost grows to 34% at 32K context
  
  TheTom
  and
  claude
  committed
  7673d48
- feat: sparse V dequant — +12% decode at 32K, zero quality loss
  
  TheTom
  and
  claude
  committed
  00a5423
- feat: sparse V dequant — +22% decode at 32K on M5, auto-enabled
  
  TheTom
  and
  claude
  committed
  7d1bd95
- Merge remote-tracking branch 'upstream/feature/turboquant-kv-cache' into feature/turboquant-kv-cache
  seanrasch
  committed
  7b885a5
- fix: VEC flash-attn Q/K stride mismatch in vec_dot_fattn_vec_KQ_turbo3_0
  
  signalnine
  and
  claude
  committed
  4c91451
- Merge pull request #4 from seanrasch/feature/turboquant-kv-cache
  TheTom
  authored
  065ef53
- Revert "Merge pull request #4 from seanrasch/feature/turboquant-kv-cache"
  TheTom
  committed
  a52586e
Commits on Mar 28, 2026
- experiment: dedicated turbo4 SET_ROWS kernel + prefill FA kernels
  
  TheTom
  and
  claude
  committed
  0a6078c
- fix: graceful fallback for turbo3 with non-128-aligned head dims (issue #13)
  
  signalnine
  and
  claude
  committed
  972c76e
- fix: graceful fallback for turbo3 on non-128-aligned head dims (issue #13)
  
  signalnine
  and
  claude
  committed
  9cdb872
- experiment: turbo4 2+1 bit packing — +33% decode, drop QJL
  
  TheTom
  and
  claude
  committed
  f89c4f2
- experiment: direct-extract turbo4 dequant — matches turbo3 speed
  
  TheTom
  and
  claude
  committed
  f284cc0
- feat: 64-element WHT groups + MLA Q rotation fix (issue #13)
  
  signalnine
  and
  claude
  committed
  75e2769
- experiment: 4-bit half-precision centroid LUT for turbo4 vec path
  
  TheTom
  and
  claude
  committed
  eddfff7
- experiment: fix turbo4 struct for 4-bit — Codex-caught OOB bug
  
  TheTom
  and
  claude
  committed
  fef2832
- feat: mixed turbo3/q8_0 KV cache types (-ctk turbo3 -ctv q8_0 and vice versa)
  
  signalnine
  and
  claude
  committed
  d0d37b3
- experiment: 8-mag LUT tested, reverted — direct 16-LUT faster on M5
  
  TheTom
  and
  claude
  committed
  c168011
- experiment: add turbo4_dequant_f16 compute shader (prefill prep)
  
  TheTom
  and
  claude
  committed
  661794f
- fix: implement CPU turbo3 quantize (was a stub that zeroed qs/signs)
  
  signalnine
  and
  claude
  committed
  53f1298
- feat: TURBO4_USE_4BIT ifdef for ABI compatibility
  
  TheTom
  and
  claude
  committed
  d2ca3c9
- feat: complete 4-bit C reference for turbo4 — quantize + dequant
  
  TheTom
  and
  claude
  committed
  3ef4d98
- Merge turbo4 4-bit PolarQuant into main
  
  TheTom
  and
  claude
  committed
  ca25246
- feat: GGML_TYPE_TURBO2_0 — 2-bit TurboQuant KV cache (6.4x compression)
  
  signalnine
  and
  claude
  committed
  da6b0fd
- fix: MLA inverse WHT group_size derived from K (not V) — fixes GLM-4.7
  
  signalnine
  and
  claude
  committed
  00ecbbe
- feat: InnerQ per-channel equalization + turbo2 64-group fallback
  
  signalnine
  and
  claude
  committed
  6fb85a6
- fix: add turbo WHT rotation to ISWA build_attn — fixes Gemma 2
  
  TheTom
  and
  claude
  committed
  4cf7145
- perf: sparse V dequant — skip negligible attention weights in VEC kernel
  
  signalnine
  and
  claude
  committed
  a5efe54
Commits on Mar 29, 2026
- fix: require head_dim % 128 for turbo KV — fall back to q8_0 otherwise
  
  signalnine
  and
  claude
  committed
  4c4511c
- Merge signalnine/feature/turboquant-kv-cache (PR #3) — CUDA port
  
  TheTom
  and
  claude
  committed
  172fc85
- feat: Metal support for turbo2 (2-bit KV cache, 6.4x compression)
  
  TheTom
  and
  claude
  committed
  3380d3c
- feat: asymmetric K/V quant support for Metal flash attention
  
  TheTom
  and
  claude
  committed
  43f7d3d
- feat: zero-pad non-128 heads for full 7-stage WHT (replaces q8_0 fallback)
  
  signalnine
  and
  claude
  committed
  c1d9b34
- perf: CUDA MMA flash attention for D=640 (GLM-4.7 turbo3: 37→192 t/s)
  
  signalnine
  and
  claude
  committed
  d46ac77
- fix: add turbo3/turbo2 cross-type VEC FA instances (issue #25 bug 2)
  
  signalnine
  and
  claude
  committed
  05b7fe3
- feat: CUDA port of turbo4 (4-bit, 3.8x compression) — fixes issue #25 bug 1
  
  signalnine
  and
  claude
  committed
  b90b5e0
- feat: asymmetric K/V support + q8_0 × turbo FA kernel instantiations
  
  TheTom
  and
  claude
  committed
  965a6ca
- fix: turbo4 on GLM-4.7 — context init check accounts for zero-padding (issue #28)
  
  signalnine
  and
  claude
  committed
  ae70214
- Merge branch 'pr-24' into codex/pr24-integration
  TheTom
  committed
  2dd602a
- feat: Boundary V (experimental) — layer-aware V compression
  
  TheTom
  and
  claude
  committed
  70b35c7
- fix: KV state serialization uses padded tensor width (issue #28 follow-up)
  
  signalnine
  and
  claude
  committed
  89d267c
- Merge PR #30: KV state serialization fix for padded tensor widths
  TheTom
  authored
  1b7165f
Commits on Mar 30, 2026
- feat: HIP/ROCm porting for TheTom's turbo3/turbo2 warp-cooperative kernels
  
  Tuklus
  and
  claude
  committed
  58d51a6
- Merge PR #31: HIP/ROCm support for turbo3/turbo2 (7900 XTX)
  TheTom
  authored
  64dd362
- Increase turbo3/turbo2 block size from 32 to 128
  
  TheTom
  and
  claude
  committed
  adac2c6
- fix: CUDA warp-to-block mapping for block_size=128 (turbo3, turbo2)
  Chris Grant
  committed
  aca4594
- Merge pull request #32 from HyperionMS2040/fix/cuda-block-size-128
  TheTom
  authored
  7b75078
Commits on Mar 31, 2026
- WIP: Add PlanarQuant (planar3) KV cache type — 2D Givens rotation
  johndpope
  committed
  b8eccf5
- Add vec flash attention templates for planar3 — decode now works
  
  johndpope
  and
  claude
  committed
  abfb7c8
- Add IsoQuant (iso3) cache type — quaternion 4D rotation, best quality
  
  johndpope
  and
  claude
  committed
  406bfbb
- Add iso4 and planar4 (4-bit) cache types
  
  johndpope
  and
  claude
  committed
  e11d7e2
- planar4: real Givens rotation (not turbo4 alias). PPL 5085 — poor quality.
  
  johndpope
  and
  claude
  committed
  b345da0
- iso4: real quaternion 4D rotation (not turbo4 alias). PPL 74.
  
  johndpope
  and
  claude
  committed
  c301c13
- Use turbo centroids for iso/planar, calibration shows centroids not the issue
  
  johndpope
  and
  claude
  committed
  2c9d286
- Register iso/planar types in llama-context and llama-kv-cache
  
  johndpope
  and
  claude
  committed
  3dee460
- Deferred quantization: allocate K cache as F16 for iso/planar types
  
  johndpope
  and
  claude
  committed
  bcfd846
- Add CUDA F16→quantized conversion kernels for planar3/4 and iso3/4
  
  johndpope
  and
  claude
  committed
  26c90d6
- Double-buffer deferred quantization with CUDA conversion kernels
  
  johndpope
  and
  claude
  committed
  25f896f
- Fix ggml context size for double-buffer, disable conversion (scheduler issue)
  
  johndpope
  and
  claude
  committed
  0971ed5
- Add CUDA set_rows kernels for planar3/iso3/planar4/iso4
  
  johndpope
  and
  claude
  committed
  1ed0453
- Fix k_stream views, disable conversion (missing CUDA FA dequantize)
  
  johndpope
  and
  claude
  committed
  b69ae13
- Add CUDA flash attention dequantize for planar3/iso3/planar4/iso4
  
  johndpope
  and
  claude
  committed
  a75b16f
- COMPRESSION WORKS: 5.1x K-cache + 200 tok/s decode on CUDA
  
  johndpope
  and
  claude
  committed
  9d4ece5
- Guard deferred conversion behind GGML_USE_CUDA
  
  johndpope
  and
  claude
  committed
  e7bde1f
- Add asymmetric FA kernels: q8_0 K + iso3/planar3 V (and reverse)
  
  johndpope
  and
  claude
  committed
  79da661
- Fix FA dispatch: static constants, V=f16 check, asymmetric support
  
  johndpope
  and
  claude
  committed
  b719b2e
- Fix planar3/q8_0 asymmetric: add F16+Q8_0 VEC template for deferred prefill
  
  johndpope
  and
  claude
  committed
  985fd96
- All 8 K/V configs working: real Givens/quaternion rotation for planar4/iso4
  
  johndpope
  and
  claude
  committed
  b83a09f
Commits on Apr 1, 2026
- planar3/turbo3: 5x total compression, PPL 10.19 (vs Tom's 3.5x at 10.14)
  
  johndpope
  and
  claude
  committed
  a730624
- Fix symmetric V=planar3/iso3: add inverse rotation to V dequant
  
  johndpope
  and
  claude
  committed
  6e5a4aa
- Add inverse rotation V dequant for planar4/iso4
  
  johndpope
  and
  claude
  committed
  326f7fb
- Add symmetric planar4/iso4: V dequant, template instances, FA dispatch
  
  johndpope
  and
  claude
  committed
  20efe75
Commits on Apr 9, 2026
Commits on Apr 10, 2026
Commits on Apr 11, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  7ecdecb
Commits on Apr 12, 2026
Commits on Apr 13, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  e550990
Commits on Apr 14, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  8d3756e
Commits on Apr 15, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  4088b9a
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  1624323
Commits on Apr 16, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  9bade9c
- Type check fix
  Addy-ad
  committed
  941b03b
Commits on Apr 17, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  75c8890
Commits on Apr 18, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  dac3a82
Commits on Apr 19, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  d5f8666
Commits on Apr 21, 2026
- For hip.h conflicts, turboquant code was kept
  Addy-ad
  committed
  03c91d8
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  78ebf35
Commits on Apr 22, 2026
- Merge remote-tracking branch 'upstream/master' into addyad-latest
  Addy-ad
  committed
  7565885

Commits

Commits on Mar 26, 2026

Commits on Mar 27, 2026

Commits on Mar 28, 2026

Commits on Mar 29, 2026

Commits on Mar 30, 2026

Commits on Mar 31, 2026