Repository navigation
[FEATURE REQUEST] - Turbo Quant #2075
Description
It is looking like a new method for handling KV Cache has arrived, one that improves over Q8 KV, both in speed and reducing the memory footprint. IMO, it seems likely to become a feature in LlamaCPP, as I have seen several people use Turbo Quant.
The thread below details the subject.
TurboQuant - Extreme KV Cache Quantization
I figured that I should draw attention to this, as it seems very promising.
Activity
JohnLoveJoy commentedon Mar 26, 2026
It will likely be implemented in GGML, and eventually the optimizations will end up in Kobold, not the other way around.
SabinStargem commentedon Mar 26, 2026
Presumably, some work would still be needed on KoboldCPP's end. The GUI, documentation, ect.
birchamp commentedon Mar 27, 2026
This guy's got instructions for a llama.cpp port with turboquant in the readme. https://github.com/TheTom/turboquant_plus
henk717 commentedon Mar 27, 2026
This is not the kind of thing we typically add on our side, that would risk breaking compatibility with future versions of llamacpp if we did.
TurboQuant is having a lot of eyeballs on it upstream like you said so I am sure that even if they don't add it as TurboQuant in the end the improvements will make its way to us trough the llamacpp ecosystem itself.
One to watch is this one : ggml-org#21038 unlike the vibe coded PR's which won't land this is ggerganov himself implementing a bit of this for every quant type rather than just the new ones. And that kind of change we get automatically when we update llamacpp.
henk717 commentedon Apr 4, 2026
Considering this is purely on llamacpp's side and they added something that was inspired by turboquant ill close this. If they make further additions we should obtain those automatically also.