Feature Request: opt-in FP8 KV-cache support for long-context MoE inference #280

Summary

Please consider adding an opt-in FP8 key/value (KV) cache for long-context inference.

Quantizing the KV cache can substantially increase the usable context capacity on
consumer GPUs, especially when serving large MoE models whose weights already use
NVFP4/FP8 formats. This would complement the broader quantized-cache request in
#262 with a specific FP8
design point that should preserve more quality than lower-bit cache formats.

Proof of concept

I built and tested an experimental implementation in a public fork:

• Repository: https://github.com/mbetrifork/FreeToken
• Branch: https://github.com/mbetrifork/FreeToken/tree/experimental/fp8-kv-cache
• Commit: mbetrifork@09eb4bf

The proof of concept:

• Adds an opt-in --kv-cache-dtype fp8_e4m3 option.
• Stores K/V values as E4M3 FP8 with independent FP32 dynamic scales.
• Dequantizes inside the Triton attention kernels.
• Includes KV-pool sizing/accounting and configuration validation.
• Includes CPU/configuration tests and optional CUDA kernel tests.

On an 8 GB RTX 2000 Ada laptop GPU, it was used to serve
Qwen3.6-35B-A3B-NVFP4 with a 131K-token KV pool and a coding-agent client.
The regular attention KV data is approximately half the BF16 storage, with a
small scale-table overhead.

Current proof-of-concept limitations

The fork intentionally keeps the first implementation narrow:

• Qwen3.5/Qwen3.6 MoE checkpoints only.
• BF16 compute only.
• Triton attention only.
• Ordinary paged MHA KV storage only.
• Tensor parallel size 1 only.
• NVIDIA compute capability 8.9 (Ada) or newer.
• Qwen's recurrent Gated DeltaNet state remains unquantized.
• Dynamic per-token/per-KV-head scales are used; checkpoint calibration scales
  are not yet supported.

Unsupported combinations fail during configuration validation, and the feature
is disabled by default. The fork is intended as a reproducible demonstration of
the design space; maintainers may prefer a different abstraction or kernel
implementation.

Possible next steps

• Benchmark BF16 versus FP8 KV memory, prefill/decode throughput, and quality.
• Validate FP16 compute and tensor-parallel configurations.
• Extend support to other MHA/GQA model families and attention backends.
• Decide on scale granularity and optional checkpoint calibration data.
• Add a maintained CUDA test/benchmark path for supported GPUs.