Skip to content

Add FP8 KV-cache support - #28392

Open
ORippler wants to merge 24 commits into
masterfrom
osimons/ocp_fp8_kv
Open

Add FP8 KV-cache support#28392
ORippler wants to merge 24 commits into
masterfrom
osimons/ocp_fp8_kv

Conversation

@ORippler

@ORippler ORippler commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Overview

Add support for OCP-FP8-based KV-caches. Run-time, static per-tensor scales are supported, but optional.

Additional information

https://huggingface.co/nvidia/Mistral-Medium-3.5-128B-NVFP4 is an example for a model with non-unit-scaled FP8-KV-cache

  • Used graph-level scaling/multiplication instead of modifying ggml_flash_attn_ext. This matches the ownership-model we have in place for NVFP4/FP8, and makes it easier to share FP8 logic across FA and standard attention layers.
  • Added E2E tests for:
    • conversion + loading
    • context shifts
    • general correctness of FP8-scaled KV caches

would love to get input from @ggml-org/ci if our CI can handle this.

Requirements

@github-actions github-actions Bot added testing Everything test related examples conversion labels Sep 4, 2026
@ORippler
ORippler force-pushed the osimons/ocp_fp8_kv branch 2 times, most recently from 51f6a0f to b5f3dfd Compare October 1, 2026 13:55
@ORippler
ORippler force-pushed the osimons/ocp_fp8_kv branch from c802335 to 4602e64 Compare October 1, 2026 15:59
@ORippler
ORippler force-pushed the osimons/ocp_fp8_kv branch from 4602e64 to 10e80e6 Compare October 9, 2026 19:14
@ORippler ORippler mentioned this pull request Oct 9, 2026
Merged
@ORippler
ORippler marked this pull request as ready for review October 13, 2026 15:35
@ORippler
ORippler requested review from a team, CISC and ggerganov as code owners October 13, 2026 15:35
pwilkin
pwilkin reviewed Oct 13, 2026
View reviewed changes
Comment thread src/llama-graph.cpp Resolved
am17an
am17an reviewed Nov 6, 2026
View reviewed changes
Comment thread src/llama-context.cpp Resolved
@ORippler

ORippler commented Nov 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

ggerganov
ggerganov reviewed Nov 9, 2026
View reviewed changes
Comment thread src/llama-graph.cpp Resolved
Comment thread src/llama-model.h Resolved
@ORippler
Add FP8 KV-cache support
2809b37
@ORippler
Add tests for scaled, FP8 KV-cache to test-backend-ops
71f4652
@ORippler
Fix get_*_scale mapping and dequnatize FP8 also before rope shift
8e02eb9
@ORippler
Normalizes cache-scale names independently of projection layout.
6dc7807 
While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless
@ORippler
Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs
5a377e5
@ORippler
Update comment wording
9950a40
@ORippler
Move E2E FP8 KV cache tests to test-llama-archs
c836130 
They belong there as they are not just a test for ggml's backend ops
@ORippler
Increase support surface, remove seed
f54ef5c
@ORippler
make scaled kv generic
8e1e243
@ORippler
Fix CI failings for test-kv-cache
d35e68a 
https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392
@ORippler
CI: execute kv-cache-scale tests
e9bebbe
@ORippler
exclude fp8-kv tests from OpenVINO
b57c5a7
@ORippler
Fix linting errors of test_kv_cache_scales
612cc29
@ORippler
Disable kv-ffload=false + multi-GPU tests for FP8
c17e30c 
See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413
@ORippler
Adjust test label for test-kv-cache.cpp
06be8c3
@ORippler
Fix conflict-resolution mistakes in test-kv-cache.cpp
2313af1 
https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392
@ORippler
Move KV tests to test-fp8.cpp
a8ce8c8 
#27512 (comment)
@ORippler
Remove tmp file for test-kv-cache-scales.py on Windows as well
8847745
@ORippler
Fix fp8 kv cache test by population kv_source_impl
b8da77e
@ORippler
Exclude models that consume KV cache directly
43d6929
@ORippler
Remove obsolete comment
a66395d
@ORippler
Allow BF16 scales as well for KV cache
430e514
@ORippler
Rename *_cache_scale to *_cache_s
d9ac655 
#28392 (comment)
@ORippler
Move is_type_scaled() to llama-kv-cache.cpp
9ebe7f3

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes
@am17an am17an am17an approved these changes
@pwilkin pwilkin pwilkin approved these changes
@CISC CISC Awaiting requested review from CISC CISC is a code owner
@0cc4m 0cc4m Awaiting requested review from 0cc4m

Assignees

@ggerganov ggerganov

Labels

conversion devops improvements to build systems and github actions examples testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ORippler @ggerganov @am17an @pwilkin @0cc4m