Quant: OCP FP8 E4M3 support by ORippler · Pull Request #27512 · ggml-org/llama.cpp

ORippler · 2026-08-21T16:52:21Z

Overview

Add OCP FP8 (E4M3) in saturating mode to ggml.

Additional information

E4M3 is the format most commonly used for inference, and E5M2 is most often used for training.
Same inference focus is also why non-saturating mode/NaN preservation is not implemented.
Other PRs in this stack will add
- KV cache (Add FP8 KV-cache support #28392)
- CUDA (CUDA: Add OCP FP8 support #28413)
- more robust handling of scaled tensors to ggml (ggml: support quant scales for fp8 and nvfp4 #28898)
- VULKAN support (vulkan: add fp8 and scaled matmul support #29791)

This stacked PR chain does not add plumbing of scales ala ORippler#2, which would have to be done eventually (#28898 will make misuse of scaled types error at least)

I'll mark each of them as ready when I have cleaned them up (functionally everything is there)

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes - paired with codex on this

0cc4m · 2026-08-22T08:02:58Z

AMD FNUZ is only CDNA3, see https://rocm.docs.amd.com/projects/HIP/en/latest/reference/low_fp_types.html#id4

RDNA4 and CDNA4 use FP8 OCP.

michaelw9999 · 2026-08-31T04:46:43Z

Hi @ORippler wondering if you saw my MXFP8 #26157 PR? (or older MXFP6) if NV is interested in them? I didn't put up CUDA kernel PRs, since since CPU was not in yet, but I put the CUDA forks up on the HF model links. The MXFP8 is superfast... 😃 I'll be putting up NVFP4/MXFP6/MXFP8 Qwen3.8-27B soon.

ORippler · 2026-09-04T13:51:59Z

Hi @ORippler wondering if you saw my MXFP8 #26157 PR? (or older MXFP6) if NV is interested in them? I didn't put up CUDA kernel PRs, since since CPU was not in yet, but I put the CUDA forks up on the HF model links. The MXFP8 is superfast... 😃 I'll be putting up NVFP4/MXFP6/MXFP8 Qwen3.8-27B soon.

I don't have BW to look at MXFP8 atm, but intuitively feel this should come as an extension to OCP FP8, as MXFP8 uses OCP FP8

ORippler · 2026-10-07T17:05:13Z

+        if (type_a == GGML_TYPE_F8_E4M3 && backend_has_feature(backend, "NATIVE_FP8")) {
+            return 5e-3;
+        }


Based on testing with a B4500 & CUDA, it could be 1e-3 also. I don't have a strong stance on the strictness of this

am17an

Just a few nits. I haven't looked at the conversion/quantization code, the ggml side looks fine. My general opinion is that we should try and support the 128x128 tile which would enable us to run "true" inference for safetensors checkpoints for a lot of a Chinese OSS models, but understand if that's too much to add.

approved by mistake

Added in ggml-org/llama.cpp#27512  --- > [!NOTE] > **Low Risk** > Metadata and enum additions only; no runtime inference or security-sensitive logic changes. > > **Overview** > Adds **F8_E4M3** (OCP-FP8 E4M3) so GGUF tooling stays aligned with llama.cpp (ggml-org/llama.cpp#27512). > > In `packages/tasks`, the type is registered on **`GGMLFileQuantizationType`** (42) and **`GGMLQuantizationType`** (43), and **`GGUF_QUANT_ORDER`** places it with other 8-bit formats after `Q8_0`. In `packages/gguf`, **`GGUF_QUANT_DESCRIPTIONS`** documents the format (with an FP8 reference link) and **`GGML_QUANT_SIZES`** records **8 bits per weight** via `calcBPW(1, 1)`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 231fb19. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

… = 43 GGML_TYPE_Q4_1_G64 43 -> 99, GGML_TYPE_Q4_16 44 -> 100, LLAMA_FTYPE_MOSTLY_Q4_1_G64 42 -> 99; GGML_TYPE_F8_E4M3 = 43 and LLAMA_FTYPE_MOSTLY_F8_E4M3 = 42 as upstream (ggml-org#27512). GGML_TYPE_COUNT 101: ids 44..98 are gaps (zero type_traits, blck_size 0), handled like upstream's existing gaps 31-33/36-38 (gguf rejects them at load, the quantize/test loops skip null names / zero sizes). Upstream's future ids grow from 44 and can no longer collide with ours. gguf-py constants likewise (+ LlamaFileType.MOSTLY_Q4_1_G64); convert_to_g64.py takes the ids from gguf-py. RPC patch version 4 + static_assert on GGML_TYPE_COUNT (head and rpc-server must be the same build). Files written with the old ids are migrated in place by runs/fn-m2/rel3_retype.py (header only; --reverse for builds <= r31). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… since ggml-org#27512 (signbit/isnan/isinf undeclared in ggml_fp32_to_f8_e4m3), GCC 16.2 and clang 22 with libstdc++ 16 (cherry picked from commit d2516b5) Picked from ggml-org#31454 (open at the time of picking).

* WIP OCP FP8 E4M3 support * Fix python type-checker * Materialize input_scales also for FP8 when converting * Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4 1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on dims alone. Also check on dtype for FP8 2. Need to reshape FP8 QKV projections scales in the same way that weights are reshaped * Allow remaining FP8 tensors to be dequantized during conversion * Allow tensor-strategy for FP8-only compressed-tensor ckpts * Fix and test gguf-py behavior for FP8_E4M3 * Disallow QKV-fusion for FP8 temporarily * Fix conversion of fused qkvz & FP8 for Qwen3-Next * Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws * Emit warnings when not fusing FP8 during conversion * More FP8 conversion fixes * Add supported for strided copies to/from FP8 in CPU * remove unneeded scratch buffer from FP8 cpu cpy/dup path * add explanatory comments and TODOs * Add dedicated encoder and decoder tests for FP8 * Whitelist ggml_op_get_rows in metal instead of blacklisting required to fix failing CI, and white-listing should be more future-proof than adding FP8 to the black-list https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512 * Fix more conversion bugs * Defer weight materialization during conversion * Blacklist fp8 in metal backend instead of white-list * Adjust indents * Fix lint + execute tests * Add fast CPU kernel for F8_E4M3 * Conversion fixes * Fix remaining gaps * Enable tied weights with shared scales * Make mandatory-F32 rules apply to FP8 during conversion * Fix python type checks https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512 * Fix kaleidai unit-test failures https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150 * Relax numeric precision constraints for pre-check * Leave fast-FP8 CPU on RISC-V for follow-up We already exclude arm from the fast-path * Allow output scale tensors to be located in any gguf shard * Move test_conversion_fp8.py test execution to behavior-check.yaml * Apply batched suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it * Install all requirements also on Windows for behavior-check Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512 * Unmap lazy tensors before clean-up in Windows Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512 * LF line endings https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512 * Extract FP8 tests from llama-arch to its own test-fixture See ggml-org/llama.cpp#27512 (comment) * Update gguf-py/gguf/scripts/gguf_convert_endian.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

The port of ggml-org#27512 @ 41ee520 asked for output.scale and output.input_scale on every output weight. test-llama-archs creates every tensor a model asks for, so the generated models with a tied output got both scales and failed to load ("output scales require a separate output weight"), and the draft of --fit-borrowing-draft had two tensors more than its loader expected. The merged ggml-org#27512 (68d20b8) creates them only when the file has them; do the same. Assisted-by: Claude Opus 5.5

qwen3next now multiplies the gate projection by attn_gate.scale when the model has one (from the port of ggml-org#27512, as upstream). The models test-llama-archs generates carry every optional tensor, so the extra MUL sits between the gated norm's nodes and RMS_NORM+MUL+UNARY+MUL becomes RMS_NORM+MUL and UNARY+MUL. Models without the scale are unchanged. Assisted-by: Claude Opus 5.5

* WIP OCP FP8 E4M3 support * Fix python type-checker * Materialize input_scales also for FP8 when converting * Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4 1. Per row FP8 wrongly fell to NVFP4 conversion, as NVFP4 checked on dims alone. Also check on dtype for FP8 2. Need to reshape FP8 QKV projections scales in the same way that weights are reshaped * Allow remaining FP8 tensors to be dequantized during conversion * Allow tensor-strategy for FP8-only compressed-tensor ckpts * Fix and test gguf-py behavior for FP8_E4M3 * Disallow QKV-fusion for FP8 temporarily * Fix conversion of fused qkvz & FP8 for Qwen3-Next * Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws * Emit warnings when not fusing FP8 during conversion * More FP8 conversion fixes * Add supported for strided copies to/from FP8 in CPU * remove unneeded scratch buffer from FP8 cpu cpy/dup path * add explanatory comments and TODOs * Add dedicated encoder and decoder tests for FP8 * Whitelist ggml_op_get_rows in metal instead of blacklisting required to fix failing CI, and white-listing should be more future-proof than adding FP8 to the black-list https://github.com/ggml-org/llama.cpp/actions/runs/38433916498/job/115422880633?pr=27512 * Fix more conversion bugs * Defer weight materialization during conversion * Blacklist fp8 in metal backend instead of white-list * Adjust indents * Fix lint + execute tests * Add fast CPU kernel for F8_E4M3 * Conversion fixes * Fix remaining gaps * Enable tied weights with shared scales * Make mandatory-F32 rules apply to FP8 during conversion * Fix python type checks https://github.com/ggml-org/llama.cpp/actions/runs/39765670052/job/119780379595?pr=27512 * Fix kaleidai unit-test failures https://github.com/ggml-org/llama.cpp/actions/runs/39765670125/job/119780383150 * Relax numeric precision constraints for pre-check * Leave fast-FP8 CPU on RISC-V for follow-up We already exclude arm from the fast-path * Allow output scale tensors to be located in any gguf shard * Move test_conversion_fp8.py test execution to behavior-check.yaml * Apply batched suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests for it * Install all requirements also on Windows for behavior-check Fixes https://github.com/ggml-org/llama.cpp/actions/runs/40535852590/job/122300758223?pr=27512 * Unmap lazy tensors before clean-up in Windows Should fix https://github.com/ggml-org/llama.cpp/actions/runs/40539797824/job/122314782459?pr=27512 * LF line endings https://github.com/ggml-org/llama.cpp/actions/runs/40539798168/job/122314791118?pr=27512 * Extract FP8 tests from llama-arch to its own test-fixture See ggml-org/llama.cpp#27512 (comment) * Update gguf-py/gguf/scripts/gguf_convert_endian.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

This adds partial support for [OCP FP8](https://github.com/opencomputeproject/FP8/tree/main) (E4M3). It makes no claim of being compliant with the specification. It is a result of PR ggml-org#27512 - Quant: OCP FP8 E4M3 support. This commit is support for the ADD op, in ggml-sycl/binbcast.cpp. Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF:Q4_K via llama.cpp)

This adds partial support for [OCP FP8](https://github.com/opencomputeproject/FP8/tree/main) (E4M3). It makes no claim of being compliant with the specification. It is a result of PR ggml-org#27512 - Quant: OCP FP8 E4M3 support. This commit is FP8 e4m3 support for the MUL_MAT and MUL_MAT_ID ops, in ggml-sycl/convert.cpp. It uses the generic fp32 GEMM fallback, no DMMV/MMVQ/XMX/reorder fast path. I have not investigated oneDNN support of FP8 e4m3, but there may be opportunties for low-effect / high-impact speed improvements there. Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF:Q4_K via llama.cpp)

This adds partial support for [OCP FP8](https://github.com/opencomputeproject/FP8/tree/main) (E4M3). It makes no claim of being compliant with the specification. It is a result of PR ggml-org#27512 - Quant: OCP FP8 E4M3 support. This commit is support for the CPY op, in ggml-sycl/cpy.cpp. Infrequent (~5-10% of runs) the backend-ops test may fail, due to a precision / tolerance issue in quantizing f16 to low-bit quants. This is pre-existing, and outside of the scope of this work. Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF:Q4_K via llama.cpp)

This adds partial support for [OCP FP8](https://github.com/opencomputeproject/FP8/tree/main) (E4M3). It makes no claim of being compliant with the specification. It is a result of PR ggml-org#27512 - Quant: OCP FP8 E4M3 support. This commit is support for the SET_ROWS op, in ggml-sycl/set_rows.cpp. Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF:Q4_K via llama.cpp)

ggml, cuda : FP8 E4M3 weights and FP8 prefill attention for temp-11739 (port of ggml-org#27512, ggml-org#28392, ggml-org#28413)

Squashed port of the runtime part onto fabric. GGML_TYPE_F8_E4M3 keeps upstream's ID 43; fabric's KV-cache-only TBQ/PQ types move to 52-59. Converter changes and the test-llama-archs FP8 cases are not ported yet. b12032 rebase: b12032 already has upstream ggml-org#27512, so only the move of the TBQ/PQ types to 52-59 remains. RPC goes to 115 here so peers with the old ids are rejected at HELLO, and the assert on TBQ3_0 follows to 52. Assisted-by: Claude Opus 5.5 (cherry picked from commit 7291978)

qwen3next now multiplies the gate projection by attn_gate.scale when the model has one (from the port of ggml-org#27512, as upstream). The models test-llama-archs generates carry every optional tensor, so the extra MUL sits between the gated norm's nodes and RMS_NORM+MUL+UNARY+MUL becomes RMS_NORM+MUL and UNARY+MUL. Models without the scale are unchanged. Assisted-by: Claude Opus 5.5 (cherry picked from commit b35ec1e)

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning conversion labels Aug 21, 2026

github-actions Bot mentioned this pull request Aug 24, 2026

FreeToken ROCm Watch — notification channel Castoff995/FreeToken-rocm-test#1

Draft

ORippler force-pushed the osimons/ocp_fp8 branch from 7ab744c to d21c273 Compare August 31, 2026 13:31

ORippler force-pushed the osimons/ocp_fp8 branch from d21c273 to cef6c52 Compare September 4, 2026 18:51

happyyzy mentioned this pull request Sep 14, 2026

hexagon: add an experimental Qualcomm FP8 DiT and VAE path leejet/stable-diffusion.cpp#1970

Open

ORippler force-pushed the osimons/ocp_fp8 branch 5 times, most recently from 91fe47f to d904f34 Compare October 1, 2026 11:58

github-actions Bot added the model Model specific label Oct 1, 2026

ORippler force-pushed the osimons/ocp_fp8 branch 2 times, most recently from 479c902 to 574eb98 Compare October 1, 2026 15:59

ORippler commented Oct 7, 2026

View reviewed changes

ORippler force-pushed the osimons/ocp_fp8 branch from 574eb98 to 8efdff0 Compare October 9, 2026 19:14

ORippler marked this pull request as ready for review October 9, 2026 19:27

ORippler requested review from CISC and ggerganov as code owners October 9, 2026 19:27

ORippler mentioned this pull request Oct 9, 2026

ggml : add FP8 types and CPU conversions #28485

Draft

ORippler force-pushed the osimons/ocp_fp8 branch 2 times, most recently from 41dbb3b to 62a9875 Compare October 14, 2026 14:51

ORippler requested a review from a team as a code owner October 14, 2026 14:51

github-actions Bot added the Apple Metal https://en.wikipedia.org/wiki/Metal_(API) label Oct 14, 2026

am17an previously approved these changes Oct 15, 2026

View reviewed changes

Comment thread ggml/src/ggml-cpu/ops.cpp Resolved

Comment thread ggml/src/ggml-cpu/ops.cpp Resolved

Comment thread ggml/src/ggml-metal/ggml-metal-device.m Resolved

leehack mentioned this pull request Oct 30, 2026

Qualify post-v0.6.0 upstream correctness fixes and consumer compatibility leehack/llamadart-native#112

Open

3 of 6 tasks

This was referenced Oct 30, 2026

Compile bug: Vulkan backend fails to build since #27512 (signbit/isnan/isinf undeclared in ggml_fp32_to_f8_e4m3), GCC 16.2 and clang 22 with libstdc++ 16 #31445

Closed

fixes issue #31445 Compile bug: Vulkan backend fails to build since #… #31454

Closed

This was referenced Oct 31, 2026

sycl : accept only implemented types in MUL_MAT, SET_ROWS and CPY #31460

Closed

sycl: Add device-side MoE scheduler #30156

Merged

flyingdoog mentioned this pull request Nov 1, 2026

vulkan : fix out-of-bounds NVFP4 UE4M3 scale lookup #31079

Merged

canonrebel04 mentioned this pull request Nov 2, 2026

sync: merge upstream/master c6fb3b5be + 785322d17 (2026-10-26, 2026-11-01) — stacked on #85 canonrebel04/llama.cpp#86

Open

ggerganov added the highlight Changes that will be highlighted in the next release notes label Nov 4, 2026

This was referenced Nov 4, 2026

ggml : bump version to 0.29.0 ggml-org/ggml#1680

Merged

llama.cpp : bump version to 0.9.0 #31646

Merged

danielhanchen mentioned this pull request Nov 4, 2026

metal: add NVFP4 mul_mat, mul_mat_id and get_rows kernels (mirror of #330 by @Lyxot) unslothai/llama.cpp#348

Open

BrewTestBot mentioned this pull request Nov 4, 2026

ggml 0.29.0 Homebrew/homebrew-core#323401

Merged

1 task

fosskar Bot mentioned this pull request Nov 5, 2026

llama-cpp: 0.8.0 -> 0.9.0 fosskar/nixfiles#2265

Merged

amangupta-tether mentioned this pull request Nov 5, 2026

ggml, cuda : FP8 E4M3 weights and FP8 prefill attention for temp-11739 (port of #27512, #28392, #28413) tetherto/qvac-fabric-llm.cpp#487

Merged

mctylr-gh mentioned this pull request Nov 8, 2026

Feature Request: SYCL support for OCP FP8 e4m3 #31870

Open

4 tasks done

mctylr-gh mentioned this pull request Nov 8, 2026

[SYCL] F8 e4m3 initial support SET_ROWS, CPY, ADD, MUL_MAT, MUL_MAT_ID #31871

Open

Navigation Menu

Repository navigation

Quant: OCP FP8 E4M3 support - #27512

Quant: OCP FP8 E4M3 support#27512
ORippler merged 40 commits into
masterfrom
osimons/ocp_fp8

ORippler commented Aug 21, 2026 •

edited

Loading

Uh oh!

0cc4m commented Aug 22, 2026

Uh oh!

michaelw9999 commented Aug 31, 2026

Uh oh!

ORippler commented Sep 4, 2026

Uh oh!

ORippler Oct 7, 2026 •

edited

Loading

Uh oh!

am17an left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

ORippler commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

0cc4m commented Aug 22, 2026

Uh oh!

michaelw9999 commented Aug 31, 2026

Uh oh!

ORippler commented Sep 4, 2026

Uh oh!

ORippler Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

am17an left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ORippler commented Aug 21, 2026 •

edited

Loading

ORippler Oct 7, 2026 •

edited

Loading

am17an left a comment •

edited

Loading