[leejet]

Overview

This PR adds CPU support for FP8, including the ability to load FP8 weights directly and convert them to FP16, BF16, or FP32.

Many diffusion models already support FP8, and some are even released with FP8 weights natively. On hardware with native FP8 support, FP8 computation can also be faster than FP16/BF16, so having proper FP8 support in GGML would be useful for both model loading and execution.

I'm not sure whether upstream would be happy with this approach, so for now I've only included the CPU-related changes in this PR. If this approach is acceptable, I can follow up with another PR that adds the conversion code for the other backends, as well as FP8 mul_mat support for CUDA.

This is the GGML implementation I'm currently using in sd.cpp:

• leejet/ggml@032b699
• leejet/ggml@e20c3a1

For FP8 scale handling, my current idea is to keep the scale as a separate tensor instead of packing it into the FP8 tensor itself. I think this is more flexible and also reduces the maintenance burden.

Additional information

leejet/stable-diffusion.cpp#1913
leejet/stable-diffusion.cpp#1916

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, I use AI to help with debugging and porting code from ggml to llama.cpp.

[am17an]

There are some pending PRs w.r.t FP8 #28413. It is something we're planning to add to the project

[ORippler]

@leejet feel free to take a look at #27512, I consider it ready for review. How often do you encounter e5m2 for diffusion-based workloads? wondering how important it will be to add as a follow-up

[leejet]

Thanks for the heads-up! I’ll take a look at #27512. In my experience, E4M3 is much more common for diffusion model weights. I do occasionally encounter E5M2 checkpoints, so support would be useful, but I wouldn’t consider it a high priority. Adding it as a follow-up sounds reasonable.