Repository navigation
Add quantize operator (bfloat16 → int8) - #98
Closed
albiol2004 wants to merge 2 commits into
Closed
albiol2004 wants to merge 2 commits into
albiol2004 wants to merge 2 commits into
Conversation
Add a per-group symmetric quantization operator that converts bfloat16 tensors to int8 with per-group scale factors. Completes the W8A8 quantized inference pipeline alongside INT8 GEMM (amd#93) and dequant i32→bf16 (amd#95). - C++ kernel: two-pass (find max abs, then scale+round+saturate) - Python operator following existing dequant/ pattern - 32 parametrized tests (1-8 columns, 1-2 channels, 1K-8K elements)
Align operator directory name with the kernel file convention (quantize_bf16_i8.cc). This establishes a consistent full input_output type naming pattern for quantization operators.
Contributor
Author
|
Very very suboptimal against fused kernels |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added
aie_kernels/generic/quantize_bf16_i8.cc: C++ AIE kernel that quantizes bfloat16 values to int8 with per-group symmetric scaling. Two-pass: find max absolute value per group, then scale/round/saturate.iron/operators/quantize/: Python operator (op.py, design.py, reference.py, test.py) following the existingdequant/pattern.Changed
Nothing. This is a new operator with no modifications to existing code.
Removed
Nothing.
Motivation
Completes the W8A8 quantized inference pipeline for the NPU:
Together these three operators enable end-to-end quantized matrix multiplication on the NPU without CPU roundtrips, leveraging the NPU's ~50 TOPs INT8 throughput.
Test results
Notes
aie::to_fixed<int8>if supported on AIE2/AIE2P targets (marked with TODO).Checklist
Tests pass locally
Code formatted (black + clang-format)
License headers on all files
No changes to existing code
Closes Add quantize operator (bf16->int8) #97