Repository navigation
Add dequant operator (int32 → bfloat16) - #96
Closed
albiol2004 wants to merge 2 commits into
Closed
albiol2004 wants to merge 2 commits into
albiol2004 wants to merge 2 commits into
Conversation
Add a dequantize operator that converts int32 GEMM accumulator outputs to bfloat16 with per-group scale factor multiplication. This is needed to use INT8 GEMM (amd#93) output in the rest of the inference pipeline. - C++ kernel: int32 → float32 (scale in float32) → bfloat16 - Python operator following existing dequant/ pattern - 32 parametrized tests (1-8 columns, 1-2 channels, 1K-8K elements)
4 tasks done
Align operator directory name with the kernel file convention (dequant_i32_bf16.cc). The existing dequant (int4→bf16) and quantize operators use incomplete type suffixes; this rename establishes the full input_output naming pattern.
Contributor
Author
|
Very sub optimal too |
Soket1
pushed a commit
to Soket1/IRON-windows
that referenced
this pull request
Jul 20, 2026
- depth=4 weight buffers (A0..A3) exceed 16-lock cap (all locks used) - L1 memory at 31.7KB > AIE2P tile capacity → compile fails silently - Real expanded ping-pong needs MemTile-level prefetch (512KB SRAM) - Reverted to proven depth=2 (A0/A1) baseline at 29.2 t/s - Cache key restored to _mc_preq_vexp (no _dp4 suffix) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added
aie_kernels/generic/dequant_i32_bf16.cc: C++ AIE kernel that converts int32 values to bfloat16 with per-group scale factor multiplication. Scales in float32 before final bf16 conversion for precision.iron/operators/dequant_i32/: Python operator (op.py, design.py, reference.py, test.py) following the existingdequant/pattern.Changed
Nothing. This is a new operator with no modifications to existing code.
Removed
Nothing.
Motivation
The INT8 GEMM operator (#93) produces int32 accumulators, but the rest of the inference pipeline (RMSNorm, SiLU, RoPE, residual adds) operates in bfloat16. This operator converts INT8 GEMM output back to
bfloat16, enabling end-to-end W8A8 quantized inference on the NPU.
Test results
Checklist
Closes #95