Repository navigation
HIP gfx103x FP16 prompt (#835 follow-up): STRATA_F16_RANGE measures what reaches FP16's range, and hip_prefill_gemm covers the FP16-io route - #1149
Closed
Vop3p wants to merge 2 commits into
Conversation
… any HIP card, and its BF16 inputs are no longer zero The test ran only the FP32-out GEMMs, so the gfx103x prompt path's FP16-out GEMM (f16_inplace, widen_rows_f16) and bf16()'s W conversion through the scratch were not under ctest. Every shape now runs a second time with set_f16_io(true), forced on whatever HIP card runs the test: X as the FP16 image for the BF16 products, a scratch of 64 rows so a 96-row weight converts in two slices (the second written at Y + 64), ldy > N, an odd N (257), T = 1, and a beta = 1 f16() call, which keeps the FP32-out GEMM. FP16-out results are checked against the double reference at FP16's rounding (6e-4 relative per element plus 1e-4, rel L2 < 1e-3); the row padding and the guards around Y must be untouched, as before. The FP32-out runs keep their tolerance. Found while adding them: the BF16 inputs were all zero. hip_bfloat16(float) produced data 0 in this test's build (ROCm 10.0, gfx1030; not traced further), so the two BF16 cases passed on nothing (rel L2 printed 0). The BF16 bits are now made by hand (round to nearest even on the float's bits); with real inputs the BF16 cases pass at rel L2 1.9e-07 (FP32 out) and 2.2e-04 (FP16 out). On an RX 6900 XT (gfx1030, ROCm 10.0): 11 of 11 pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 4c6c9fe)
…es FP16; the range measured (outputs peak at 410 of 65504) and the FP32 route compared The README's range check said "none beyond 65504" over 110K tokens without the numbers. The engine now has the counter: STRATA_F16_RANGE=1 records, per device, the largest |value| and the count beyond 65504 (or not finite) where a value enters or leaves FP16 - the activation images (hf_sat, the image writers' one funnel), the BF16 weights converted to FP16 (bf16_to_f16_rows) and the FP16 GEMM outputs as they are widened (widen_rows_f16: an output rocBLAS wrote beyond the range is Inf there) - and prints one line per prompt. Off, the cost is one flag read per value; on, a 34.4K-token prompt read at 889.8 tok/s against 888.5. to_f16 (the hc image for the FP16-weight GEMMs, the PLE key image) saturates like the other image writers now, so every FP16 image is under the same rule. The HIP shim gains cudaMemcpyFromSymbol. Measured on one RX 6900 XT over 466K tokens in nine prompts (corpus slices, two prompts built to push activations - delimiter floods, a token repeated 4,000 times, 6,000-character runs, base64, five scripts - and four batches of real requests): outputs peak at 409.5 (160x below 65504), activations at 112.2, the BF16 weights at 11.56; nothing beyond the range, nothing non-finite. The prompts built to push activations stayed below the ordinary corpus. One model (GSQ-RCO IQ3_S); the table is in the README's range section. The README also gains the same-card comparison with Niko1221#1006, which takes these GEMMs as FP32 SGEMMs: 8.4K / 32.8K / 104K-token prompts 412 / 455 / 456 tok/s on the old path, 622 / 742 / 750 with Niko1221#1006, 730 / 900 / 904 with this change (the card's FP32 rate against its FP16 rate), with the phase times and what the SGEMM route covers that this change does not. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit e780c85)
This was referenced Oct 6, 2026
Niko1221
added a commit
that referenced
this pull request
Oct 17, 2026
…s cudaMemcpyFromSymbol Fix-up to the ported #1149 commits. The range counters and the saturating to_f16 are compiled for HIP only, so the CUDA kernels (hf_sat in the SwiGLU product, the embedding image) stay exactly as they were, with no flag read; set_f16_range / f16_range_read are no-ops there. strata_hip.h (the STRATA_HIP_GFX906 compat layer, which goes first on that backend's include path) gets the cudaMemcpyFromSymbol mapping the HIP shim already has, so the counters also compile for gfx906.
Owner
|
Thanks xjc10. Ported (0.1.45) with your commits; the counters and the test are in, opt-in via STRATA_F16_RANGE=1. We kept the CUDA kernels out of it and mapped cudaMemcpyFromSymbol for the gfx906 layer so it still compiles. With the counters on, an R9700 run is byte-identical to the build without them. |
Owner
|
Thanks for the pull request. Your change is already in If part of your work did not make it in and you want it considered, open a new pull request on top of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two follow-ups to #835 that were on its branch when
mainwas rewritten, re-filed on 0.1.40.1 with the review changes (opt-inSTRATA_HIP_PROMPT_F16=1, the mode decided once per stage) kept.STRATA_F16_RANGE=1records, per device, the largest |value| and the count beyond 65504 (or not finite) at the three places a value enters or leaves FP16 - the activation images (hf_sat), the BF16 weights converted to FP16 (bf16_to_f16_rows) and the FP16 GEMM outputs as they are widened (widen_rows_f16) - and prints one line per prompt. It sits beside the review'sSTRATA_DBG_NANcount inwiden_rows_f16. Off, the cost is one flag read per value; on, a 34.4K-token prompt read at 889.8 tok/s against 888.5 without it. Measured on one RX 6900 XT over 466K tokens in nine prompts (three corpus slices, two prompts built to push activations - delimiter floods, repeated tokens, base64, mixed scripts - and four batches of real API requests): FP16 GEMM outputs peak at 410 of 65504, nothing beyond the range or non-finite. The numbers are inbench/results/2026-10-04-rdna2-fp16-prompt/README.md, which also gains the same-card comparison against the FP32 SGEMM route of HIP RDNA2 (gfx103x): run the prompt path's 16-bit GEMMs as FP32 SGEMMs (RX 6800: prompts 312 -> 452 tok/s) #1006.hip_prefill_gemmruns its shapes a second time withset_f16_io(true)on any HIP card: X as the FP16 image for the BF16 products, a scratch of 64 rows so a 96-row weight converts in two slices,ldy > N, an odd N, T = 1 and abeta = 1f16() call (which keeps the FP32-out GEMM); the FP16-out results are checked at FP16's rounding, the row padding and the guards around Y must be untouched. Its BF16 inputs are no longer all zero.HIP ctest on the RX 6900 XT at 0.1.40.1 (
82f46a8), with this change and #1148 (thehip_prefill_wmma_gemm_parityskip) applied: 19 tests, 13 pass, 6 skip (the WMMA / hipBLASLt / fused-kernel tests that do not apply to gfx1030), 0 fail.Original branch: #835's
hip-gfx103x-fp16-promptafter the merge.Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030, PCIe 4.0 x8 each) / Ryzen 5 5600X, ROCm 10.0.