Repository navigation
GDN batch path skips activation quantization when STRATA_QFUSE=1 #1139
Description
Static inspection of the public upstream source shows that the batched GDN path can skip writing its Q8 activation buffer when STRATA_QFUSE=1.
Affected source: src/core/verify.cpp at 1735d6471df29b42c26170efaac1f1446a58640f, lines 872–887. The same control flow is present at current main 82f46a8c8f475f001ad76d92f58f4a4f8ffb0253.
- With
batch_rec_, each slot callsgdn_step_norm_multiwithout the optional fused Q8 output pointer; it defaults to null. - The solo branch supplies
xq_wheng_qfuse()is true. - After both branches,
native_quantize_q8_1runs only when!g_qfuse(). - The output
native_mmvqthen consumesxq_.
For batch_rec_ && g_qfuse(), neither the recurrence call nor the explicit quantizer writes this projection's activation into xq_. It can therefore consume an earlier activation. This appears to be a correctness defect, rather than permissible numerical rounding drift.
A possible correction is to run the explicit quantizer when batch_rec_ || !g_qfuse(), or pass a correctly offset fused output buffer for each slot. Neither proposed source fix has been tested here. A focused regression should exercise QFUSE-on batched GDN and check that the output projection receives the quantized current GDN result.
The configuration workaround is STRATA_QFUSE=0. This report is limited to public-source analysis; it includes no private benchmark logs or deployment configuration.
Activity
johan-windahl commentedon Oct 6, 2026
Confirmed at runtime on Strix Halo (gfx1151). On that GPU the bug is on by default:
STRATA_QFUSE=1is one of the gfx1151 arch defaults (src/core/arch_defaults.cpp:36), so every Strix Halo user who runs"parallel"> 1 gets corrupted output as soon as two requests share a batch window.Setup. Ryzen AI Max+ 395, 128 GB (BIOS UMA 512 MB, GTT 124 GiB), Linux 7.0. v0.1.40 (
1cbcacb), built from source per docs/STRIX_HALO.md with ROCm 7.14.1 (hipBLASLt 100401). ISTA GSQ-RCO IQ3_S,--family qwen,--parallel 4,--vision cpu, MTP on (setup's--spec 4 --spec-min-p 0.5),--expert-cache 4000,--kv int8 --kv-resident 32768.Symptom. A request that runs alone answers correctly. Once a second request arrives and both decode in batch slots, both replies degenerate within a few tokens into mixed-script junk or one token repeated 256 times. The server log then says
the reply repeated one token ('CLR') 256 times in a row, and'qiq','affer'and'_depend'showed up the same way. The first request derails mid-sentence at the moment it moves into a slot:'A heat pump works by using a refrigerant and an electrical compressor to absorb thermal energy向前 objectivesite…'There is no crash and nothing in the engine log points to an error. Only the content is wrong, which makes it easy to miss in throughput benchmarks.
Repro. Send 2 concurrent chat completions with
enable_thinking: falseandmax_tokens100, staggered by 0.5 s, then read the text. It reproduces 100% of the time.Bisect. For each case I ran the 2-concurrent test twice and read every reply.
STRATA_GFX1151_DEFAULTS=0: correctTSUM, LFUSE, GDN_SPLIT, QFUSE, PLE_BATCH, SH_STREAM: correct. OffQ8_PACKED, Q6_PACKED, MMVF_ROWS, ATTN_LANECELL, EXPERT_V2: still corrupt.PLE_BATCH, QFUSE, GDN_SPLIT: correct. OffTSUM, LFUSE, SH_STREAM: still corrupt.PLE_BATCH=0: corrupt. OnlyGDN_SPLIT=0: corrupt. OnlyQFUSE=0: correct.These results match the static analysis above. On the
batch_rec_pathgdn_step_norm_multigetsnullptrfor the fused q8_1 output, andnative_quantize_q8_1is skipped underg_qfuse(). The GDN output projection then reads a stalexq_.With the workaround (
STRATA_QFUSE=0and every other gfx1151 default left on), output is correct:There is no measurable speed cost on our box. Solo decode was 43.8 → 43.7 tok/s at short context and 38.8 → 39.2 at about 19K depth. The QFUSE=1 numbers were measured before we noticed the corruption, but solo output was correct then.
I have not tested the source fix here. The minimal change looks like
if (!g_qfuse() || batch_rec_) native_quantize_q8_1(...)at verify.cpp:887. Until that lands, it may be worth droppingSTRATA_QFUSEfrom the gfx1151 default table, because gfx1151 is currently the only place where it is on without being asked for.huppiflupp commentedon Oct 6, 2026
Reproduced on gfx1151 with
"parallel": 2(6 of 6 concurrent replies corrupt on 82f46a8) and verified that PR #1209 fixes it there: concurrent replies byte-identical to the solo ones, solo ids and speed unchanged. Details in #1209.Written with an AI assistant (Claude Code).
jiayeZhu commentedon Oct 7, 2026
Another gfx1151 confirmation, with a few data points that are not in the thread yet.
Setup. Ryzen AI Max+ 395, 128 GB, Ubuntu 26.04.1 (Linux 7.0.0), ROCm 7.14.1, engine built from source at 82f46a8 (v0.1.40.1), HIP backend. Server with
"parallel": 2–4,--max-context 262144 --kv int8 --kv-resident 32768 --spec 4 --spec-min-p 0.5 --mtp … --mmap-experts --no-prefill-borrow --prefill 8192 --conversation-cache-mib 16384, gfx1151 defaults on.Test. N chat requests sent at the same moment, each with a prompt that has one fixed answer (first 25 primes, weekdays and months, count 1–60, the alphabet …),
enable_thinking: false,max_tokens220, and the reply checked for the expected sequence.More files affected. With the defaults as shipped:
parallelEach corrupt reply starts with the right first token and is junk from the second one on (
'2,有意reeimeDTaleyonyidy_depend blo bloaffer…'). Sending no sampling fields, or only one oftemperature/top_p/top_k, makes no difference, and neither does dropping--kv int8.QFUSE alone is sufficient. The bisect above turns switches off; this one starts from
STRATA_GFX1151_DEFAULTS=0(2 of 2 correct) and turns them back on, UD-IQ4_XS,"parallel": 2:Q8_PACKED, Q6_PACKED, MMVF_ROWS, ATTN_LANECELL, EXPERT_V2on: correctTSUM,LFUSE,GDN_SPLIT,PLE_BATCH,SH_STREAM, each alone: correctQFUSE=1alone: 0 of 2Workaround checked on four files.
STRATA_QFUSE=0, everything else unchanged:No speed cost here either: solo decode 68.4–68.6 tok/s with
QFUSE=0against 68.7–68.9 with it on (500 tokens of counting, which the draft head predicts almost completely, so this is far above a prose rate).How it showed up. A coding agent sent out four subagents in one call. None of them returned: each one thought up to its
reasoning_budget_tokens, then produced junk for as long as it was left running, so no tool call ever parsed. Nothing in the server or engine log marks it as an error. Our earlier concurrency benchmarks had recorded token rates for those streams without reading the text, and looked fine.I have not tested PR #1209 on this box.
verify: a batch's GDN output is quantized to q8_1 under STRATA_QFUSE …
Niko1221 commentedon Oct 7, 2026
Fixed in 0.1.40.2. With STRATA_QFUSE=1 (the gfx1151 default) the batched GDN path passed no q8_1 destination and still skipped the quantization, so the out-projection read stale bytes. It now quantizes there. Checked on an R9700 with STRATA_QFUSE=1 and --batch 2: 0.1.40.1 gives garbage ('Leadletest ...'), the fix gives output identical to solo. On an RTX 3060 the same flag no longer gives garbage but a slot can still part from its solo run after a few tokens, because with STRATA_QFUSE=1 the solo output itself changes with the draft window size there (QFUSE is off by default on NVIDIA). Please retest on Strix Halo.
huppiflupp commentedon Oct 7, 2026
Retest on Strix Halo (Ryzen AI MAX+ 395, Radeon 8060S, gfx1151, ROCm 7.14.1, HIP): 0.1.40.2 fixes it.
Setup.
"parallel": 2, gfx1151 defaults (soSTRATA_QFUSE=1),--no-prefill-borrow,--spec 4 --mtp,--kv int8, greedy (temperature 0), up to 220 new tokens. Four prompts with a fixed answer (first 25 primes, weekdays + months, count 1-60, alphabet). Each prompt run alone first, then every pair of two prompts sent at the same moment, 6 pairs x 3 rounds = 36 concurrent answers, each compared character by character with its solo answer.2,àmlad特别onyidyidy…instead of2, 3, 5, 7, …)Both builds were made with the same CMake options (HIP, gfx1151,
STRATA_PREFILL_MMQ=ON) and the same model/config.Not covered:
parallelabove 2, and long outputs beyond 220 tokens.Written with an AI assistant (Claude Code); the measurements were run on my machine.
verify: a batch's GDN output is quantized to q8_1 under STRATA_QFUSE …
verify: a batch's GDN output is quantized to q8_1 under STRATA_QFUSE …