Repository navigation
HIP (gfx1030): intermittent 'verify: timed out at layer N (#267)' with UD-Q4_K_XL — repro data #649
Closed as not planned
Description
Environment
- RX 6950 XT (gfx1030), ROCm 7.1, HIP backend, engine 0.1.37 and current main (99f3dbd) — same behavior on both
- 31 GB RAM, UD-Q4_K_XL (
--nativeGGUF-in-place,--resident-budget-gib),--spec 2,STRATA_HC_SPLIT=0(the default hc staging path also hangs on this card) - Model: the Huihui abliterated re-split of Qwen3.8-Flash-Next UD-Q4_K_XL (needs the split-shard arch-guard fix from native dense: run the architecture guard only on the first shard of a split #648 to load at all)
Symptom
Decode usually starts and can generate 24+ tokens correctly, then a verify window dies:
strata generate: verify: timed out at layer 11; its GPU waits were released but the GPU did not finish within 5 s (#267)
- Always at a full-attention layer (3, 11, 13, 25, 35, 46 seen — every N was (L+1)%4==0).
- Intermittent: same command, same budget: one run completes 24 tokens, the next hangs on the first verify round.
- When the release message says 'the GPU finished' the spin ended at release; when it says 'did not finish within 5 s' the graph was still running 25 s in.
- Frequency scales with
--resident-budget-gib: 12 GiB hung 2/2 runs; 7 GiB roughly 1-in-3.
Ruled out
--sync-every-layer— still hangsSTRATA_ADAPT_NOWAIT=1(main) — still hangs--no-pool— different failure (--spec needs the device residency table), so the pool interleave cannot be fully bypassed- It is not the native dense: run the architecture guard only on the first shard of a split #648 loader issue — that is fixed in these runs (prefill + many decode rounds complete before the hang)
Suspect
Timing-sensitive flag handshake between the CPU expert pool and the verify window's spin kernels — the warm-page-cache runs (fast PLE, fast prefill) hang more often than the first cold run, which completed. Happy to run any instrumented build or capture rocm-smi/rid traces if useful.
Activity
Niko1221 commentedon Oct 3, 2026
Thanks for the careful repro data. The pattern (always a full-attention layer, more often with a larger
--resident-budget-giband a warm page cache, not fixed by--sync-every-layer) points at the handshake between the CPU expert pool and the verify window's GPU waits on HIP. We'll look there and come back with an instrumented build for you to run.verify: opt-in handshake trace for the gfx1030 timeouts, HIP A/B swit…
sycl: port 0.1.39 (re-migration with dpct 2025.3, 3-way merge per doc…
Niko1221 commentedon Oct 4, 2026
0.1.39 adds an opt-in trace for this,
STRATA_VERIFY_TRACE=1. It records:On a timeout it prints all of this before and after the GPU waits are released, so we can see where the GPU stopped and whether it moved again once released. One note from reading your report: the layers you saw (3, 11, 13, 25, 35, 46) aren't all full-attention layers. In this model those are every 4th layer (3, 7, 11, ...), so 13, 25 and 46 are linear-attention layers. We're no longer treating full attention as the common factor.
Could you build 0.1.39 (or main once it's tagged) and run the same command with
STRATA_VERIFY_TRACE=1until it hangs? Please paste thestrata verify traceblock from the log. If you have time, three A/B runs (one setting each) would also help:STRATA_VERIFY_COHERENT=1: the handshake words in explicitly coherent host memory.STRATA_DOORBELL_STORE=1: the GPU stores each layer's ring value instead of a read-modify-write over PCIe.STRATA_RESIDENT_PIN=0with the same--resident-budget-gib 12.Also,
dmesg | grep -iE "amdgpu|kfd|ttm"right after a hang, androcm-smi --showusewhile it hangs (0% busy vs 100% tells a stalled queue from a spinning kernel).Merge Hardin22/Strata-DualGPU (v0.1.38-dual) onto 0.1.39 with the for…
jollyroger1480 commentedon Oct 4, 2026
Update from your trace build (thanks — it's a nice instrument): 4/4 clean runs on 0.1.39+ with the trace enabled and our tuned low-RAM flags (STRATA_RESIDENT_PIN=0, STRATA_ADAPT_NOWAIT=1, --resident-budget-gib 7, --kv int8, --expert-cache 2048, ctx 2048), decode 0.22-0.64 tok/s on 31 GB RAM. The hang has not reproduced since we stopped pinning the resident budget (STRATA_RESIDENT_PIN=0): on this 31 GB box mlock-ing the arena was both the slowdown and, we suspect, the trigger — memory pressure stalls the CPU pool thread, which misses the verify handshake window. If that holds up on your side, the signature is worth a look: budget-pinned + low RAM = pool starvation presenting as #267 timeouts. (Separately: your fix/split-shard-arch-guard from #648 is what lets this abliterated re-split load at all — still needed on latest main.)
Merge the dual-GPU fork into main
verify: opt-in handshake trace for the gfx1030 timeouts, HIP A/B swit…
sycl: port 0.1.39 (re-migration with dpct 2025.3, 3-way merge per doc…
verify: opt-in handshake trace for the gfx1030 timeouts, HIP A/B swit…
sycl: port 0.1.39 (re-migration with dpct 2025.3, 3-way merge per doc…
verify: opt-in handshake trace for the gfx1030 timeouts, HIP A/B swit…
sycl: port 0.1.39 (re-migration with dpct 2025.3, 3-way merge per doc…
docs: troubleshooting for #781 #831, #649, #961 and #654
docs: troubleshooting for Niko1221#781 Niko1221#831, Niko1221#649, Ni…
xoykor commentedon Oct 10, 2026
Additional repro: 6700XT gfx1031 (runs as gfx1030, ROCm gfx103X-all wheels), CachyOS, 32GB RAM, coder-IQ1_M, engine 0.1.41 commit fb58e0d, --max-context 65536 --kv q4_0 --mmap-experts --resident-budget-gib 12 (self-caps to 10.82-12.00 by RAM) --kv-resident 32768 --expert-cache auto --prefill auto, env PIN=0 HEADROOM=6 HSA_USERPTR_FOR_PAGED_MEM=0 STRATA_HIP_ADAPT_KERNEL_COPY=1.
#267 timeouts on layers 9, 15, 38 and #29 watchdog on 45 (all GDN; QSA here are 3,7,...,47). Layer-45 stall report: host step45 GPU rang45 served45 planA45 copiesB45, pool 36/36 done 11 parked, host idle 1.7-4s, disk D 0/36, mem 15.4GB resident 9GB avail then 14.4/6.6GB, 7993 faults. 5s drain fail every time (exit -15), once abort -6. VRAM free 318-426MiB throughout. HSA and ADAPT_KERNEL changed nothing (ADAPT active confirmed in log). Continuing with STRATA_VERIFY_TRACE=1 to send the trace block.
xoykor commentedon Oct 10, 2026
Workaround that stabilized it here (6700XT gfx1031, CachyOS, 32GB RAM, coder-IQ1_M, engine 0.1.41, 64k): #267 timeouts on layers 1/9/15/38/45 plus the #29 watchdog on 45 are gone.
Final config: --max-context 65536 --kv q4_0 --mmap-experts --resident-budget-gib 12 (self-caps to 7-12 by RAM) --kv-resident 32768 --expert-cache auto --prefill auto --no-prefill-borrow --spec 4 --spec-min-p 0.5 --mtp rt --adapt-every 0 --adapt-swaps 0, env PIN=0 HEADROOM=6 HSA_USERPTR_FOR_PAGED_MEM=0 STRATA_HIP_ADAPT_KERNEL_COPY=1.
Measured: 13.3k prompt at 239.9 tok/s, 14.6k + 8192 generated at 10.3 tok/s, KV streaming 99.9% VRAM, 0 exchanged with the VRAM tier, no timeout since the switch. Note: --spec 0 does not start on this IQ pack (needs --spec >= 2), so spec stayed on; what removed the hang was no prefill borrow plus zeroed adaptive swaps.
Niko1221 commentedon Oct 13, 2026
Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.
If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.
jollyroger1480 commentedon Oct 14, 2026
Update after running a 0.1.44.1-based build (our port branch rebased onto the tag) on the same box and model as the original report (Qwen3.8-Flash-Next Q2_0, 31 GB RAM):
STRATA_RESIDENT_PIN=0stays in our config per the Oct 4 finding, and Windows: the verify-window stall wedges the GPU into an unrecoverable "device lost" state (manual power cycle required), with no TDR logged #267 has not recurred since — the engine has since taken several long-prompt ingests (8k-16k tokens) and generations across repeated server restarts.If it recurs on this build we'll reopen with the
STRATA_VERIFY_TRACEblock per your instructions. Thanks for the 0.1.39 trace instrument — that's what surfaced the PIN connection for us.verify: STRATA_VERIFY_DEFER, review: the zeroed rows by a kernel, the…
Verify profile: time the window routing apart from the plan/doorbell;…