[bong-water-water-bong]

Adds an HRX correctness gate that runs on the Strix Halo GPU before merge (ARCH-GAPS goal P2, gap G2). ci.yml has no GPU, and the HRX bugs #308 #309 #310 #313 #318 #319 #369 #370 were all found by hand.

Workflow: .github/workflows/hw-gate.yml, job hw_gate, on the self-hosted runner strixhalo-engine. It runs for:

• same-repo PRs labelled hw-gate;
• every bump-hrx/* PR;
• by hand (workflow_dispatch with a ref; posts the hw-gate commit status).

Fork PRs never run. Actions are SHA-pinned, permissions are contents: read, plus statuses: write for the by-hand status.

Gate (scripts/hw-gate.sh): builds the pinned HRX pair (same configuration as cmake/hrx.cmake), then:

• test-backend-ops -b HRX0 on MUL_MAT, MUL_MAT_ID, FLASH_ATTN_EXT, GATED_DELTA_NET, SSM_CONV, GET_ROWS, ROPE, RMS_NORM and SOFT_MAX, in one run. The known HRX: intermittent MUL_MAT f32 m=1 n=1 k=2048 error on common.mul_mat.f32_f32_decode (about 1 in 40 runs at fork tip e44c9d0) #370 flake is re-run alone 3x.
• tools/kat.py check on Qwen3-0.6B, ZAYA1-8B and Qwen3.8-27B against goldens recorded at the release pin (main d2a56a4, llama.cpp 0e255d77, hrx-system 98d05d94).
• KLD against stored base logits (llama-perplexity --kl-divergence, 512 tokens x 8 chunks).
• The 4.7K-token needle (tests/hrx_long_prompt.py).

Box rules: each GPU step (ops, then one per model) holds ~/npu.lock only while it runs. Before starting, each step:

• waits for Tctl <= 75 C;
• waits for coder-llm (:8090) to be idle;
• passes a tools/mem_budget.py pre-check that refuses anything reaching the engine#315 GTT zone (>= 95 GiB) or MemAvailable.

Each step then runs under scripts/mem-guard.sh and ~/lb/thermal-run.sh. New amdgpu ring timeouts are reported. Running on the pr-agent runner means no PR-Agent review generates during the gate.

Goldens:

• kat goldens are in tests/golden/hw-gate/.
• The KLD bases (0.6-1 GB each) stay on the gate box under ~/.cache/hw-gate/kld, pinned by sha256 plus model and text fingerprints in tests/golden/hw-gate/manifest.json. The gate refuses a mismatch. Spare copy: ryzen ~/backups/hw-gate-kld-0e255d77.

Fixes included:

• tools/kat.py (ours, copied from amd-oss with the header): every determinism repeat now sends cache_prompt: false. The fresh-vs-cached path is a separate check: a WARN with the diverging token and the top-2 margin on a near-tie, a FAIL when the margin is > 1.0 or the cached run loses an answer the fresh run got. A case the golden itself misses is held to the golden text.
• tests/hrx_long_prompt.py passes enable_thinking through chat_template_kwargs. At the release pin the current llama-server ignores the top-level flag, so Qwen3 spent all 16 tokens thinking and the probe got an empty reply. With the kwarg it answers 7341.
• New ctest hw_gate_judges: kat and the KLD judge against a fake server and synthetic logs, no GPU.

Measured on main d2a56a4 (shared box, about ten GPU jobs queueing on the lock):

step	GPU lock held
build, incremental / cold	6 s / about 20 min (no lock)
ops: 611/611 pass	269 s
Qwen3-0.6B kat + needle	67 s
ZAYA1-8B kat	172 s
Qwen3.8-27B kat	213 s
KLD record 0.6B / 8B / 27B	6 / 24 / 29 s

That is about 13 min of GPU time. Each lock acquisition waited 10-60 min, because flock is not FIFO; the summary reports waiting and holding separately.

Pending a lock window (the release-blocking fixes have GPU priority): 3 clean full runs on main, and the revert test. The scratch build that re-introduces #319 (fork b4e7515 reverted on 0e255d77, worktree strixhalo ~/wt/hw-gate-revert319, never pushed) is built and ready for scripts/hw-gate.sh --bin ~/wt/hw-gate-revert319/build-gate/bin. After that, the hw-gate label goes on this PR for the CI run. Not for merge before those results.

🤖 Generated with Claude Code

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit f90ec26)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 308 - Partially compliant Compliant requirements: The PR introduces a hardware gate that runs known-answer tests (including chat-template probes) on HRX The gate includes checks for models like Qwen3-Coder-30B-A3B (via kat.py) The gate runs on Strix Halo hardware, which is the environment where the issue was found Non-compliant requirements: The PR does not explicitly mention checking the chat template and tool-call parsing for Coder in this build The PR does not explicitly compare with the CPU backend Requires further human verification: The specific chat template and tool-call parsing behavior for Coder needs to be verified manually The CPU backend comparison needs to be validated by running the same tests on CPU 309 - Partially compliant Compliant requirements: The PR introduces a hardware gate that runs needle probe tests (including the 4.7K-token needle) on HRX The gate includes checks for ZAYA1-8B model The gate runs on Strix Halo hardware where the issue was found Non-compliant requirements: The PR does not explicitly mention running the same probe on the CPU backend The PR does not explicitly mention running with the F16 GGUF to separate model capability from HRX long-context issue Requires further human verification: The CPU backend comparison needs to be validated by running the same tests on CPU The F16 GGUF test needs to be run separately to confirm model capability vs HRX issue 310 - Partially compliant Compliant requirements: The PR introduces a hardware gate that runs KLD (KL divergence) checks against stored base logits The gate includes checks for models like GLM-4.7-Flash that are affected by the MLA decode path drift The gate runs on Strix Halo hardware where the issue was found Non-compliant requirements: The PR does not explicitly mention adding a repeated-request check with random data in masked KV cells Requires further human verification: The repeated-request check with random data in masked KV cells needs to be implemented and validated

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Memory Estimation Inaccuracy The mem_budget.py tool estimates memory usage for HRX models by summing weights, KV state, compute buffer, and program cache. However, it uses a fixed compute buffer size (512 MiB) and program cache size (4 GiB) without considering that these values might vary based on model characteristics or runtime conditions. This could lead to underestimating or overestimating memory requirements, potentially causing memory allocation failures or unnecessary rejections of valid model loads. compute = int(a.compute_mib * MIB) progcache = int(a.program_cache_gib * GIB) Cache Path Divergence Handling The cache path divergence check in kat.py compares fresh and cached runs, but it may not fully account for legitimate differences in rounding behavior between the prefill and decode paths. The current implementation flags a divergence as a failure if the margin is above --cache-margin (1.0) or if the cached run misses an answer the fresh run got. However, it doesn't distinguish between cases where the divergence is due to kernel rounding differences (which may be acceptable) versus cases where it indicates a real correctness issue. if m is not None and m > a.cache_margin: return [], [msg + f": not a near-tie (> {a.cache_margin})"], detail return [msg], [], detail Potential Race Condition in GPU Step Execution In scripts/hw-gate.sh, the GPU step execution uses flock to acquire the lock, but there's a potential race condition where the memory check (memcheck) could pass while the actual GPU step is running, leading to a situation where the GPU step might still exceed memory limits. The memory pre-check is done before the lock is acquired, but the actual GPU step might take longer than the pre-check duration, potentially causing memory issues. flock -w "$LOCK_WAIT" -E 75 "$LOCK" "$0" --cool --mem "${GPU_MODEL:--}" "$ROOT/scripts/mem-guard.sh" "$MEM_FLOOR" "$THERMAL" "$@"

---

⚠️ Review coverage: The following files were not included in this review because of the token budget:

• tests/golden/hw-gate/kat-zaya1-8b.json
• tests/golden/hw-gate/kat-qwen3-0.6b.json
• docs/hw-gate.md
• .github/workflows/hw-gate.yml
• tests/golden/hw-gate/manifest.json
• README.md
• CMakeLists.txt

[github-actions]

Persistent review updated to latest commit f90ec26