Repository navigation
[Bug] test_nvfp4_gather_grouped_gemm_act_fusion_blackwell flaky on B300: ~0.05% mismatch vs rtol 1e-4, varying parametrizations #16281
Description
Summary
unittest/_torch/thop/parallel/test_cute_dsl_moe.py::test_nvfp4_gather_grouped_gemm_act_fusion_blackwell fails intermittently in pre-merge stage B300-PyTorch-2 with:
Exception: Mismatch percentage is ~0.05 for rtol 0.000100
Observed twice on unrelated PR #15730 with different parametrizations:
[8192-8-1-256-relu2]— L0_MergeRequest_PR build 47303 (mismatch 0.050421)[128-1-32-128-relu2]— L0_MergeRequest_PR build 47362 (mismatch 0.054688)
Both runs were on current main bases; the test passes in other runs, so this is a marginal-tolerance flake on B300 rather than a deterministic regression.
Impact
Pre-merge blocker (fail-fast) for unrelated PRs when B300-PyTorch-2 is selected and the tolerance coin-flip loses.
Ask
Loosen rtol / deflake the kernel comparison for B300, or waive until fixed.
Activity
[None][chore] Temporarily waive test_nvfp4_gather_grouped_gemm_act_fu…
[None][chore] Temporarily waive test_nvfp4_gather_grouped_gemm_act_fu…
lucifer1004 commentedon Oct 20, 2026
@Wanli-Jiang @zongfeijing could you take a look at this one? It still blocks unrelated PRs, and it now shows up on B200 as well as B300.
Over the last two days the test failed once on each of four unrelated PRs in B200 pre-merge CI. The parametrization was different each time, and every failing case is
relu2:[128-1-32-256-relu2][1024-1-32-128-relu2][8192-8-1-256-relu2][1024-1-32-128-relu2][8192-8-1-128-relu2]None of these PRs touch the CuTe DSL NVFP4 MoE kernels or this test, and the same cases pass in other runs.
The check is
check_accuracy(..., atol=1e-4, rtol=1e-4, percent=0.95), which fails when 5% or more of the elements fall outside the tolerance. The printed "Mismatch percentage" is a fraction, so these runs had 5.0–5.4% of elements outside it, just over the 5% budget. That is the same band as the two B300 failures in the description (0.050421, 0.054688). It looks like the budget sits at the edge of the kernel's normal run-to-run spread rather than a regression.Would you consider either waiving the
relu2cases until the comparison is deflaked, or setting the budget from the observed spread (for examplepercent=0.93, or a somewhat largeratol/rtolfor therelu2path, if that is acceptable for this kernel)? Each failure aborts the rest of the pre-merge run through fail-fast, so every hit costs the affected PR a rerun.