fix(diag): print the NIC queue state next to every timeout by Xuan-1998 · Pull Request #29 · amazon-contributing/DeepEP

Xuan-1998 · 2026-10-22T23:45:52Z

A timeout says that a peer's data or signal did not arrive, not whether this GPU's own NIC delivered what it was given. comm::report_gin_queue reads the posted and completed counts of the waiting warp's GIN context from the EFA GDA device handle and prints them after every timeout line of the unordered dispatch and combine, the Gin barrier and the GIN flush. After the kernel timeout every WQE should have completed long ago, so a non-zero count means this NIC or the fabric still holds our writes.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

whn09 · 2026-11-03T08:36:25Z

Tested on 2× p6-b300 (NCCL 2.31.2). Full matrix + pressure pass; perf flat with and without EP_DEBUG_CHECKS. On the a227152 stale-header hang with EP_DEBUG_CHECKS=1, it printed not completed: 0 on all 19 queue-state lines, correctly ruling out the NIC. The output is only there with EP_DEBUG_CHECKS=1; worth saying so in the description. Conflicts with #27 (flush_with_timeout now takes handle::NCCLGin, #27 passes a raw ncclGin) and with #13 (barrier signature).

hershys-aws

Main suggestion is on handle.cuh:216: a timed flush in the report gives the same answer without changing put or the QP it posts on. The rest are smaller: the thread-0 guards can drop the report, the elected lane needs a __syncwarp before the trap, and a few report sites (local waits, barrier) can't say much. Tested on 2x8 H200 with planted faults; happy to share the scripts.

A timeout says that a peer's data or signal did not arrive, not whether this rank's own NIC delivered what it was given. With EP_DEBUG_CHECKS=queue the unordered dispatch and combine, the Gin barrier and the GIN flush timeouts give the waiting warp's GIN context a short timed flush and print whether the NIC had completed every write posted on it, so a NIC that holds our writes can be told from a peer that never sent. The lowest active lane does the flush and the print and the other lanes wait for it, so the report is not lost when another lane traps first; a GIN flush timeout then also names its context and waits a second before trapping. EP_DEBUG_CHECKS is a comma-separated list of check names (`1` or `all` turns on every check, an unknown name stops at buffer creation); the host maps the names to bits before the JIT compile, so the default build carries none of this code. The check is off by default because an inlined gin.flush at every wait site changes the register allocation of the hot kernels even though it never runs.

Xuan-1998 force-pushed the fix/timeout-queue-state branch from 58dfc03 to 2c36506 Compare October 27, 2026 17:29

Xuan-1998 requested review from anshumang and bhasunit and removed request for bhasunit October 27, 2026 17:30

bhasunit reviewed Oct 27, 2026

View reviewed changes

Comment thread deep_ep/include/deep_ep/common/comm.cuh Resolved

hershys-aws previously approved these changes Oct 28, 2026

View reviewed changes

Xuan-1998 dismissed hershys-aws’s stale review via fdd2b68 October 31, 2026 23:36

Xuan-1998 force-pushed the fix/timeout-queue-state branch from 2c36506 to fdd2b68 Compare October 31, 2026 23:36

Xuan-1998 requested review from bhasunit, hershys-aws and vladimiraerov November 1, 2026 00:18