[Xuan-1998]

The Gin barrier flushed every context without a timeout. It now goes
through flush_with_timeout, so a NIC that never completes its writes
prints a flush timeout and traps instead of spinning.

The combine scale-up wait prints one timeout line per scale-up rank
instead of one per lane, and the epilogue drain timeout names the ranks.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[whn09]

Tested on 2× p6-b300 (NCCL 2.31.2). Full matrix + pressure pass; perf flat. With EP_DEBUG_CHECKS=1 on a227152's stale-header hang it prints grid sync timeout (barrier start) ... SMs arrived: 10 of 12, which does point at the stuck SMs. One note: after a timed-out grid sync, the workspace counter is left mid-round, so every later grid sync on that buffer will hang (debug-only, but worth a comment). The EP_DEBUG_CHECKS line in compiler.hpp is the same as #33's, and the barrier signature change conflicts with #13 and #29.

[akkart-aws]

I think overflow should still not be an issue, if we use this
cudaDeviceSetLimit
to set the size of the printf buffer. But according to the documentation, this only works
before the first kernel that uses printf is launched. So if the user application launches
any kernel that uses printf before buffer creation, this fails, but the buffer is normally
created before any such kernel, so in practice it should work, and we can print a warning
when it does fail.

[Xuan-1998]

I think overflow should still not be an issue, if we use this cudaDeviceSetLimit to set the size of the printf buffer. But according to the documentation, this only works before the first kernel that uses printf is launched. So if the user application launches any kernel that uses printf before buffer creation, this fails, but the buffer is normally created before any such kernel, so in practice it should work, and we can print a warning when it does fail.

Thanks Arun. Yes we can, just a bit worried when the user is not setting it to be enough and then we need to ask them to set and run the logs for us etc. When the log buffer is overflow it just silently drops the earliest ones.

offline currently