[Bug] Multi-rank comm_init fails in os.fork()'d chip worker — rank 0 dies during HCCL bring-up, version-invariant #1018

Repository: hw-native-sys/simpler

Re-filed after the previous issue was closed. This version isolates the
failure to simpler's own in-tree example (no pypto-lib / model code) and
adds a version-invariance matrix that rules out CANN, simpler, Python, and
PTOAS versions as the cause. Net: the bug tracks the os.fork() + raw
HcclGetRootInfo chip-worker bootstrap, not any dependency version.

Summary

Any multi-rank Worker(level=3) dispatch that reaches _ensure_comm_base
crashes during HCCL base-communicator bootstrap:

• rank 0 (the rootinfo writer) dies inside comm_init
  (simpler/task_interface.py:589 → C++ ChipWorker::comm_init →
  comm_hccl.cpp HcclGetRootInfo) before it writes the rootinfo
  handshake file.
• every other rank then reports
  [ERROR] comm_init: [comm_hccl.cpp:286] [comm rank N] Timeout waiting for rootinfo
  after HCCL_CONNECT_TIMEOUT, because rank 0 never produced the file.

This blocks every multi-rank L3 deployment in our environment.

It reproduces with simpler's own l3/allreduce_distributed example — no
model, no pypto-lib, and the parent process does no NPU/ACL work before
forking the chips.

Minimal reproducer (simpler in-tree example only)

# venv with simpler installed, pto-isa available, >=2 NPUs, ptoas on PATH
export PTO_ISA_ROOT=/path/to/pto-isa
cd /path/to/simpler/examples/workers/l3/allreduce_distributed   # in-tree example
PYTHONFAULTHANDLER=1 python -X faulthandler main.py -p a2a3 -d 0-1

Observed:

[chip_process pid=14062 dev=0] ready
[chip_process pid=14064 dev=1] ready
Fatal Python error: Segmentation fault
Current thread (most recent call first):
  File ".../simpler/task_interface.py", line 589 in comm_init
  File ".../simpler/worker.py", line 732 in _handle_ctrl_comm_init
  File ".../simpler/worker.py", line 937 in _run_chip_main_loop
  File ".../simpler/worker.py", line 999 in _chip_process_loop
  File ".../simpler/worker.py", line 1959 in _start_hierarchical
  File ".../simpler/worker.py", line 2573 in run
  ...
  File ".../examples/workers/l3/allreduce_distributed/main.py", line 208 in run
[ERROR] comm_init: [comm_hccl.cpp:286] [comm rank 1] Timeout waiting for rootinfo

In main.py, the parent only builds CPU torch tensors and compiles kernels
before worker.init() forks the chips (the base comm is lazy). Verified that
import torch does not load libhccl/libascendcl in the parent
(checked /proc/self/maps) — so the parent is clean at fork time.

Crash locus (rank-0 Ascend plog)

The forked rank-0 chip sets its device and brings up ACL/runtime cleanly, then
dies right at the start of HCCL bring-up — last lines before SIGSEGV:

DRV ... Setup device succeeded. (logical_devid=0; devid=8; ...)
RUNTIME ... StarsEngine: Constructor.
ASCENDCL ... GetAllPackageVersion: Version of hccl package is 9.0.0.beta1
ASCENDCL ... GetAllPackageVersion: Version of driver package is not found   <-- last line, then SIGSEGV

No HcclGetRootInfo completion; no rootinfo file written. The crash is inside
the HCCL library call path, in the os.fork()'d child.

Version-invariance matrix (the decisive evidence)

The same minimal repro was run across four independent version axes. Every
combination fails identically (rank 1 Timeout waiting for rootinfo, rank 0
gone):

For the 8.5.1 row we confirmed the chip actually loaded the 8.5.1 HCCL (not a
stale 9.0.0 one) via the live process map:

$ grep libhccl /proc/<rank1-chip-pid>/maps
/usr/local/Ascend/cann-8.5.1/x86_64-linux/lib64/libhccl.so

So CANN version, simpler version, Python version, and PTOAS version are all
ruled out. The only invariant is the chip-worker bootstrap: a process created
by os.fork() (worker.py:1935/1955/2002, lazily at Worker.init()) calling
raw HcclGetRootInfo from comm_hccl.cpp.

Control: vLLM (spawn) works on the identical hardware/CANN

On the same pod, same 8 cards, same CANN 9.0.0-beta.1, same driver 25.5.1,
vllm serve <model> -tp 8 (vLLM-Ascend, which uses torch.distributed
init_process_group("hccl") with spawned workers) initializes HCCL across
all 8 ranks and serves cleanly. So the driver / HCCL / hardware stack is
healthy; the differentiator is how the rank process is created — vLLM
spawns fresh processes, simpler forks.

Root-cause hypothesis

HcclGetRootInfo / HCCL bring-up is not safe to call from an os.fork()'d
child under these CANN builds, even when the parent never touched HCCL/ACL
before the fork. vLLM avoids it by spawning fresh worker processes.

Candidate fixes (all upstream / simpler-side):

1. Spawn (or fork+exec) chip workers for the HCCL/comm path instead of
   bare os.fork(), so each rank brings up HCCL in a clean process.
2. Guarantee the chip is forked before any ACL/HCCL/driver state is loaded
   in the parent, and that the child performs a complete clean ACL init before
   HcclGetRootInfo — then document/verify that ordering invariant.
3. Provide a distributed L3 ST that exercises cw.comm_init end-to-end from a
   forked chip on CANN 8.5.1 / 9.0.0 + driver 25.5.1, so this is covered in CI.

What we ruled out

• CANN version — fails on both 9.0.0-beta.1 and 8.5.1 (8.5.1 libhccl load verified).
• simpler version — fails on 6e84154d and latest main afb5c5a9.
• Python version — fails on 3.10 and 3.11.
• PTOAS version — fails on v0.43 and v0.44 (PTOAS is not in the comm path).
• Parent pre-initialized HCCL — the allreduce parent does no NPU work before fork; import torch does not load libhccl/libascendcl.
• Missing HCCL env — HCCL_CONNECT_TIMEOUT=3600, HCCL_INTRA_ROCE_ENABLE=1, HCCL_INTRA_PCIE_ENABLE=0, HCCL_OP_EXPANSION_MODE=AIV all set; HCCL_SOCKET_IFNAME/HCCL_IF_BASE_PORT overrides change nothing.
• NPU contention — cards are exclusively assigned (private pod); verified free before each run.

Environment

Note on a symbolized C++ backtrace

We could not produce a gdb/core backtrace in our pod: gdb is not installed,
ulimit -c is hard-capped at 0, and core_pattern pipes to apport (not
running in-container). The Ascend ATRACE SIGSEGV handler did not leave a
symbolized stack in plog either. Happy to capture one if you can advise a
symbol build / point us at a gdb-enabled image.