Summary
tests/parallel/test_0760_swarm_cache_migration.py hangs at mpirun -np 4 on current development (pre-existing — verified on a pristine baseline during the gating of #313, by stashing that branch's src changes and rebuilding).
Mechanism
With rank-biased particle coordinates, global_evaluate leaves some ranks with zero interior points. Those ranks diverge inside the DMInterpolation machinery in petsc_interpolate while the remaining ranks block in its collectives — an MPI deadlock. It passes at np2, which is why CI (test.sh --p 2) never sees it.
Per-rank stack traces were captured during #313's gating and confirm the hang is inside the DMInterpolation path; the swarm-cache fixes in #313 enter and exit their collective hooks symmetrically before the hang (i.e. #313 neither causes nor masks this).
Relation to the 2026-07 quality audit
Corresponds to the provisional findings SWARM-15/16 in docs/reviews/2026-07/SWARM-SUBSYSTEM-REVIEW.md (parallel-evaluation empty-rank asymmetry family; see also BF-16 in the remediation worklist — substep advection evaluating on the wrong rank).
Reproduction
mpirun -np 4 python -m pytest tests/parallel/test_0760_swarm_cache_migration.py on development — hangs. Same at np2 passes.
Underworld development team with AI support from Claude Code
Summary
tests/parallel/test_0760_swarm_cache_migration.pyhangs atmpirun -np 4on currentdevelopment(pre-existing — verified on a pristine baseline during the gating of #313, by stashing that branch's src changes and rebuilding).Mechanism
With rank-biased particle coordinates,
global_evaluateleaves some ranks with zero interior points. Those ranks diverge inside theDMInterpolationmachinery inpetsc_interpolatewhile the remaining ranks block in its collectives — an MPI deadlock. It passes at np2, which is why CI (test.sh --p 2) never sees it.Per-rank stack traces were captured during #313's gating and confirm the hang is inside the DMInterpolation path; the swarm-cache fixes in #313 enter and exit their collective hooks symmetrically before the hang (i.e. #313 neither causes nor masks this).
Relation to the 2026-07 quality audit
Corresponds to the provisional findings SWARM-15/16 in
docs/reviews/2026-07/SWARM-SUBSYSTEM-REVIEW.md(parallel-evaluation empty-rank asymmetry family; see also BF-16 in the remediation worklist — substep advection evaluating on the wrong rank).Reproduction
mpirun -np 4 python -m pytest tests/parallel/test_0760_swarm_cache_migration.pyon development — hangs. Same at np2 passes.Underworld development team with AI support from Claude Code