Skip to content

Latent MPI deadlock: spring smoother returns early on rank-local n_free==0 before collectives #346

Description

@lmoresi

Found (marker-only, Charter §9) during the Wave D smoothing readability pass, 2026-07-08. A TODO(BUG) marker lands with PR #345 in src/underworld3/meshing/smoothing/spring.py.

The spring-equilibrium mover takes a rank-local early return when a rank has no free (unpinned) nodes, BEFORE reaching the collective reduction calls later in the routine. On a partition where some rank owns only pinned/boundary nodes (small partitions, heavily graded meshes at high rank counts), that rank exits while the others enter the allreduce — a classic collective mismatch deadlock.

Not observed in the campaign's np2 gates (both ranks own free nodes in the test meshes); the hazard is latent and partition-dependent.

Suggested fix: replace the early return with a zero-work path that still participates in every collective (empty contribution), or hoist the collectives above the return. Needs an np>2 test with a deliberately starved rank.

Related: the Wave D smoothing PR #345 (marker placed, behaviour untouched per readability-wave scope discipline).

Underworld development team with AI support from Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions