fix(constrained): guard empty stratum IS in interior-multiplier reduction (#291)#318
Merged
Merged
Conversation
`ksp.solve` never raises on `KSP_DIVERGED_*`, so the two linear paths in `rotated_bc` (the default fieldsplit-Schur iterative solve at `_solve_rotated_iterative` and the opt-in direct MUMPS LU at `solve_rotated_freeslip`) both returned partial answers silently. That silence is what let the 3D rotation-nullspace bug fixed by #306 look like a working solve for as long as it did: the outer Krylov hit its `ksp_max_it` ceiling (300 in the pre-#306 code) with a residual well above tolerance, and neither the KSP nor the caller flagged it. The solution then propagated into downstream diagnostics that eventually exposed it as wrong physics rather than as a solver failure. Add a small `_warn_if_ksp_diverged` helper and call it immediately after each of the two `ksp.solve` sites. Rank-0 warning only (via `uw.mpi.pprint`), same style as the existing nonlinear-driver warning at the end of `solve_rotated_freeslip_nonlinear`. No behaviour change on the happy path; non-convergence is now loud instead of silent. test_1018_rotated_freeslip.py: 14/14 pass (2D behaviour unchanged). Underworld development team with AI support from Claude Code
…erged Copilot review on #315 noted the convention split: * KSP.getConvergedReason() > 0 → converged * KSP.getConvergedReason() == 0 → KSP_CONVERGED_ITERATING (initial state; unexpected after ksp.solve() returns) * KSP.getConvergedReason() < 0 → diverged The previous `reason >= 0` early-return silently accepted a hypothetical `reason == 0` outcome, undermining the guard's purpose. Change to `reason > 0` so a stray `KSP_CONVERGED_ITERATING` also fires the warning, matching the project convention documented in petsc_generic_snes_solvers.pyx:2979. test_1018_rotated_freeslip.py: 14/14 pass (unchanged). Underworld development team with AI support from Claude Code
…tion (#291) `Stokes_Constrained` segfaulted at np>1 in `_constrain_interior_multipliers_in_section`: on a rank owning zero points with a given boundary label value, `dm.getLabel("UW_Boundaries").getStratumIS(bvalue)` returns a VALID PETSc IS with `getSize() == 0` — not None, non-zero handle, `bool(IS) == True`. The existing `if bd_is is not None:` guard passed through, and the subsequent `bd_is.getIndices()` crashed with SIGSEGV in the PETSc IS binding. The failure was reliably reproduced by #291's repro (StructuredQuadBox + free-slip on all four walls, np=2) and by a lighter unit-viscosity variant which now solves bit-identically at np=1/2/4 with the fix. Fix: replace `if bd_is is not None:` with `if bd_is and bd_is.getSize() > 0:`. `bool(bd_is)` short-circuits the null-handle case (project convention, see `utilities/boundary_flux.py:70`); `.getSize() > 0` catches the valid-but-empty case that neither the None check nor the truthiness check protects against. ### Verification Lightweight unit-viscosity repro (16x16 quad, sinusoidal body force, free-slip on all four walls; `_reduce_interior_multiplier = True` default): np=1: |u|_inf = 2.533e-02 (6.6 s) np=2: |u|_inf = 2.533e-02 (8.5 s) [previously: SIGSEGV] np=4: |u|_inf = 2.533e-02 (5.0 s) [previously: SIGSEGV] Bit-identical parallel-vs-serial in this regime. `tests/test_1062_constrained_solcx.py` (canonical SolCx eta_B=1e6 test): serial passes (198 s); np=2 with-mpi rerun exceeds the default 300 s timeout under this session's compute — a longer-timeout parallel check is warranted as follow-up (probably belongs in `tests/parallel/`). Underworld development team with AI support from Claude Code
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a parallel crash in Stokes_Constrained interior-multiplier reduction by correctly guarding against valid-but-empty PETSc stratum IS objects (a case that occurs when a rank owns zero points for a given boundary value). Also adds explicit rank-0 warnings when the rotated free-slip helper KSPs diverge, preventing silent “last-iterate” returns.
Changes:
- Widened the
getStratumISguard in_constrain_interior_multipliers_in_sectionto skip empty strata and avoidgetIndices()segfaults in MPI. - Added
_warn_if_ksp_diverged()and invoked it in both direct-LU and fieldsplit-Schur rotated free-slip solve paths.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/underworld3/cython/petsc_generic_snes_solvers.pyx |
Prevents MPI segfault by guarding empty boundary stratum IS before calling getIndices(). |
src/underworld3/utilities/rotated_bc.py |
Adds rank-0 warnings on KSP divergence to avoid silently returning a partial rotated free-slip solution. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
6999
to
7007
| bd_is = dm.getLabel("UW_Boundaries").getStratumIS(bvalue) | ||
| # Guard for parallel: a rank owning no points with this label value | ||
| # gets back a valid non-None PETSc.IS with size 0 (bool(bd_is) = True, | ||
| # bd_is.handle != 0). Calling .getIndices() on that IS segfaults. | ||
| # `if bd_is` catches the null-handle case; `.getSize() > 0` catches | ||
| # the valid-but-empty case that None / handle checks miss (issue #291). | ||
| keep = set() | ||
| if bd_is is not None: | ||
| if bd_is and bd_is.getSize() > 0: | ||
| for bp in bd_is.getIndices().tolist(): |
…llel test Copilot review on #318 rightly pointed out that a parallel-only SIGSEGV regression should not merge without a re-enabled parallel test. Two changes: ## 1. New lightweight parallel regression (tests/parallel/test_1062_constrained_stratum_guard_parallel.py) Unit-viscosity 16×16 quad box, sinusoidal body force, free-slip on all four walls at np≥2. On a 2-way axis-aligned partition (the default) this puts one full boundary on each rank and zero points on the other — the exact empty-stratum configuration that #291's guard now handles. Runs in ~10 s at np=2 (vs the canonical test_1062 SolCx eta_B=1e6 which timed out at 300 s in this session's parallel runs, per #244). Asserts `|u|_inf` matches the serial GOLDEN of 2.53305e-02 to rtol 1e-3 (actual serial-vs-parallel spread on this workload is < 1e-9). ## 2. Un-skip test_parallel_custom_fmg_stokes_constrained (tests/parallel/test_1017_custom_mg_parallel_mpi.py) Its skip reason explicitly cited "#291; auto-enables once #291 is fixed." The guard fix is now in place, so the test can run. Marked `slow` (SolCx eta_B=1e6 is not a fast benchmark) so CI selects it via `pytest -m slow` or `-m "not slow"` per environment. Underworld development team with AI support from Claude Code
Member
Author
|
Thanks @copilot-pull-request-reviewer — added in 958f6f4:
|
lmoresi
added a commit
that referenced
this pull request
Jul 5, 2026
… IS (#319) (#321) Same class of latent parallel segfault as #291 (PR #318), at four more sites in `boundary_flux.py` and `rotated_bc.py`: * `boundary_flux.py:70` — in `_boundary_field_nodes` * `boundary_flux.py:106` — in `_boundary_reaction_by_facet` * `boundary_flux.py:154` — in `_boundary_facet_elements` * `rotated_bc.py:78` — in `_boundary_velocity_nodes` Each uses the guard if sis is None or sis.handle == 0: ... which catches None and null-handle IS values, but MISSES the case that crashed `_constrain_interior_multipliers_in_section` in #291: on a rank owning zero points with a given boundary label value, `dm.getLabel("UW_Boundaries").getStratumIS(bvalue)` returns a valid non-None PETSc IS with `getSize() == 0`, non-zero `handle`, and `bool(IS) == True`. The subsequent `sis.getIndices()` segfaults on that IS. These sites fire in: * `add_nitsche_bc` (writes go through boundary_flux), * `add_rotated_freeslip_bc` (rotated_bc), * Consistent-Boundary-Flux traction / heat-flux / Nusselt recovery, * σ_nn / dynamic-topography hand-off. Empirically none has bitten yet because most 2D partitions place at least one point of each boundary on each rank. Fine decompositions or axis-aligned splits on axis-aligned walls (exactly the #291 shape) can still hit it. Fix: replace each guard with if not (sis and sis.getSize() > 0): ... `bool(sis)` short-circuits the null-handle case (project convention); `.getSize() > 0` catches the valid-but-empty case that the None / handle checks miss. test_1018_rotated_freeslip.py: 14/14 pass. test_1019_boundary_flux.py: 1/1 pass. No behaviour change on the happy path. Underworld development team with AI support from Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #291.
Summary
`Stokes_Constrained` segfaulted at np>1 in
`_constrain_interior_multipliers_in_section`
(`src/underworld3/cython/petsc_generic_snes_solvers.pyx:7001`).
The root cause was the boundary-IS guard. On a rank that owns zero points
with a given boundary label value,
`dm.getLabel("UW_Boundaries").getStratumIS(bvalue)` returns a valid PETSc
IS with `getSize() == 0` — not None, non-zero handle, and
`bool(IS) == True`. So the existing guard `if bd_is is not None:` passed
through, and the subsequent `bd_is.getIndices()` crashed with SIGSEGV.
Change
`petsc_generic_snes_solvers.pyx:7001` — replace
```python
if bd_is is not None:
```
with
```python
if bd_is and bd_is.getSize() > 0:
```
`bool(bd_is)` short-circuits the null-handle case (the same idiom
`utilities/boundary_flux.py:70` uses); `.getSize() > 0` catches the
valid-but-empty case that neither the None check nor the truthiness
check protects against.
Verification
Lightweight unit-viscosity repro (16×16 quad, sinusoidal body force,
free-slip on all four walls; `_reduce_interior_multiplier = True` default):
| np | `|u|_inf` | wall | pre-fix |
|----|-------------|-------|---------------|
| 1 | 2.533e-02 | 6.6 s | (unchanged) |
| 2 | 2.533e-02 | 8.5 s | SIGSEGV |
| 4 | 2.533e-02 | 5.0 s | SIGSEGV |
Bit-identical parallel-vs-serial in this regime.
`tests/test_1062_constrained_solcx.py` (canonical SolCx `eta_B=1e6` test):
serial passes (198 s); the np=2 with-mpi rerun exceeded the 300 s timeout
under this session's compute (`Stokes_Constrained` is 34× slower than
Nitsche per #244), so a longer-timeout parallel variant of this test
(probably `tests/parallel/test_1062_constrained_solcx_parallel.py`) is
warranted as follow-up. Not blocking — the guard is a strict guard-widening
that cannot produce wrong physics; it only stops a previously-crashing rank
from calling `.getIndices()` on an empty IS.
Related
`test_1062` np=2 fitting a 300 s timeout on this machine.
Underworld development team with AI support from Claude Code