Stokes_Constrained parallel correctness: gauge, convergence, knockout, rotation-gauge#265
Conversation
… audit Three fixes from a focused parallel-correctness investigation of the in-saddle Lagrange-multiplier free-slip solver (Stokes_Constrained), validated against the Zhong (2008) spherical-shell response (3-D, SphericalShellInternalBoundary). Item 1 — consistent pressure gauge (raw-field reproducibility). On an enclosed constrained problem the constant pressure and constant multiplier are independent gauge freedoms; the solver lands on a partition-dependent level for each, so raw pressure/multiplier are not reproducible across ranks. Added a guarded automatic pressure gauge (auto_pressure_gauge, default on) that pins the surface-mean pressure on the first constraint boundary when a pressure null space is active and the user has registered no gauge of their own. It is physics- neutral (velocity bit-identical) and fixes the raw mean pressure (10.5% -> 0.4% serial vs np=8). The pressure pin does NOT fix the raw multiplier (separate gauge freedom): dynamic topography is read gauge-invariantly via topography(reference="mean"). New parallel regression in test_1063. Item 2 — the grouped-Schur solve was false-converging. The grouped u | [p,h] Schur preconditioner uses inner iterative sub-solves (a variable/nonlinear preconditioner). A non-flexible Krylov method (the base's gmres) is invalid for that and false-converged: it reported CONVERGED_RTOL in ~2 iterations on the preconditioned norm while the TRUE residual ||r||/||b|| blew up to ~1e3, landing on a wrong, partition-dependent answer (the ~0.4% velocity spread and the failure to reproduce the Zhong response). Fixed with four defaults in Stokes_Constrained: ksp_type=fgmres (flexible Krylov), ksp_norm_type= unpreconditioned (honest true-residual stopping test), Eisenstat-Walker rtol0=rtolmax=tolerance*0.1 (EW's default 0.3 capped the linear solve at one iteration), and a bounded ksp_max_it. The solve now genuinely converges (true residual ~1e-13) and reproduces Zhong (Ut within 1%, Ub within 0.1%); new test_1064 default-constrained test asserts this. Velocity partition spread 0.36% -> 0.12%. The residual 0.12% is a separate, pre-existing bug: add_natural_bc on an INTERNAL surface assembles a partition-dependent load (the force *function* integrates identically to 1e-16, but the assembled vector differs 0.027% because interior facets at partition seams lack their neighbour support cell on the non-overlapped assembly DM). Marked with TODO(BUG) at the natural-BC assembly; filed in the planning file. Universal (affects Nitsche too), amplified here by the augmented conditioning. Item 3 — interior-multiplier knockout audit. Confirmed the knockout is genuinely lossless (a converged solve with it disabled moves the answer ~1e-8 in velocity, no interior-h blow-up); the previously observed "2-5% when off" is an iterative-conditioning artifact, not lost physics. Rewrote the docstrings accordingly. Added a guard that warns a monolithic direct solve (pc_type lu/cholesky) of the constrained saddle point is a serial diagnostic only (wrong response + segfaults in parallel). Underworld development team with AI support from Claude Code
…p solution For a free-slip (non-Dirichlet) velocity the rigid-body rotations are a true nullspace of the velocity block (A_uu·rotation = 0). The monolithic nullspace attached for the solve is not removed inside the fieldsplit/Schur iteration: the inner velocity KSP has no rotation nullspace, so it leaves an unconstrained rigid rotation in the velocity. The operator is blind to it (true residual still converges to machine precision), but the rotation amplitude is partition- dependent — the tangential velocity differed serial-vs-parallel by ~0.1–1% even at a converged residual, while the physical (radial / gauge-invariant) flow was already partition-clean. SNES_Stokes_SaddlePt.solve() now projects the velocity rotation modes out of the converged solution via MatNullSpaceRemove (_remove_velocity_rotation_gauge). Because the rotation is a genuine nullspace, removing it does not change the residual and yields the same rotation-free solution on any decomposition. This is the velocity analogue of the constant-pressure gauge (set_pressure_gauge). Surgical: only the velocity rotation modes (zero in the pressure/multiplier blocks), so the existing pressure/multiplier gauge handling is untouched; a no-op when there are no rotation modes (e.g. Dirichlet velocity BCs). Validated on the SphericalShellInternalBoundary constrained free-slip problem: the tangential surface velocity serial-vs-np8 relative difference drops from 1.2% to 2.5e-14, with the iteration count unchanged. Regression clean: test_1064 + test_1010 (9 passed, 2 pre-existing xfails) and test_1063 parallel (np=2) all pass. Also refines the TODO(BUG) at the natural-BC assembly loop to document the underlying PETSc interior-facet support[0] partition-dependence and the workarounds that were tested and ruled out. Underworld development team with AI support from Claude Code
Adds a fast serial tier-a/level-1 test that solves a fully free-slip annulus (both boundaries constraint BCs, no Dirichlet velocity BC -> rigid-rotation nullspace active) driven by a purely radial body force, and asserts the converged velocity carries no rigid-rotation component. The existing test_1063 cases all pin the rotation nullspace with a Dirichlet BC, so they did not cover this path; this test fails without _remove_velocity_rotation_gauge (the rotation coefficient is ~1e-3) and passes with it (~4e-15). Underworld development team with AI support from Claude Code
There was a problem hiding this comment.
Pull request overview
This PR improves parallel correctness and solver robustness for the constrained free-slip Stokes solver (SNES_Stokes_Constrained / SNES_Stokes_SaddlePt) by addressing gauge freedoms (pressure and rigid-rotation), fixing false convergence in the grouped Schur fieldsplit configuration, and adding targeted regression tests to lock in partition-independent behavior.
Changes:
- Add guarded automatic pressure gauge pinning for enclosed constrained problems (
auto_pressure_gauge) to make raw mean pressure partition-reproducible. - Switch default outer Krylov/convergence settings for constrained solves to a flexible method with true-residual norm monitoring (FGMRES + unpreconditioned norm) and tighten EW behavior.
- Project rigid-body rotation modes out of the converged velocity to remove a partition-dependent rotation gauge; add regression coverage (serial + MPI).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/underworld3/systems/solvers.py |
Adds constrained-solver defaults (fgmres/unpreconditioned norm/EW tightening) and guarded automatic pressure gauge installation. |
src/underworld3/cython/petsc_generic_snes_solvers.pyx |
Adds post-solve rigid-rotation gauge removal plus extensive notes on internal-surface natural BC partition dependence. |
tests/test_1065_rotation_gauge_freeslip.py |
New serial regression test asserting the free-slip solution has ~0 projection onto the rigid-rotation mode. |
tests/parallel/test_1063_constrained_freeslip_parallel.py |
Adds an MPI regression for raw pressure gauge reproducibility and updates the CLI recomputation helper. |
tests/test_1064_constrained_spherical_shell_response.py |
Adds a new test ensuring the default constrained path matches Zhong benchmark velocity response. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| self._tolerance = value | ||
| self.petsc_options["snes_rtol"] = value | ||
| self.petsc_options["ksp_rtol"] = value * 1.0e-1 | ||
| self.petsc_options["ksp_atol"] = value * 1.0e-6 | ||
| self.petsc_options["snes_ksp_ew_rtol0"] = value * 1.0e-1 | ||
| self.petsc_options["snes_ksp_ew_rtolmax"] = value * 1.0e-1 |
| if getattr(self, "_velocity_rotation_nullspace", None) is not None: | ||
| return self._velocity_rotation_nullspace | ||
| if not self._petsc_velocity_nullspace_basis: | ||
| return None |
| rot_ns = self._build_velocity_rotation_nullspace() | ||
| if rot_ns is not None: | ||
| rot_ns.remove(gvec) |
…e finding Two clean-ups to the free-slip rotation-gauge handling: 1. Fix a latent staleness bug: the cached velocity rotation nullspace (_velocity_rotation_nullspace) is built from node coordinates but was never invalidated, so it would go stale after a solver rebuild / mesh deformation. It is now initialised in __init__ and reset in _reset_stokes_nullspace and the _build teardown path, alongside the monolithic Stokes nullspace. 2. Record, in _remove_velocity_rotation_gauge, the empirical result of trying the in-solve alternative: a rotations-only DMSetNullSpaceConstructor on the velocity field DOES attach the rotation nullspace to the fieldsplit velocity sub-block (verified), but the converged GLOBAL velocity still retained the rotation gauge (coefficient ~0.18, unchanged) — because removing a nullspace gauge is a projection on the converged global solution, and attaching it to a sub-block operator only constrains the inner Krylov solves (fgmres + a variable fieldsplit/Schur preconditioner reintroduces the rotation). So the explicit post-solve projection is the correct and necessary mechanism, not a stopgap. No behaviour change. Underworld development team with AI support from Claude Code
|
Reviewed (high-recall code review of the full diff): Approve. The four parts are mathematically — not just empirically — justified:
The internal-natural-BC Caveat carried forward to #254: the headline parallel numbers (1.2% → 2.5e-14 tangential at np8) still want the remote np=8 confirmation, but serial correctness and the mathematics stand independently. Merging via admin override (team-authored; branch protection needs a non-author review). Underworld development team with AI support from Claude Code |
…og, value-first call-site sweep (WE-01..03,05,06,08,09,10) (#338) * docs(WE-01): adopt the one-governing-doc-per-topic authority map Repoint CLAUDE.md's Data Access 'Authoritative Reference' from the stale UW3_Style_and_Patterns_Guide.md to subsystems/data-access.md (the guide it crowned teaches patterns the code deprecates at runtime — DOC-04), and record the Style Charter §10 authority table in docs/developer/index.md as the master authority index. The Charter is added to the Getting Started toctree (removes a baseline 'not included in any toctree' warning). Finding: DOC-04 (docs/reviews/2026-07/DOCS-STANDARDS-COHERENCE.md). Underworld development team with AI support from Claude Code * docs(WE-02): de-drift the Style Guide's four stale normative sections Rewrites the sections DOC-01 verified as contradicting the settled standards: - Docstring format: the 'Markdown Docstrings for pdoc/pdoc3' section is replaced by the NumPy/Sphinx RST standard (worked example with :math: and Parameters/Returns/Examples/Notes; conversion tracked in docs/plans/docstring-conversion-plan.md), per Style Charter section 6. - Doc file format: Quarto .qmd prescription (zero .qmd files exist in the repo) replaced by MyST .md/Sphinx guidance matching CLAUDE.md; migration table row updated. - Data access examples: 'Preferred' coordinate examples now use the real, runnable API — mesh.X.coords (read), mesh.deform() (coordinate changes), and the swarm.coords getter/setter for particle positions. The previous 'Preferred' example swarm.data += displacement raises AttributeError (getter-only property — SWARM-13 evidence); mesh.data warns at runtime. The private-attribute migration advice (swarm._particle_coordinates, mesh._deform_mesh presented as the NEW pattern) is deleted. - Front matter: the 21-line Quarto YAML header is replaced by a minimal MyST title block, and the guide now states that the UW3 Style Charter is the normative contract and wins on conflict. All replacement examples verified against current source: Swarm.coords setter (swarm.py), Mesh.deform (discretisation_mesh.py:3133), uw.synchronised_array_update / NDArray_With_Callback.delay_callbacks_global. Findings: DOC-01, SWARM-13 (style-guide part). Underworld development team with AI support from Claude Code * docs(WE-03): regenerate the docstring review queue; add the sweep to the release checklist The queue (last generated 2026-01-13, cdf5bb2) misrepresented the codebase both ways: it flagged now-complete items (solve, SNES_Scalar) as missing and contained zero entries for the June 2026 API (DOC-02). Regenerated over src/underworld3/**/*.py + **/*.pyx at the current tip. Two bugs in scripts/docstring_sweep.py's regex-based Cython parser made the regenerated queue lie about .pyx docstrings and are fixed as part of making the regeneration meaningful: - the indent group '(\s*)' with re.MULTILINE consumed preceding blank lines, shifting the computed definition line so the docstring search started ON the def/class line and always missed; - the docstring search started at the definition line rather than after the (possibly multi-line) signature, so long signatures hid their docstrings; - raw-string docstrings (r""", the norm in the solver .pyx) were not recognised. DOC-02 cross-validation on the regenerated queue now passes: solve / SNES_Scalar in the solver pyx are no longer flagged 'none'; the queue contains the June API (add_nitsche_bc, add_rotated_freeslip_bc, boundary_flux, set_custom_fmg, consistent_jacobian: 13 mentions) and flags the DOC-05 targets (Swarm.advection x2, read_timestep, write_proxy) as undocumented. Also adds the sweep to the quarterly release checklist (guides/release-process.md) so the queue cannot go stale unnoticed again. Findings: DOC-02 (docs/reviews/2026-07/DOCS-STANDARDS-COHERENCE.md). Underworld development team with AI support from Claude Code * docs(WE-05): backfill the changelog for May - early July 2026; add the changelog sweep to the release checklist The changelog (the quarterly CIG/stakeholder record) ended in April 2026 while ~117 first-parent commits landed May through early July (DOC-03). Backfilled at the existing conceptual granularity — 14 grouped entries, grouped by subsystem rather than by PR, matching the established format (### Title (Month Year), bold lead sentence, hyphen bullets, inline PR references): - New '2026 Q3 (July - September)' section: the July 2026 quality campaign (#309-#313, #317, #322-#326, #329, #334 as grouped entries), rotated strong free-slip / boundary traction / dynamic topography (#293, #294, #298, #306), generalized geometric multigrid via custom prolongation (#290, #297), consistent Jacobian tangent (#258), swarm correctness (#216, #313, #323, #329), numpy 2 support (#301, #305). - Extended '2026 Q2' section with the May-June entries: mesh adaptation movers (#190, #209, #213, #228, #259, #264, #266), moving-mesh field transfer / deform() (#246, #249, #251), semi-Lagrangian accuracy controls (#164, #183, #185-#189, #208, #220), snapshot/checkpoint toolkit (#146, #195, #196, #198), Stokes_Constrained (#224, #229, #240, #265), local-h Nitsche + boundary-slip surfaces (#225, #241, #275), units interoperability (#277, #278, #283, #284), memory/evaluation/solver infrastructure (#161, #177-#179, #181, #182, #222, #237, #250, ...). Every entry is backed by a merged commit on development (verified against git log --first-parent aed517f..3184a40). Also adds a quarterly-changelog sweep step beside the docstring sweep in the release checklist (guides/release-process.md) per DOC-03's proposed fix. Findings: DOC-03 (docs/reviews/2026-07/DOCS-STANDARDS-COHERENCE.md). Underworld development team with AI support from Claude Code * docs(WE-06): status headers on the unmarked design docs (per-doc git verification) Adds one-to-three-line Status markers to the 13 design docs that lacked one, following the directory's existing conventions (**Status**: line under the title; status: key inside existing YAML frontmatter for the three frontmatter-only docs), and corrects the stale 'Design Phase' marker on MATHEMATICAL_MIXIN_DESIGN.md (the mixin ships in utilities/mathematical_mixin.py). Every stamp was verified against git history (git log --follow dates) and the current source tree before writing: - Implemented: jacobian-consistent-tangent (PR #258, c63cd70), fmg-checkpoint-hierarchy (3cd73cd), petsc-dmplex-checkpoint-reload-plan (PR #146, write_timestep(petsc_reload=True) in tree), fault-refinement-simplification (smooth_mesh_interior / metric_density_from_gradient / fault_comb_metric all in tree), MATHEMATICAL_MIXIN_DESIGN. - Current reference/contract: mesh-adaptation-formulation, ND_UNITS_BOUNDARY_CONTRACT (PR #278, e0ece9a). - Investigation records (preserved via PR #245, 34a9dd4; production geometric-MG is custom prolongation, PR #290): snesfas-feasibility, snesfas-vanka-feasibility-study. - Design notes / prototypes with honest gaps: in_memory_checkpoint_design (not implemented, per its own trailing Status section), submesh-solver-architecture (extract_region/extract_surface exist; coarsened_companion does not). - Historical: ARCHITECTURE_ANALYSIS (persistence.py layout superseded), COORDINATE_MIGRATION_GUIDE (transition shipped), WHY_UNITS_NOT_DIMENSIONALITY (decision record). The audit's ~16 estimate over-counted: re-derived at this tip, 13 docs were unmarked plus one marked-but-stale (DOC-07). Findings: DOC-07 (docs/reviews/2026-07/DOCS-STANDARDS-COHERENCE.md). Underworld development team with AI support from Claude Code * docs(WE-08): convert units.py public docstrings Google -> NumPy style Docstring-only conversion of the 18 public module-level functions that carried Google-style Args:/Returns:/Raises:/Examples: labels (check_units_consistency, get_dimensionality, get_units, non_dimensionalise, show_nondimensional_form, simplify_units, create_quantity, convert_units, to_base_units, to_reduced_units, to_compact, get_scaling_coefficients, set_scaling_coefficients, validate_expression_units, assert_dimensionality, validate_coordinates_dimensionality, enforce_units_consistency, require_units_if_active, convert_angle_to_degrees) to the NumPy/Sphinx standard (Style Charter section 6). dimensionalise was already NumPy style; one-line docstrings and private helpers are untouched. No code, signature, or behaviour changes (verified: every diff hunk is inside a docstring; ast.parse clean). Finding: API-12 (docs/reviews/2026-07/API-CONSISTENCY-REVIEW.md). Underworld development team with AI support from Claude Code * docs(WE-09): sweep call sites of the newer BC methods to value-first (conds, boundary, ...) order Wave C (#334) made the ORIGINAL value-first order canonical for add_nitsche_bc / add_rotated_freeslip_bc / add_constraint_bc (maintainer decisions D2/D3; Style Charter section 6) with deprecation shims for the legacy boundary-first and g= spellings. This sweep updates every call site of those THREE methods to the canonical order so nothing in the repository exercises the shims — 74 sites total: - tests/: 63 call sites across 12 files (test_1017, test_1018, test_1060, test_1061, test_1062, test_1064, test_1065 x2 serial; parallel test_1017, test_1062, test_1063, test_1064). tests/test_0641_wave_c_api_shims.py is deliberately untouched — its legacy-order calls ARE the deprecation contract. - docs/: 7 sites (curved-boundary-conditions.md x4, CONSTRAINED_FREESLIP_MULTIPLIER.md call + signature line, examples/submesh_investigation/test_region_ds_nitsche.py). - .claude/skills/: 3 sites (adapt-on-top-faults x2, free-surface-convection x1). - CLAUDE.md: 1 signature reference (free-slip BC preference section). The ~1,370 legacy-trio (add_dirichlet_bc/add_natural_bc/add_essential_bc) sites already conform and are untouched per the D2 decision. The audit review documents under docs/reviews/2026-07/ record the pre-decision state as evidence and are not swept. Discovered while verifying the swept tests run warning-free: the Wave C zero-datum guard in add_rotated_freeslip_bc rejects FLOAT zero (sympy.sympify(0.0) != 0 is structurally True), so the canonical add_rotated_freeslip_bc(0.0, boundary) raises NotImplementedError while conds=0 works. Filed as issue #336 with a TODO(BUG) marker at the guard (comment-only src touch); the swept call sites use the working integer form add_rotated_freeslip_bc(0, boundary). No fix applied here (Charter section 9 scope discipline). Findings: API-01/API-02 sweep (WE-09, REMEDIATION-WORKLIST.md). Underworld development team with AI support from Claude Code
Parallel-correctness work for
SNES_Stokes_Constrained(the in-saddle Lagrange-multiplier free-slip solver), bringing the constrained free-slip solve to round-off partition-independence.What's here
1. Auto pressure gauge. Guarded
auto_pressure_gauge(default on) pins the surface-mean pressure on the first constraint boundary when a pressure null space is active and the user set no gauge. Physics-neutral; raw mean pressure 10.5%→0.4% at np=8, velocity bit-identical. Topography is read gauge-invariantly viatopography(reference="mean")— the raw multiplier keeps its own independent gauge freedom the pressure pin does not touch.2. Solver convergence. The grouped
u | [p,h]Schur preconditioner uses inner iterative sub-solves = a variable preconditioner; plaingmresis invalid for that and false-converged (preconditioned norm reported CONVERGED while the true residual blew up), landing on a wrong, partition-dependent answer. Fixed withksp_type=fgmres,ksp_norm_type=unpreconditioned, a tightened Eisenstat–Walker default, and a defensiveksp_max_it. Now genuinely converges (true resid ~1e-13) and reproduces the Zhong benchmark.3. Knockout audit + LU guard. Confirmed the interior-multiplier knockout is lossless (the earlier "2–5% off" was iterative conditioning, not lost physics). Added a warning that a monolithic direct solve (
pc_type lu/cholesky) of the constrained saddle point is a serial diagnostic only.4. Rigid-rotation gauge fix (the main new result). On a fully free-slip velocity the rigid rotations are a true nullspace of the velocity block (
A_uu·rotation = 0). The monolithic nullspace attached for the solve is not removed inside the fieldsplit/Schur iteration — the inner velocity KSP has no rotation nullspace, so it leaves an unconstrained, partition-dependent rigid rotation in the velocity. The operator is blind to it (the true residual still converges to machine precision), so the tangential velocity differed serial-vs-parallel by ~0.1–1% while the physical (radial) flow was already clean.SNES_Stokes_SaddlePt.solvenow projects the velocity rotation modes out of the converged solution viaMatNullSpaceRemove(_remove_velocity_rotation_gauge) — the velocity analogue of the constant-pressure gauge. Because the rotation is a genuine nullspace, removing it does not change the residual and yields the same rotation-free solution on any decomposition.Validated on the constrained free-slip problem: tangential surface velocity serial-vs-np8 relative difference 1.2% → 2.5e-14, iteration count unchanged. New serial guard
tests/test_1065_rotation_gauge_freeslip.py(the existingtest_1063cases all pin the rotation nullspace with a Dirichlet BC, so they did not cover this path).Known PETSc limitation (documented, not fixed here)
add_natural_bcon an internal surface assembles a slightly partition-dependent load:DMPlexComputeBdResidual_Single_Internalattributes each interior facet's integral to a partition-dependentsupport[0]cell closure, so a seam facet's non-owned closure DOFs are dropped at global assembly (F0 differs ~3e-4). The refinedTODO(BUG)records this and the workarounds that were tested and ruled out (owner-only label strip, 1-cell overlap, codim-1 submesh). Once the rotation gauge is removed, this leaves only ~3e-6 in the velocity (the elliptic solve smooths the localized load error); applying the internal traction as a volume body force makes it round-off.Tests
test_1064(constrained spherical shell) +test_1010(Cartesian Stokes): 9 passed, 2 pre-existing xfails.test_1063(parallel constrained free-slip, np=2): passed.test_1065(new rotation-gauge guard): passed.Underworld development team with AI support from Claude Code