Job hangs at 1000+ CPUs in uw.timing.print_table()

[gthyagi]

Jobs using Underworld3 hang when calling uw.timing.print_table() at large MPI scale (≈1000+ CPUs). The computation (including boundary integrals and metric evaluation) completes successfully, but the job does not exit and appears to stall during PETSc timing/log output.

this is what i am doing

metrics = {
    "m": params.uw_m,
    "cellsize": params.uw_cellsize,
    "v_l2_norm": v_err_l2,
    "p_l2_norm": p_err_l2,
    "p_l2_norm_abs": p_err_l2_abs,
    "v_l2_norm_lower": v_err_l2_lower,
    "v_l2_norm_upper": v_err_l2_upper,
    "p_l2_norm_lower": p_err_l2_lower,
    "p_l2_norm_upper": p_err_l2_upper,
    "p_l2_norm_lower_abs": p_err_l2_lower_abs,
    "p_l2_norm_upper_abs": p_err_l2_upper_abs,
    "sigma_rr_l2_norm_lower": sigma_rr_err_l2_lower,
    "sigma_rr_l2_norm_upper": sigma_rr_err_l2_upper,
    "u_dot_n_l2_norm_lower_abs": u_dot_n_l2_lower_abs,
    "u_dot_n_l2_norm_upper_abs": u_dot_n_l2_upper_abs,
}

if uw.mpi.rank == 0:
    print("=== L2 Error Metrics ===")
    for key, value in metrics.items():
        print(f"{key}: {value}")
integrals_stage_event.end()

if not params.uw_metrics_from_checkpoint_only:
    uw.timing.print_table(filename=os.path.join(output_dir, "integrals_timing.txt"))

# %% [markdown]
# ### Save Metrics Output

# %%
uw.pprint("Stage start: saving metric output")

if uw.mpi.rank == 0:
    metrics_h5 = os.path.join(output_dir, metrics_filename)
    if os.path.isfile(metrics_h5):
        os.remove(metrics_h5)
    with h5py.File(metrics_h5, "w") as f_h5:
        for key, value in metrics.items():
            f_h5.create_dataset(key, data=value)
        f_h5.create_dataset("git_sha", data=np.bytes_(git_sha))
        f_h5.create_dataset("command", data=np.bytes_(cli_args))
        for key, value in run_metadata.items():
            f_h5.create_dataset(key, data=value)

uw.pprint("Stage complete: saving metric output")

if i do not call this line uw.timing.print_table(filename=os.path.join(output_dir, "integrals_timing.txt")) then job completes successfully.

[gthyagi]

Source-level note from PETSc logging, tested against PETSc 3.25.0 (petscversion.h: PETSC_VERSION_MAJOR=3, MINOR=25, SUBMINOR=0; local vendored PETSc checkout: d3616053e302ec9a00f9e1eb098278d7544f73aa).

The hang appears specific to the default ASCII/text log viewer, not metric computation. In PETSc logdefault.c, the viewer dispatch sends default/text output to PetscLogHandlerView_Default_Info() but CSV output to PetscLogHandlerView_Default_CSV():

https://gitlab.com/petsc/petsc/-/blob/v3.25.0/src/sys/logging/handler/impls/default/logdefault.c#L1854-1866

The CSV path writes synchronized event rows and returns:

https://gitlab.com/petsc/petsc/-/blob/v3.25.0/src/sys/logging/handler/impls/default/logdefault.c#L1100-1166

The text path does much more. After the performance summary it runs extra MPI timing checks, including repeated barriers and a zero-size send/recv ring test:

https://gitlab.com/petsc/petsc/-/blob/v3.25.0/src/sys/logging/handler/impls/default/logdefault.c#L1770-1814

In the failing large-rank runs, the .txt file reaches Average time for MPI_Barrier() but does not reach Average time for zero size MPI_Send(). That points to the text-only communicator duplicate / zero-size ring-send section after the barrier timing, not to UW metric evaluation.

Practical fix/workaround for large CPU counts: write timing output as CSV instead of default text, e.g.

uw.timing.print_table(
    filename=os.path.join(output_dir, "integrals_timing.csv"),
    format="csv",
)

This avoids the problematic text-only MPI microbenchmark/final-report path and has completed successfully at large MPI scale. A longer-term UW3-side fix could make CSV the default/recommended timing format for large-rank runs, or avoid PETSc's default text log viewer for benchmark postprocessing stages.

[lmoresi]

Is this genuinely a UW-bug issue that we need to leave open, or should it be a documentation recommendation ?

[gthyagi]

@lmoresi, this appears to be a PETSc-related issue rather than Underworld. It would be good to have clearer documentation on this. In particular, at very high CPU counts, writing BD integrals to a .txt file can cause the job to hang, whereas using .csv output avoids the problem.

[lmoresi]

@gthyagi — agreed, this is a PETSc-side scaling characteristic of `PetscLogView` rather than a UW3 bug. Worth capturing as a documentation recommendation rather than leaving as an open bug, given the fix isn't in our codebase.

Suggested doc note (happy to incorporate phrasing tweaks):

When running on very high CPU counts (≳1000 ranks), call `uw.timing.print_table()` with `format="csv"` rather than the default ASCII format. The PETSc `PetscLogView` ASCII output path can hang at extreme rank counts, while the CSV path uses a different (less collective-heavy) write strategy and avoids the issue.
If you've already started the run with the default and need the timing data anyway, write to a `.csv` filename — the format is auto-detected from the extension.

I'll fold this into the relevant docstring (`uw.timing.print_table`) and add a short note under the parallel-computing docs (`docs/advanced/parallel-computing.md` or similar). Once that's in I'll close this issue. Sound good?

Underworld development team with AI support from Claude Code

[lmoresi]

Documented in #165 (merged to development). The PETSc PetscLogView ASCII path can hang at very high CPU counts; uw.timing.print_table("results.csv") uses PETSc's CSV write path and avoids the issue at any scale. Captured in both the uw.timing.print_table docstring and docs/advanced/parallel-computing.md (new "Timing Output at Extreme Scale" section). The behaviour is in PETSc, not Underworld; documenting CSV-at-scale as the recommended workaround. Closing manually since the PR merged to development not main.