Leverage BFBFLAG to use reproducible sums in place of ESMF_VMAllreduce

@Katetc

As pointed out by @Katetc - we have a number of places in CMEPS where we use ESMF_VMAllreduce to do global sums. She found this in the course of testing answer changes with changing processor counts for T compsets (CISM forced by DLND), and for that use case, the issue is the global sums in the SMB renormalization (she says that turning off SMB renormalization allows answers to be bit-for-bit when changing processor counts - i.e., glc_renormalize_smb). But, from a quick look through the CMEPS code, it looks like this also comes up in post_rof_mod (dealing with negative runoff) and in prep_atm_mod (enthalpy correction).

I'm not sure we want to go so far as to require that global sums always be bit-for-bit with changing processor count, since that could impact performance. (My impression is that this isn't actually a CESM-wide requirement, but some components have it as a requirement. I'm not sure I'm right about that, though.) But it seems like we should at least have an option for reproducible global sums if a flag is set, and we can then use that flag in our testing. We have a currently-deprecated BFBFLAG, and the easiest course may be to bring that flag back into operation and use it for this purpose.

So I propose:

Use BFBFLAG to set a nuopc runtime variable that controls whether we do more expensive global reductions that are bit-for-bit across processor counts
If this flag is set, we can use the reproducible sums in https://github.com/ESCOMP/CESM_share/blob/main/src/shr_reprosum_mod.F90 in place of ESMF_VMAllreduce

@briandobbins @mvertens - also looping you in here; I'd welcome your thoughts.

@billsacks

@billsacks - this sounds like a very useful addition to have in CMEPS. I agree that we should resurrect the deprecated BFBFLAG for this.

We also find that global sums with ESMF_VMAllreduce are not deterministic (on repeated runs on the same processor count) , is that not true in all machines/compiler options/cases ?

@briandobbins

We also find that global sums with ESMF_VMAllreduce are not deterministic (on repeated runs on the same processor count) , is that not true in all machines/compiler options/cases ?

Very good question. This is at the edge of my knowledge. Could it have something to do with whether a machine is set up (at the system level or in the job submission) to bind processes to the same layout of physical processors each time???

@briandobbins @mvertens can either of you advise on this or suggest someone who would be good to pull into this?

I honestly don't see how this is the case for us - since we do so many reproducibility tests and we are using ESMF_VMAllreduce in quite a few places in the mediator. That being said - Bob Oehmke would be the best person from the ESMF team to get input from on this.

I'll ping ESMF support (including Bob & others) to provide input on this.

@billsacks

@billsacks ESMF team can confirm that ESMF_VMAllreduce is a wrapper around MPI and so any of the non-reproducible observations are coming from MPI directly. + @oehmke

If bit-for-bit is needed then you can try tuning the MPI all reduce collective operation. In order for bit-for-bit reproducibility the order of additions matter. This is because the significand possibly varies per term and therefore rounds during each addition. MPI all reduce algorithms are dynamically determined based on message size and ranks for performance reasons.

@danrosen25

Thanks, @danrosen25 .

@mvertens and I find it puzzling that we haven't seen any reproducibility issues in NorESM or CESM (at least with consistent processor counts), yet @anton-seaice is finding these. I'm not sure what can explain this....

To clarify - I added some code to balance global freshwater in CMEPS when using DATM + DROF, so it wasn't in existing code but it's code that appears similar to me to other subroutines in post_rof. We don't use the post_rof negative runoff phases, so I don't know conclusively if the issue occurs there but I would guess it does.

I found the global sums were deterministic to about 15 sig figs., so it may depend on the details of how the sum is used whether that impacts determinism with the whole model.

We use an openmpi (4.1.7) build provided by our HPC (Gadi) and don't set anything specific about this - so maybe it depends on the openmpi build / environment ?

From

$ ompi_info --param coll tuned --level 9

MCA coll tuned: parameter "coll_tuned_gather_algorithm" (current
                          value: "ignore", data source: default, level: 5
                          tuner/detail, type: int)
                          Which gather algorithm is used. Can be locked down
                          to choice of: 0 ignore, 1 basic linear, 2 binomial,
                          3 linear with synchronization. Only relevant if
                          coll_tuned_use_dynamic_rules is true.
                          Valid values: 0|ignore, 1|basic_linear, 2|binomial,
                          3|linear_sync

The docs for these algorithms appear incomplete: https://docs.open-mpi.org/en/v5.0.x/tuning-apps/coll-tuned.html#collectives-and-their-algorithms

@anton-seaice

Thanks for this extra information, @anton-seaice . I could imagine that this depends on the details of the system; I could also imagine that - like you say - it depends on exactly how the global sum is actually used. It does make me wonder if we should switch to using reproducible sums by default, though this would add a dependency (or else require duplicating this module, which isn't ideal), and I'd want to know the performance impact.

I'm curious: do you have a mechanism for reproducible sums in your model - the equivalent of https://github.com/ESCOMP/CESM_share/blob/main/src/shr_reprosum_mod.F90? Or would you be open to having a dependency on that repository for this and possibly other purposes in CMEPS?

Yeah we already use CESM_share (and compile with #CESMCOUPLED set turned on)

I think CICE and MOM6 use similar reproducible sums already for all prognostic calculations

@anton-seaice

The original intent of this issue is addressed in #704 . By default we'll still use ESMF_VMallreduce (which is sensitive to processor count because it simply uses the underlying MPI reduction), but it provides an option to use reproducible sums. @anton-seaice , this does not address the issue you raised with changing results even with the same processor count, which I still don't totally understand - though turning on bfbflag with the changes in #704 might resolve that problem for you.

This is implemented in three places: prep_glc (for SMB renormalization), post_rof (for removal of negative runoff) and prep_atm (for an enthalpy correction). There is one other possibly relevant global sum in CMEPS - for med_diag_mod - which I am not handling here; I think this is just for budget diagnostics, which I feel are okay to differ by roundoff with processor changes, and from a quick look, I think that fixing that would require more extensive changes.

Merge pull request ESCOMP#704 from billsacks/glc_renormalize_smb_repr…

Navigation Menu

Repository navigation

Leverage BFBFLAG to use reproducible sums in place of ESMF_VMAllreduce #650

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Leverage BFBFLAG to use reproducible sums in place of ESMF_VMAllreduce #650

Description

Activity

mvertens commented on May 8, 2026

anton-seaice commented on Jun 17, 2026

billsacks commented on Jun 18, 2026

mvertens commented on Jun 24, 2026

billsacks commented on Jun 24, 2026

anntsay commented on Jun 24, 2026

danrosen25 commented on Jun 30, 2026

billsacks commented on Jun 30, 2026

anton-seaice commented on Jul 1, 2026

anton-seaice commented on Jul 1, 2026

billsacks commented on Jul 1, 2026

anton-seaice commented on Jul 1, 2026

billsacks commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions