compare(validation): make an absent observable fail the trajectory comparator

[akutuva21]

The defect

tests/validation/compare.py::compare_trajectories computed its shared set as an intersection:

common = [c for c in ref_cols if c in test_cols and c.lower() != "time"]

So a leg that had silently dropped an observable, or invented one, was compared only on the columns both legs still had — and scored a perfect match. That is the defect class this comparator is most likely to be asked about: a population ceasing to be reported is precisely how a seed-handling or observable-emission regression presents, and max_rel_err = 0.0 reads as "identical trajectory".

Failing-first, verbatim

Fault injection on synthetic legs (ref columns ['time','Otot','RD_Ba']), rtol=0.0, atol=0.0:

removed observable (RD_Ba missing):     ok=True  max_rel_err=0.0  max_rel_col='' note=''
phantom extra observable (RD_Ba added): ok=True  max_rel_err=0.0  max_rel_col='' note=''

Same injection repeated against real native-NFsim .gdat output for each tier-NF model, read through the shared parse_gdat, both policies side by side (seed 7, t_end 50, n_steps 50, construction_path=direct):

[localfunc]     native_cols=['time','Atot','Btot','Ctot','AB0','AB1','AB2','AB3','AB_motif']
  dropped  intersect ok=True  max_rel_err=0.0  note=''
  dropped  exact     ok=False max_rel_err=inf note="observable column sets differ: only in ref ['AB_motif'], only in test []"
  phantom  intersect ok=True  max_rel_err=0.0  note=''
  phantom  exact     ok=False max_rel_err=inf note="observable column sets differ: only in ref [], only in test ['Ophantom']"
[motor]         ... 'MotCCW' dropped
[simple_system] ... 'Xtotal' dropped
[tlbr]          ... 'Lfree'  dropped

Passing-after

Same instrument, same real legs, intact — all four tier-NF models at rtol=atol=0:

[localfunc]      intact+exact  ok=True  max_rel_err=0.0
[motor]          intact+exact  ok=True  max_rel_err=0.0
[simple_system]  intact+exact  ok=True  max_rel_err=0.0
[tlbr]           intact+exact  ok=True  max_rel_err=0.0

Pinned permanently in tests/validation/test_compare_trajectories.py (7 tests), which states both policies — including the intersect blind spot, so the default cannot drift back silently.

What changed

An explicit columns parameter. The default is unchanged.

mode	behaviour	caller
COLUMNS_INTERSECT (default)	shared observables only; absent/phantom column invisible by construction	test_parity_ode.py::test_ode_parity
COLUMNS_EXACT	same set of observable names required first; the differing names are named. Order deliberately not part of it — values are located by name	every trajectory-identity claim (below)

An unknown mode raises ValueError rather than defaulting.

Every caller, and the mode it now uses

exact — trajectory-identity claims:

• test_parity_nfsim.py::_SWEEP_CHILD direct-vs-XML sweep record (a lost observable previously recorded as compared at 0.0 in the tier-P report)
• test_parity_nfsim.py::test_nf_fixed_seed_direct_matches_native_at_final_endpoint
• test_parity_nfsim.py::test_nf_ast_direct_matches_xml
• test_parity_nfsim.py::test_nf_seed_site_state_is_resolved_by_name
• test_parity_nfsim_seed.py::test_fixed_seed_reproduces_the_same_trajectory_on_both_legs
• test_parity_nfsim_seed.py::test_seed_state_order_matches_native_nfsim
• test_parity_nfsim_seed.py::test_an_family_seed_state_order_matches_native_nfsim
• test_parity_nfsim_seed.py::_identity_report (delegates; keeps only the ordered-column check)
• test_stochastic_comparator.py, 2 sites (fixed known column list)

intersect — one deliberate caller:

• test_parity_ode.py::test_ode_parity compares a BNG2-produced .gdat against a BNG3-produced one. Two independent writers: the column set is an input, not a claim, and the committed reference's observable block need not enumerate everything the engine emits. It is a numeric comparator over a column set allowed to vary and makes no claim about which observables exist. Unchanged.

The full table also lives in the compare_trajectories docstring, so it travels with the code.

_identity_report folded, not duplicated

test_parity_nfsim_seed._identity_report no longer carries its own value/shape/time comparison. It delegates those to compare_trajectories(..., columns=COLUMNS_EXACT, rtol=0.0, atol=0.0) and keeps exactly the one check the shared comparator deliberately does not make: ordered column identity. Two comparators remain, not three.

Which half of the module is consumer-backed, and which is not

Adjacent to the numbers, because they are not the same kind of claim.

Consumer-backed. parse_gdat is the shared reader for every oracle .gdat in this tree. The trajectory values compared above are read by the same code that feeds the native-NFsim and BNG2 oracles, and the native leg is an independently built binary (RuleWorld/nfsim @ c51c7a3, sha256 093707031f70e0c376179e1d8bf89ab1373b3132b6211d7d9759f1d646c4ce9e) — never BNG3's embedded NFsim. That half has a consumer outside the file.

Not consumer-backed. The column-set policy. Before this PR nothing asserted it, which is exactly why the blind spot survived. Its evidence is this PR's own fault injection plus a test written by the same change whose behaviour it pins. That is a real check and it fails on the defect — but it does not independently corroborate the oracle the way parse_gdat does, and it should not be read as if it did.

The adjacent gap: simple_system vs the documented four-model set

docs/CI_PARITY.md:38 and docs/BNG3_INTEGRATION_PLAN.md:39 have described a four-model fixed-seed direct/native endpoint set since the 2026-09-15 checkpoint. parity.yml's node — test_nf_fixed_seed_direct_matches_native_at_final_endpoint, the only tests/validation/ module any CI job runs — hard-coded model_name = "simple_system". The doc described a gate that measured one model.

The code was changed, not the doc. nfsimParity measured all four against native NFsim at rtol=atol=0 before the parametrization went in (their measurement, reproduced here on the worktree engine in the tables above), so the documented claim was the truthful side and the gate now measures what it says. Softening a correct claim to match an under-tested gate would have been the wrong direction.

The fixed-seed gate is now parametrized over corpus.tier_nf() (simple_system, tlbr, motor, localfunc) and parity.yml's -k selector is unchanged.

Verification

• tests/validation/test_compare_trajectories.py + test_stochastic_comparator.py + test_compare_net.py: 30 passed.
• tests/validation/test_parity_nfsim.py + test_parity_nfsim_seed.py -m "nf and not slow" (the selection parity.yml runs, extended to this branch): 21 passed, 0 skipped, 0 deselected-by-oracle.
• Node collection after the change: test_nf_fixed_seed_direct_matches_native_at_final_endpoint[{localfunc,motor,simple_system,tlbr}].

Artifact provenance. The engine is not the shared build/cpp: sha256 887c81252c567073db2c1029d5f37805dd82c2a475ffb8db4583797eec5de8a6, built in this worktree at 9f840a5. The shared build at BNG3/build/cpp is 15 cpp/ commits behind main; the numbers above do not describe it. The oracle is sha256 093707031f70e0c376179e1d8bf89ab1373b3132b6211d7d9759f1d646c4ce9e.

not_claimed

• No BNG2 oracle run. BNG2.pl is not present on this host, so test_parity_ode.py skips. The one caller left on intersect is verified by its stated contract and by the unit test that pins the policy — not by an end-to-end comparison. Its behaviour is byte-for-byte unchanged either way; I did not switch it and then discover it needed switching.
• No claim about the tier-P sweep's aggregate outcome. Changing _SWEEP_CHILD to exact means a model whose XML and direct legs report different observable names now records max_rel_err=inf with the note observable column sets differ, where it previously recorded 0.0 and compared. That is the intended change and it is the correct reading — but the sweep is slow, runs in no CI job, and I did not run it. Whether any committed tier-P model trips it is unmeasured by me. @TierParityExpand is re-measuring the sweep at the full horizon and has been told this change is mine.
• Column order is still not checked by exact. Ordered identity lives only in _identity_report, with three callers inside one file. Promoting it into the shared mode would have made exact reject a leg that reports the same observables in a different order — which this comparator, which aligns by name, handles correctly. That is a deliberate boundary, not an oversight.

Files

• tests/validation/compare.py — COLUMNS_INTERSECT / COLUMNS_EXACT, the exact-mode gate, the caller/mode table, ValueError on an unknown mode
• tests/validation/test_compare_trajectories.py — NEW, 7 tests pinning both policies
• tests/validation/test_parity_nfsim.py — 4 call sites to exact; fixed-seed gate parametrized over corpus.tier_nf()
• tests/validation/test_parity_nfsim_seed.py — 3 call sites to exact; _identity_report folded onto the shared comparator
• tests/validation/test_stochastic_comparator.py — 2 call sites to exact
• docs/CI_PARITY.md — one paragraph naming the four-model set and the exact mode
• tests/validation/test_parity_ode.py — deliberately unchanged

[akutuva21]

Superseded after six-file residual audit and merge of PR #91 at exact head 9aff259, remote merge b08e1be. The replacement carries exact/intersection column policies, duplicate-column rejection, strict oracle callers, the two previously uncovered default-intersection/time-only contracts, and corrected four-model seed documentation. Redundant endpoint expansion, explicit exact mode on identical stochastic fixtures, and a behavior-neutral helper refactor were not duplicated; the working child native-extension path was preserved. Final-head hosted checks were 41 successful and 5 skipped; fresh own-source comparator tests passed 12/12 and four seed cases collected. Original author branch preserved. The separate michment fixed-cohort strict failure remains unresolved.