Skip to content

fix(python): stop the conftest guard from hiding a missing build - #56

Closed
akutuva21 wants to merge 4 commits into
mainfrom
agent/contract-tighten-guard-fix
Closed

akutuva21 wants to merge 4 commits into
mainfrom
agent/contract-tighten-guard-fix

Conversation

@akutuva21

Copy link
Copy Markdown
Member

Fixes a defect I introduced in #48

tests/python/conftest.py is my file and this is my bug. Reported by
@sciStochastic, confirmed by @memWatch; I reproduced it before changing
anything.

Mechanism. #48 removed every sys.path entry containing the substring
BioNetGen:

sys.path[:] = [p for p in sys.path if "BioNetGen" not in p]
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2] / "python"))

Every checkout on this host matches that substring, so it also deleted this
worktree's own build/cpp
— the only place a fresh worktree has the compiled
extension — and then re-added only python/.

Consequence, measured in one worktree with identical .so bytes:

PYTHONPATH Result
python:<ext dir> 888 passed, 54 skipped, 0 failed
python:build/cpp 46 failed, 685 passed, 91 skipped

The 46 failures carry a misleading AttributeError: 'NoneType' object has no attribute 'parse_file' — which reads like a broken binding, not a missing path
entry — and importorskip-guarded modules skip silently. So a guard written to
make shadowing loud was making a missing build quiet, which is the same
failure shape it was meant to prevent.

Two fixes, both verified

  1. Scope the strip to this checkout. An entry survives if it is not a
    BioNetGen path or it resolves under this tree's root — another lane's
    checkout goes, this one's build output stays.
  2. Add this checkout's own build/cpp when present. bionetgen's model
    modules do import bionetgen._bionetgen_cpp, so the module must be reachable
    as a submodule, and a fresh worktree has it only there.

Fixing only (1) still failed — I measured 46 failures with the strip scoped but
no build directory added, which is how the second half surfaced. I did not
assume it was a one-line fix.

Verification

On a worktree carrying its own build/cpp:

tests/python/            -> 909 passed, 78 skipped, 0 failed   (was 46 failed)
bionetgen.__file__       -> this worktree's python/
bionetgen._bionetgen_cpp -> this worktree's build/cpp/_bionetgen_cpp...so

The guard's original purpose is intact: another checkout's tree no longer wins.
Still a no-op when no editable finder is present, so CI keeps testing the
installed wheel. black and ruff clean.

My own error, on the record

When I first saw these 46 failures during development I attributed them to the
stale shared binary and stated that in PR #48's body. That was wrong. The
controlled A/B above — same tree, guard the only variable — shows the guard
caused them, and I should not have dismissed a number that matched a known
hazard without running the comparison. The three-variant placement table in #48
was correct; this claim was not.

…sion

Embedding libstdc++ inside _bionetgen_cpp gives the host Python process a
second C++ runtime whenever another loaded library also provides
libstdc++.so.6 -- SciPy's C++ extensions do. The runtimes keep independent
locale facet tables, so ostringstream numeric formatting dispatches through
the wrong facet and the process segfaults.

Linux/GCC evidence, all from the Batch SSA CPU reference parity job
(16 consecutive failures before this change):
- faulthandler: crash at tests/test_batch_ssa_statistical_parity.py:164,
  generate_network, first model, ~4ms into the gate
- gdb backtrace: Document -> CompiledModel -> CompiledRateLaw::compile ->
  Expression::toString -> ostream::_M_insert<double> -> codecvt<char16_t>::
  do_length -- a call ostream cannot make without a facet mix-up
- ASan: SEGV on a READ at 0x03e90000131f inside
  read_utf8_code_point<char>, with the locale facet table already corrupt
- /proc/self/maps: libstdc++.so.6.0.30 mapped alongside the module, and ten
  SciPy extension modules linking libstdc++.so.6

After the change ldd reports libstdc++.so.6 as a dependency of the module and
the same job reports ALL CPU REFERENCE PARITY TESTS PASSED SUCCESSFULLY.

CLI executables keep their static runtimes via CMAKE_EXE_LINKER_FLAGS.
The extension crashed in Expression::toString() because _bionetgen_cpp
embedded a second libstdc++ while the host interpreter already had
libstdc++.so.6 mapped (SciPy's C++ modules do). Two runtimes in one process
means two sets of locale facet tables, so std::ostream dispatches numeric
formatting through the wrong facet. Assert the contract so the flag cannot
come back, and so the executable-side static runtime cannot be "cleaned up"
by someone reasoning from the extension side.

Also pin the Batch SSA CPU reference parity job in ci.yml and the parity
script's fail-closed exit wiring. That job is the one that loads the
freshly built extension into the same interpreter that imported SciPy; it
had never been green, so the defect shipped behind it.

Verified by reverting each protection in a scratch copy: restoring
target_link_options(_bionetgen_cpp ... -static-libstdc++/-static-libgcc),
making the module SHARED, dropping CMAKE_EXE_LINKER_FLAGS, deleting the
batch-ssa-cpu-reference job, appending "|| true" to its run line, ignoring
the gate result in main(), and swallowing a gate exception each produce a
failure. No C++ or workflow file is modified.
A scikit-build editable install registers a meta_path finder, and
meta_path finders run BEFORE sys.path. In a git worktree `import bionetgen`
therefore resolves to whichever tree that editable install points at --
typically the shared main worktree -- so tests/python/ passes while
exercising someone else's code. PYTHONPATH cannot fix this: it loses to
the finder, and once sys.modules['bionetgen'] is bound the wrong module
persists for the rest of the session.

This strips such finders and prepends this checkout's python/ directory.

PLACEMENT IS LOAD-BEARING, and this is demonstrated, not asserted.
tests/python/conftest.py imports bionetgen at module scope:
`requires_perl = pytest.mark.skipif(not _has_perl_bng(), ...)` at lines
22-25 calls _has_perl_bng() while the module is still being imported. So
the guard must be the first executable code in the file.

Measured on this host, one probe test printing bionetgen.__file__:

  guard at top of file (correct)
    -> /Users/akutuva/Documents/BioNetGen/BNG3-contract2/python/bionetgen/__init__.py
  guard moved into a pytest_configure hook
    -> /Users/akutuva/Documents/BioNetGen/BNG3/python/bionetgen/__init__.py

The hook variant does not error, does not warn, and the test still PASSES.
It silently stops protecting anything. If you move this block into a hook,
re-run that probe -- a green suite is not evidence the guard works.

The guard is also deliberately conditional on an editable finder being
present. CI installs a real wheel and has none; prepending the checkout's
python/ there would shadow the installed wheel and test something CI never
built. Verified: with the editable finders removed, this block leaves
sys.path untouched.

No production code is touched.
The guard added in 2521f83 removed every sys.path entry containing the
substring "BioNetGen". Every checkout on this host matches that substring,
so it also deleted this worktree's OWN build/cpp -- the only place a fresh
worktree has the compiled extension. It then re-added only python/.

Consequence, measured in one worktree with identical .so bytes:

  PYTHONPATH=python:<ext dir>                 -> 888 passed, 54 skipped, 0 failed
  PYTHONPATH=python:build/cpp                 ->  46 failed, 685 passed, 91 skipped

The 46 failures carry a misleading `AttributeError: 'NoneType' object has
no attribute 'parse_file'`, which reads like a broken binding rather than a
missing sys.path entry, and importorskip-guarded modules SKIP silently.
So a guard written to make shadowing loud was making a missing build quiet
instead. Reported by @sciStochastic and confirmed by @memWatch; the file is
mine and the defect is mine.

Two fixes, both verified rather than assumed:

1. Scope the strip to this checkout. An entry survives if it is not a
   BioNetGen path OR it resolves under this tree's root, so another lane's
   checkout goes and this one's build output stays.

2. Add this checkout's own build/cpp when it exists. bionetgen's model
   modules do `import bionetgen._bionetgen_cpp`, so the module must be
   reachable as a submodule; a fresh worktree has it only there. Fixing
   only (1) still failed -- I measured 46 failures with the strip scoped
   but no build dir added, which is how the second half surfaced.

Verified on a worktree carrying its own build/cpp:
  tests/python/ -> 909 passed, 78 skipped, 0 failed  (was 46 failed)
  bionetgen.__file__              -> this worktree's python/
  bionetgen._bionetgen_cpp        -> this worktree's build/cpp .so
and the guard's original purpose is intact: another checkout's tree no
longer wins. Still a no-op when no editable finder is present, so CI keeps
testing the installed wheel.

Also records my own error: when I first saw these 46 failures during
development I attributed them to the stale shared binary and said so in my
PR body. The controlled A/B above shows the guard caused them.
@akutuva21

Copy link
Copy Markdown
Member Author

Superseded by #53, which Main wrote after my reproduction. Closing rather than leaving two fixes to the same file in flight.

My evidence stands and Main has folded the three-point A/B (46 failed / 858 passed / 909 passed) into #53. The one behavioural difference that decides it: my filter drops any 'BioNetGen' path not under this checkout, which kills a legitimate PYTHONPATH pointing at a shared artifact directory -- the workaround most agents on this host use, and the case my run E never exercised because that worktree had its own build. #53 identifies the editable install's source structurally from the finder's search_paths, keeps everything else, and raises loudly when the extension is unimportable instead of letting importorskip turn a broken environment into a green run.

Carrying my one finding forward in a new contract: tests/test_ci_contract.py will assert that the guard preserves a shared artifact directory, which would have caught both my attempt and Main's first one in a single gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant