Repository navigation
Conversation
…sion Embedding libstdc++ inside _bionetgen_cpp gives the host Python process a second C++ runtime whenever another loaded library also provides libstdc++.so.6 -- SciPy's C++ extensions do. The runtimes keep independent locale facet tables, so ostringstream numeric formatting dispatches through the wrong facet and the process segfaults. Linux/GCC evidence, all from the Batch SSA CPU reference parity job (16 consecutive failures before this change): - faulthandler: crash at tests/test_batch_ssa_statistical_parity.py:164, generate_network, first model, ~4ms into the gate - gdb backtrace: Document -> CompiledModel -> CompiledRateLaw::compile -> Expression::toString -> ostream::_M_insert<double> -> codecvt<char16_t>:: do_length -- a call ostream cannot make without a facet mix-up - ASan: SEGV on a READ at 0x03e90000131f inside read_utf8_code_point<char>, with the locale facet table already corrupt - /proc/self/maps: libstdc++.so.6.0.30 mapped alongside the module, and ten SciPy extension modules linking libstdc++.so.6 After the change ldd reports libstdc++.so.6 as a dependency of the module and the same job reports ALL CPU REFERENCE PARITY TESTS PASSED SUCCESSFULLY. CLI executables keep their static runtimes via CMAKE_EXE_LINKER_FLAGS.
The extension crashed in Expression::toString() because _bionetgen_cpp embedded a second libstdc++ while the host interpreter already had libstdc++.so.6 mapped (SciPy's C++ modules do). Two runtimes in one process means two sets of locale facet tables, so std::ostream dispatches numeric formatting through the wrong facet. Assert the contract so the flag cannot come back, and so the executable-side static runtime cannot be "cleaned up" by someone reasoning from the extension side. Also pin the Batch SSA CPU reference parity job in ci.yml and the parity script's fail-closed exit wiring. That job is the one that loads the freshly built extension into the same interpreter that imported SciPy; it had never been green, so the defect shipped behind it. Verified by reverting each protection in a scratch copy: restoring target_link_options(_bionetgen_cpp ... -static-libstdc++/-static-libgcc), making the module SHARED, dropping CMAKE_EXE_LINKER_FLAGS, deleting the batch-ssa-cpu-reference job, appending "|| true" to its run line, ignoring the gate result in main(), and swallowing a gate exception each produce a failure. No C++ or workflow file is modified.
A scikit-build editable install registers a meta_path finder, and
meta_path finders run BEFORE sys.path. In a git worktree `import bionetgen`
therefore resolves to whichever tree that editable install points at --
typically the shared main worktree -- so tests/python/ passes while
exercising someone else's code. PYTHONPATH cannot fix this: it loses to
the finder, and once sys.modules['bionetgen'] is bound the wrong module
persists for the rest of the session.
This strips such finders and prepends this checkout's python/ directory.
PLACEMENT IS LOAD-BEARING, and this is demonstrated, not asserted.
tests/python/conftest.py imports bionetgen at module scope:
`requires_perl = pytest.mark.skipif(not _has_perl_bng(), ...)` at lines
22-25 calls _has_perl_bng() while the module is still being imported. So
the guard must be the first executable code in the file.
Measured on this host, one probe test printing bionetgen.__file__:
guard at top of file (correct)
-> /Users/akutuva/Documents/BioNetGen/BNG3-contract2/python/bionetgen/__init__.py
guard moved into a pytest_configure hook
-> /Users/akutuva/Documents/BioNetGen/BNG3/python/bionetgen/__init__.py
The hook variant does not error, does not warn, and the test still PASSES.
It silently stops protecting anything. If you move this block into a hook,
re-run that probe -- a green suite is not evidence the guard works.
The guard is also deliberately conditional on an editable finder being
present. CI installs a real wheel and has none; prepending the checkout's
python/ there would shadow the installed wheel and test something CI never
built. Verified: with the editable finders removed, this block leaves
sys.path untouched.
No production code is touched.
The guard added in 2521f83 removed every sys.path entry containing the substring "BioNetGen". Every checkout on this host matches that substring, so it also deleted this worktree's OWN build/cpp -- the only place a fresh worktree has the compiled extension. It then re-added only python/. Consequence, measured in one worktree with identical .so bytes: PYTHONPATH=python:<ext dir> -> 888 passed, 54 skipped, 0 failed PYTHONPATH=python:build/cpp -> 46 failed, 685 passed, 91 skipped The 46 failures carry a misleading `AttributeError: 'NoneType' object has no attribute 'parse_file'`, which reads like a broken binding rather than a missing sys.path entry, and importorskip-guarded modules SKIP silently. So a guard written to make shadowing loud was making a missing build quiet instead. Reported by @sciStochastic and confirmed by @memWatch; the file is mine and the defect is mine. Two fixes, both verified rather than assumed: 1. Scope the strip to this checkout. An entry survives if it is not a BioNetGen path OR it resolves under this tree's root, so another lane's checkout goes and this one's build output stays. 2. Add this checkout's own build/cpp when it exists. bionetgen's model modules do `import bionetgen._bionetgen_cpp`, so the module must be reachable as a submodule; a fresh worktree has it only there. Fixing only (1) still failed -- I measured 46 failures with the strip scoped but no build dir added, which is how the second half surfaced. Verified on a worktree carrying its own build/cpp: tests/python/ -> 909 passed, 78 skipped, 0 failed (was 46 failed) bionetgen.__file__ -> this worktree's python/ bionetgen._bionetgen_cpp -> this worktree's build/cpp .so and the guard's original purpose is intact: another checkout's tree no longer wins. Still a no-op when no editable finder is present, so CI keeps testing the installed wheel. Also records my own error: when I first saw these 46 failures during development I attributed them to the stale shared binary and said so in my PR body. The controlled A/B above shows the guard caused them.
|
Superseded by #53, which Main wrote after my reproduction. Closing rather than leaving two fixes to the same file in flight. My evidence stands and Main has folded the three-point A/B (46 failed / 858 passed / 909 passed) into #53. The one behavioural difference that decides it: my filter drops any 'BioNetGen' path not under this checkout, which kills a legitimate PYTHONPATH pointing at a shared artifact directory -- the workaround most agents on this host use, and the case my run E never exercised because that worktree had its own build. #53 identifies the editable install's source structurally from the finder's search_paths, keeps everything else, and raises loudly when the extension is unimportable instead of letting importorskip turn a broken environment into a green run. Carrying my one finding forward in a new contract: tests/test_ci_contract.py will assert that the guard preserves a shared artifact directory, which would have caught both my attempt and Main's first one in a single gate. |
Fixes a defect I introduced in #48
tests/python/conftest.pyis my file and this is my bug. Reported by@sciStochastic, confirmed by @memWatch; I reproduced it before changing
anything.
Mechanism. #48 removed every
sys.pathentry containing the substringBioNetGen:Every checkout on this host matches that substring, so it also deleted this
worktree's own
build/cpp— the only place a fresh worktree has the compiledextension — and then re-added only
python/.Consequence, measured in one worktree with identical
.sobytes:PYTHONPATHpython:<ext dir>python:build/cppThe 46 failures carry a misleading
AttributeError: 'NoneType' object has no attribute 'parse_file'— which reads like a broken binding, not a missing pathentry — and importorskip-guarded modules skip silently. So a guard written to
make shadowing loud was making a missing build quiet, which is the same
failure shape it was meant to prevent.
Two fixes, both verified
BioNetGenpath or it resolves under this tree's root — another lane'scheckout goes, this one's build output stays.
build/cppwhen present.bionetgen's modelmodules do
import bionetgen._bionetgen_cpp, so the module must be reachableas a submodule, and a fresh worktree has it only there.
Fixing only (1) still failed — I measured 46 failures with the strip scoped but
no build directory added, which is how the second half surfaced. I did not
assume it was a one-line fix.
Verification
On a worktree carrying its own
build/cpp:The guard's original purpose is intact: another checkout's tree no longer wins.
Still a no-op when no editable finder is present, so CI keeps testing the
installed wheel. black and ruff clean.
My own error, on the record
When I first saw these 46 failures during development I attributed them to the
stale shared binary and stated that in PR #48's body. That was wrong. The
controlled A/B above — same tree, guard the only variable — shows the guard
caused them, and I should not have dismissed a number that matched a known
hazard without running the comparison. The three-variant placement table in #48
was correct; this claim was not.