Repository navigation
fix(validation): make a missing compiled extension fail under strict mode - #75
Merged
Merged
Conversation
…mode
The `api` fixture called pytest.skip directly, so a checkout whose compiled
extension was not importable reported every test taking that fixture as
SKIPPED with exit 0 — including under BNG3_CI_STRICT_ORACLES=1, the flag whose
entire purpose is to turn a missing engine into a failure.
This is the same defect as the one the bng_cpp fixture three lines above
already avoids: conftest.py imports require_oracle at :25 and routes bng_cpp
through it at :56, while api at :70 bypassed it. Routing the same branch
through require_oracle makes the two consistent, and makes the trade the file
already made explicit: a missing engine fails in strict mode, and a local run
without a build still skips.
Proved by calling the branch directly rather than inferring it from a suite
run, because the question is what this fixture does on one input:
BNG3_CI_STRICT_ORACLES unset, conftest.api(False) -> pytest.skip
BNG3_CI_STRICT_ORACLES=1, conftest.api(False) -> pytest.fail (was: pytest.skip)
Regression test asserts both directions, and asserts the strict case
explicitly rather than letting a skip escape: a skip raised out of a test body
makes pytest record the test as skipped with exit 0, which is the reporting
shape being fixed and would make this test itself fail silently.
Failing-first, both directions:
conftest reverted to the bare pytest.skip -> 1 failed
"the api fixture SKIPPED under BNG3_CI_STRICT_ORACLES=1 instead of
failing ... as it already is for the bng_cpp fixture"
this branch -> 1 passed
pytest tests/validation/test_harness_paths.py -> 12 passed, 1 skipped
pytest tests/validation/test_parity_workflow_wiring.py -> 6 passed
pytest tests/test_ci_contract.py -> 38 passed
No production file touched. Blast radius is the four tests/validation modules
that take the `api` fixture, and under the strict flag their behaviour changes
from a quiet skip to a loud failure — which is the point.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
apifixture turned a missing engine into a clean skiptests/validation/conftest.py:70calledpytest.skipdirectly. A checkout whose compiled extension was not importable therefore reported every test taking that fixture as SKIPPED with exit 0 — including underBNG3_CI_STRICT_ORACLES=1, the flag whose entire purpose is to turn a missing engine into a failure.This is the same defect the fixture three lines above already avoids:
One file, two treatments of the same question. Routing the same branch through
require_oraclemakes them consistent, and makes the trade the file already made explicit: a missing engine fails in strict mode, and a local run without a build still skips.Proved by calling the branch, not inferred from a suite run
The question is what this fixture does on one input. Deleting the
.sodoes not answer it — on this host a shared editable install leaves another copy importable, which is how two of my earlier attempts reported green while measuring an adjacent property.Regression test
test_api_fixture_fails_in_strict_ci_and_skips_locallyasserts both directions. The strict case is asserted explicitly rather than with a barepytest.raises— a skip escaping the test body makes pytest record the test as skipped with exit 0, which is the exact reporting shape being fixed and would make this test itself fail silently.Failing-first, both directions
pytest.skipthe api fixture SKIPPED under BNG3_CI_STRICT_ORACLES=1 instead of failing ... as it already is for the bng_cpp fixtureWhy this matters here specifically
This is the conftest of the lane whose job is detecting that a parity gate reported green without testing anything.
test_parity_nfsim_seed.py(PR #49) exists to catch a gate that silently skips — and it takes theapifixture, so it could itself have reported15 skipped, exit 0on a misconfigured checkout, with the summary reading as healthy.The mechanism differs from the
tests/python/conftest.pyfilter defect (conftest.py:37strips the extension's ownsys.pathentry, so the import genuinely fails there;grep -c "BioNetGen" tests/validation/conftest.pyreturns 0, so mine imports via a shared artifact). The consequence is identical: a lane that should have run reports a clean skip and exits 0. The fix is the same one line either way. No relation to PR #72, which addresses the same class intests/python/test_immune_models.py.Blast radius
Four
tests/validationmodules take theapifixture. Under the strict flag their behaviour changes from a quiet skip to a loud failure — which is the point. No production file, workflow, or shared file is touched.