Skip to content

fix(validation): make a missing compiled extension fail under strict mode - #75

Merged
akutuva21 merged 1 commit into
mainfrom
fix/parity-workflow-duplicate-keys
Oct 1, 2026
Merged

akutuva21 merged 1 commit into
mainfrom
fix/parity-workflow-duplicate-keys

Conversation

@akutuva21

Copy link
Copy Markdown
Member

The api fixture turned a missing engine into a clean skip

tests/validation/conftest.py:70 called pytest.skip directly. A checkout whose compiled extension was not importable therefore reported every test taking that fixture as SKIPPED with exit 0 — including under BNG3_CI_STRICT_ORACLES=1, the flag whose entire purpose is to turn a missing engine into a failure.

This is the same defect the fixture three lines above already avoids:

tests/validation/conftest.py:25   from tests.validation.strict import require_oracle
tests/validation/conftest.py:53   def bng_cpp(request):
tests/validation/conftest.py:56       require_oracle(p is not None, "bng_cpp CLI not found ...")
tests/validation/conftest.py:70   def api(have_api):
tests/validation/conftest.py:72       pytest.skip("compiled bionetgen extension not importable")   # bypassed it

One file, two treatments of the same question. Routing the same branch through require_oracle makes them consistent, and makes the trade the file already made explicit: a missing engine fails in strict mode, and a local run without a build still skips.

Proved by calling the branch, not inferred from a suite run

The question is what this fixture does on one input. Deleting the .so does not answer it — on this host a shared editable install leaves another copy importable, which is how two of my earlier attempts reported green while measuring an adjacent property.

BNG3_CI_STRICT_ORACLES unset, conftest.api(False)  ->  pytest.skip    (unchanged local contract)
BNG3_CI_STRICT_ORACLES=1,     conftest.api(False)  ->  pytest.fail    (was: pytest.skip)

Regression test

test_api_fixture_fails_in_strict_ci_and_skips_locally asserts both directions. The strict case is asserted explicitly rather than with a bare pytest.raises — a skip escaping the test body makes pytest record the test as skipped with exit 0, which is the exact reporting shape being fixed and would make this test itself fail silently.

Failing-first, both directions

conftest result
reverted to the bare pytest.skip 1 failed — the api fixture SKIPPED under BNG3_CI_STRICT_ORACLES=1 instead of failing ... as it already is for the bng_cpp fixture
this branch 1 passed
pytest tests/validation/test_harness_paths.py         -> 12 passed, 1 skipped
pytest tests/validation/test_parity_workflow_wiring.py -> 6 passed
pytest tests/test_ci_contract.py                      -> 38 passed

Why this matters here specifically

This is the conftest of the lane whose job is detecting that a parity gate reported green without testing anything. test_parity_nfsim_seed.py (PR #49) exists to catch a gate that silently skips — and it takes the api fixture, so it could itself have reported 15 skipped, exit 0 on a misconfigured checkout, with the summary reading as healthy.

The mechanism differs from the tests/python/conftest.py filter defect (conftest.py:37 strips the extension's own sys.path entry, so the import genuinely fails there; grep -c "BioNetGen" tests/validation/conftest.py returns 0, so mine imports via a shared artifact). The consequence is identical: a lane that should have run reports a clean skip and exits 0. The fix is the same one line either way. No relation to PR #72, which addresses the same class in tests/python/test_immune_models.py.

Blast radius

Four tests/validation modules take the api fixture. Under the strict flag their behaviour changes from a quiet skip to a loud failure — which is the point. No production file, workflow, or shared file is touched.

…mode

The `api` fixture called pytest.skip directly, so a checkout whose compiled
extension was not importable reported every test taking that fixture as
SKIPPED with exit 0 — including under BNG3_CI_STRICT_ORACLES=1, the flag whose
entire purpose is to turn a missing engine into a failure.

This is the same defect as the one the bng_cpp fixture three lines above
already avoids: conftest.py imports require_oracle at :25 and routes bng_cpp
through it at :56, while api at :70 bypassed it. Routing the same branch
through require_oracle makes the two consistent, and makes the trade the file
already made explicit: a missing engine fails in strict mode, and a local run
without a build still skips.

Proved by calling the branch directly rather than inferring it from a suite
run, because the question is what this fixture does on one input:

    BNG3_CI_STRICT_ORACLES unset, conftest.api(False) -> pytest.skip
    BNG3_CI_STRICT_ORACLES=1,     conftest.api(False) -> pytest.fail   (was: pytest.skip)

Regression test asserts both directions, and asserts the strict case
explicitly rather than letting a skip escape: a skip raised out of a test body
makes pytest record the test as skipped with exit 0, which is the reporting
shape being fixed and would make this test itself fail silently.

Failing-first, both directions:

    conftest reverted to the bare pytest.skip  ->  1 failed
        "the api fixture SKIPPED under BNG3_CI_STRICT_ORACLES=1 instead of
         failing ... as it already is for the bng_cpp fixture"
    this branch                             ->  1 passed

    pytest tests/validation/test_harness_paths.py     -> 12 passed, 1 skipped
    pytest tests/validation/test_parity_workflow_wiring.py -> 6 passed
    pytest tests/test_ci_contract.py                  -> 38 passed

No production file touched. Blast radius is the four tests/validation modules
that take the `api` fixture, and under the strict flag their behaviour changes
from a quiet skip to a loud failure — which is the point.
@akutuva21
akutuva21 merged commit 6f52c86 into main Oct 1, 2026
9 of 30 checks passed
@akutuva21
akutuva21 deleted the fix/parity-workflow-duplicate-keys branch October 4, 2026 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant