Repository navigation
Conversation
The strict gate (scripts/validate_provenance.py --require-approved) fails with 13 errors and was wired into no CI job -- only the non-strict call ran, from ci.yml lint and parity.yml oracle-lock. Parsing all seven workflows confirms zero strict invocations. A gate that fails and is never invoked reads as a passing gate to anyone who sees it in the tree. Those 13 errors are not 13 unsupported claims. They are one unapproved baseline plus 12 status transitions awaiting a human decision, and each already carries a recorded revision or an explicit null. They are honestly recorded as pending, not missing. Dispositions and per-blocker evidence are in provenance/strict-gate-ratchet.json. Strict mode is not achievable today: approving the baseline requires a maintainer decision (provenance/README.md:34 says so explicitly), the two oracle digests must describe bytes that exist, and rulehub cannot be accepted from planning-snapshot evidence at all. Shipping the strict call as a CI job would be a permanently red job, which is not a fix. So CI runs a ratchet instead: it fails if the blocker set grows (new unrecorded debt) or shrinks (a decision was made and the ledger is stale), and it fails loudly if the strict gate ever starts passing, which is the signal to retire it. Verified: ratchet green at 13; three mutations caught -- an unrecorded source, a resolved blocker, and a fully approved lock. That last one also shows strict mode is reachable once rulehub is re-verified against the remote. New tests fail without the wiring and without the script.
Member
Author
|
Closing as stale: branch predates the current CI harness (missing needs:[cpp-build]/BNG3_PYTHON_TEST_MODE bootstrap) and its ratchet pins 13 strict-gate errors while main now emits 14. Please regenerate the ratchet on current main and re-apply the CI hunk fresh. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
scripts/validate_provenance.py --require-approvedfails with 13 errors and was wired into no CI job. Parsing all seven workflows for anyvalidate_provenanceinvocation returns exactly two, both non-strict:ci.ymllintparity.ymloracle-lockTotal strict invocations: 0. A gate that fails and is never invoked reads as a passing gate to anyone who sees it in the tree.
The 13 errors are not 13 unsupported claims
This is the part that changes the disposition. They are one unapproved baseline plus 12 status transitions awaiting a human decision, and every field already carries a recorded revision or an explicit
null. The repository is honest about being unapproved; the gate was invisible, not wrong.upstreams.lock.yml:6provenance/README.md:34: "Do not change the status toapproveduntil those decisions are recorded.":22sources.bng3.status=accepted7b61ecc7predates current main9f840a5:33sources.bionetgen.status=accepted:44sources.nfsim.status=accepted:55sources.nfsim-rasi.status=accepted:66sources.bngsim.status=accepted:77sources.pybionetgen.status=accepted:88sources.bngplayground.status=accepted:99sources.rulehub.status=accepted:109oracles.bng2.status=lockedbuild_recipeandartifact_digestarenull; needs a real build:115oracles.nfsim.status=locked:122compiler_images.status=locked:126python_lock.status=lockedpip install black ruff pytest numpy); no lock file governs itCounts: 0 written, 0 removed, 13 explicitly marked blocked. The diff is 384 insertions and 0 deletions — no claim was deleted to make a gate green.
#9 is the finding that matters most.
validate_provenance.pyrejectsacceptedwhenevidence.kindisplanning-snapshot, and rulehub is exactly that (checked_at 2026-08-28, the oldest evidence in the file). Flipping the baseline and all sources toacceptedstill fails on that one line. So approval alone does not clear the gate; rulehub needs re-verification against the remote first.Strict mode is not achievable today — so CI runs a ratchet, not a red job
Shipping
--require-approvedas a job would be a permanently failing job, which is not a fix. Instead thelintjob gainspython scripts/check_provenance_ratchet.py, which fails if the blocker set grows (new unrecorded debt), shrinks (a decision was made, ledger stale), or if the strict gate ever starts passing (the signal to retire the ratchet).Verified: ratchet green at 13. Three mutations caught — an unrecorded source, a resolved blocker, and a fully approved lock. That third mutation also demonstrates strict mode is reachable once rulehub is re-verified.
New tests fail without the wiring (
AssertionError: no CI job runs the strict provenance ratchet; the strict gate is invisible again) and without the script.What strict mode would additionally require
planning-snapshot.sha256artifact digests and build recipes.@sha256digests.Doc claim verified
The assignment attributed to
AGENTS.mda claim that policy gates "run exactly what thelintjob runs".AGENTS.mdcontains no such claim — 0 occurrences of "lint", no job-level CI spec. The claim actually lives indocs/DEVELOPMENT_CHECKLIST.md:23-32anddocs/VALIDATION_EVIDENCE.md:273-282, and I verified it is accurate: all five listed commands are inlint, as are Black, Ruff, and the golden-manifest loop. The only uncovered step ispip install black ruff pytest numpy(setup, not a gate). Both doc blocks are now stale — the strict gate is collected by a job.@DocsDriftowns that edit; I did not touchdocs/.not_claimed
7b61ecc7vs9f840a5) is from reading the file, not from contacting a remote.tests/validation/test_provenance.py(12 passed), Black, and ruff on my own files./tmp, not from the live tree.