Skip to content

ci(provenance): make the strict provenance gate visible and ratcheted - #81

Closed
akutuva21 wants to merge 1 commit into
mainfrom
agent/provdocs
Closed

akutuva21 wants to merge 1 commit into
mainfrom
agent/provdocs

Conversation

@akutuva21

Copy link
Copy Markdown
Member

The defect

scripts/validate_provenance.py --require-approved fails with 13 errors and was wired into no CI job. Parsing all seven workflows for any validate_provenance invocation returns exactly two, both non-strict:

workflow job strict?
ci.yml lint no
parity.yml oracle-lock no

Total strict invocations: 0. A gate that fails and is never invoked reads as a passing gate to anyone who sees it in the tree.

The 13 errors are not 13 unsupported claims

This is the part that changes the disposition. They are one unapproved baseline plus 12 status transitions awaiting a human decision, and every field already carries a recorded revision or an explicit null. The repository is honest about being unapproved; the gate was invisible, not wrong.

# file:line error disposition
1 upstreams.lock.yml:6 approved baseline blocked — maintainer decision. provenance/README.md:34: "Do not change the status to approved until those decisions are recorded."
2 :22 sources.bng3.status=accepted blocked — plus the recorded revision 7b61ecc7 predates current main 9f840a5
3 :33 sources.bionetgen.status=accepted blocked — remote-head evidence 2026-09-23
4 :44 sources.nfsim.status=accepted blocked — remote-head evidence 2026-09-23
5 :55 sources.nfsim-rasi.status=accepted blocked — a local fork branch; whether it is an upstream at all is a maintainer call
6 :66 sources.bngsim.status=accepted blocked — ADR 0003 names it, but that decision lives in a design doc, not the lock
7 :77 sources.pybionetgen.status=accepted blocked — remote-head evidence 2026-09-23
8 :88 sources.bngplayground.status=accepted blocked — local checkout of a feature branch
9 :99 sources.rulehub.status=accepted blocked, and not reachable by approval alone — see below
10 :109 oracles.bng2.status=locked blocked — build_recipe and artifact_digest are null; needs a real build
11 :115 oracles.nfsim.status=locked blocked — same
12 :122 compiler_images.status=locked blocked — the lint job runs no container, so there is no image to pin
13 :126 python_lock.status=locked blocked — lint installs unpinned (pip install black ruff pytest numpy); no lock file governs it

Counts: 0 written, 0 removed, 13 explicitly marked blocked. The diff is 384 insertions and 0 deletions — no claim was deleted to make a gate green.

#9 is the finding that matters most. validate_provenance.py rejects accepted when evidence.kind is planning-snapshot, and rulehub is exactly that (checked_at 2026-08-28, the oldest evidence in the file). Flipping the baseline and all sources to accepted still fails on that one line. So approval alone does not clear the gate; rulehub needs re-verification against the remote first.

Strict mode is not achievable today — so CI runs a ratchet, not a red job

Shipping --require-approved as a job would be a permanently failing job, which is not a fix. Instead the lint job gains python scripts/check_provenance_ratchet.py, which fails if the blocker set grows (new unrecorded debt), shrinks (a decision was made, ledger stale), or if the strict gate ever starts passing (the signal to retire the ratchet).

Verified: ratchet green at 13. Three mutations caught — an unrecorded source, a resolved blocker, and a fully approved lock. That third mutation also demonstrates strict mode is reachable once rulehub is re-verified.

New tests fail without the wiring (AssertionError: no CI job runs the strict provenance ratchet; the strict gate is invisible again) and without the script.

What strict mode would additionally require

  1. A maintainer records the import cutoffs, oracle recipes, image digests, and ledger owners.
  2. Re-verify rulehub against the remote so its evidence stops being planning-snapshot.
  3. Build BNG2 and NFsim, record real sha256 artifact digests and build recipes.
  4. Pin the CI compiler images to immutable @sha256 digests.
  5. Commit a Python lock file and record its digest.

Doc claim verified

The assignment attributed to AGENTS.md a claim that policy gates "run exactly what the lint job runs". AGENTS.md contains no such claim — 0 occurrences of "lint", no job-level CI spec. The claim actually lives in docs/DEVELOPMENT_CHECKLIST.md:23-32 and docs/VALIDATION_EVIDENCE.md:273-282, and I verified it is accurate: all five listed commands are in lint, as are Black, Ruff, and the golden-manifest loop. The only uncovered step is pip install black ruff pytest numpy (setup, not a gate). Both doc blocks are now stale — the strict gate is collected by a job. @DocsDrift owns that edit; I did not touch docs/.

not_claimed

  • I did not resolve any of the 13. All remain, each with a recorded reason. Four need artifacts that do not exist yet.
  • I did not verify the recorded revisions against their remotes — no network fetch. BNG3: integrate native backend and parity safeguards #2's staleness (7b61ecc7 vs 9f840a5) is from reading the file, not from contacting a remote.
  • No project-wide validation run. Siblings are mid-flight; I ran only tests/validation/test_provenance.py (12 passed), Black, and ruff on my own files.
  • I did not build either oracle, so I cannot say the two digests are obtainable in practice, only that they must describe real bytes.
  • The mutation results come from synthetic locks built in /tmp, not from the live tree.

The strict gate (scripts/validate_provenance.py --require-approved) fails
with 13 errors and was wired into no CI job -- only the non-strict call ran,
from ci.yml lint and parity.yml oracle-lock. Parsing all seven workflows
confirms zero strict invocations. A gate that fails and is never invoked
reads as a passing gate to anyone who sees it in the tree.

Those 13 errors are not 13 unsupported claims. They are one unapproved
baseline plus 12 status transitions awaiting a human decision, and each
already carries a recorded revision or an explicit null. They are honestly
recorded as pending, not missing. Dispositions and per-blocker evidence are
in provenance/strict-gate-ratchet.json.

Strict mode is not achievable today: approving the baseline requires a
maintainer decision (provenance/README.md:34 says so explicitly), the two
oracle digests must describe bytes that exist, and rulehub cannot be accepted
from planning-snapshot evidence at all. Shipping the strict call as a CI job
would be a permanently red job, which is not a fix. So CI runs a ratchet
instead: it fails if the blocker set grows (new unrecorded debt) or shrinks
(a decision was made and the ledger is stale), and it fails loudly if the
strict gate ever starts passing, which is the signal to retire it.

Verified: ratchet green at 13; three mutations caught -- an unrecorded
source, a resolved blocker, and a fully approved lock. That last one also
shows strict mode is reachable once rulehub is re-verified against the
remote. New tests fail without the wiring and without the script.
@akutuva21

Copy link
Copy Markdown
Member Author

Closing as stale: branch predates the current CI harness (missing needs:[cpp-build]/BNG3_PYTHON_TEST_MODE bootstrap) and its ratchet pins 13 strict-gate errors while main now emits 14. Please regenerate the ratchet on current main and re-apply the CI hunk fresh.

@akutuva21 akutuva21 closed this Oct 5, 2026
@akutuva21
akutuva21 deleted the agent/provdocs branch October 6, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant