Skip to content

Audit a Caller's workflow-model Input Against the Registry - #3466

Merged
ptr727 merged 3 commits into
developfrom
feature/auto-3332
Oct 25, 2026
Merged

ptr727 merged 3 commits into
developfrom
feature/auto-3332

Conversation

@ptr727

@ptr727 ptr727 commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Adds an audit check for the workflow-model input a caller passes to validate-task.yml. Also moves the no-build caller snippet onto a release that declares the input.

  • interface.workflow-model (spec/audit.py, spec/checks.json) reads the validate job's with: workflow-model in .github/workflows/test-pull-request.yml. It reports DRIFT in two cases: an operational registry entry that passes no workflow-model: operational, and a release entry that passes any value other than release or nothing. An empty value counts as a value rather than as the default, because the validator treats any value other than release as operational.
  • One model resolver. workflow_model_of replaces the three inline workflowModel fallbacks in spec/audit.py, used by the selectors, the rulesets check and the sweep header.
  • Snippet. catalog/snippets/workflows/test-pull-request.yml moves its pin from 2.0.376 to 2.0.1166 and passes workflow-model: operational. The snippet is the operational trigger shape. Its only other job-level setting is permissions: contents: read, which still matches every job in the pinned validate-task.yml.
  • Docs. docs/reusable-workflows.md "Adopting the Gates" now says a release-model adopter drops the input, along with the push trigger. It also names the new check.
  • Tests. WorkflowModelCallerFindingsCase covers an operational caller with the input, an operational caller without it, a release caller passing the wrong value, release callers that omit it or name release, an empty value, callers and jobs the check does not read, the real snippet, and the model fallback.

Verification: I ran the full OPERATIONS.md "Local Verification" block locally, and every gate passed (2919 tests). I also ran a live spec/audit.py over the hub and the operational and release config repos. The new check reports exactly the three operational repos whose caller omits the input, and none of the release repos.

Closes on promotion: #3332

Declined local-pass finding

  • spec/audit.py workflow_model_caller_findings, introduced: the local pass called the structured-value branch added in 3426a19 over-hardening. Declined, because the branch is not hardening against an input. The check exists to report a caller whose input disagrees with the registry. Reading a mapping as the omitted default would make the audit report a release caller as conformant when it is not, which is a false negative in the check's own output.

Pre-existing findings, not filed

  • docs/reusable-workflows.md:300, pre-existing: the paragraph's first sentence is 40 words, over sentence-length's 25. The sentence predates this change. This change only appended a separate 13-word sentence to that line.

Dropped findings

  • scripts/carry.py selector_set() and repo-config/configure.sh, pre-existing consolidation: both keep their own copy of the workflowModel fallback that workflow_model_of now holds for spec/audit.py. This change does not touch either file.

🤖 Generated with Claude Code

ptr727 and others added 2 commits October 24, 2026 21:42
Add the interface.workflow-model check. It reports an operational
registry entry whose test-pull-request.yml validate job passes no
workflow-model: operational, and a release entry passing any value
other than release. The registry's model resolution moves into one
helper that the selectors, the rulesets check, and the sweep share.

Advance the no-build caller snippet's validate-task.yml pin to
2.0.1166, the latest release carrying the input, and pass
workflow-model: operational, since the snippet is the operational
trigger shape. The adoption doc now says a release-model adopter drops
the input along with the push trigger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 25, 2026 04:47
@ptr727

ptr727 commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head f987b3811f9c7cff14107c014d9d7aefe8a5c427, the content this pull request carries at that commit against develop, and it recorded 0 findings.

@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Structured workflow-model values can be incorrectly accepted as omitted inputs for release callers.

No open issues 🎉

What changed in this PR

This pull request adds registry-aware auditing for callers’ workflow-model inputs, updates the operational workflow snippet, and documents the release-versus-operational behavior.

Changes:

  • Adds the interface.workflow-model audit check and centralizes workflow-model resolution.
  • Adds focused tests for caller validation, fallbacks, and edge cases.
  • Updates the operational snippet to use a compatible release and pass workflow-model: operational.
  • Documents the new caller requirements and audit check.

An unresolved moderate [P1] issue remains: structured YAML values such as {} or [] may be treated as omitted inputs for release callers.

File Summary
tests/​test_audit_helpers.py Tests workflow-model caller validation, fallback behavior, edge cases, and the updated snippet.
spec/​checks.json Registers the new workflow-model interface audit check.
spec/​audit.py Implements caller-input auditing and centralizes workflow-model resolution.
docs/​reusable-workflows.md Documents release and operational caller behavior and the new audit check.
catalog/​snippets/​workflows/​test-pull-request.yml Pins the operational snippet to a compatible release and passes the operational model.

Review effort: Lite

…itted

A mapping or a sequence passed as the input read as no value, so a
release caller passing one was accepted as using the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.59259% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@bfd8acb). Learn more about missing BASE report.

Files with missing lines Patch % Lines
spec/audit.py 92.59% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             develop    #3466   +/-   ##
==========================================
  Coverage           ?   80.92%           
==========================================
  Files              ?       23           
  Lines              ?     8869           
  Branches           ?        0           
==========================================
  Hits               ?     7177           
  Misses             ?     1692           
  Partials           ?        0           
Flag Coverage Δ
python-3.13 80.92% <92.59%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ptr727

ptr727 commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 3426a19cdc4a69e5f6776bd6267e21a2d2b07882, the content this pull request carries at that commit against develop, and it recorded 1 finding.

@ptr727

ptr727 commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Answering the review round on f987b38, whose headline named one finding (1) with no thread:

spec/audit.py workflow_model_caller_findings: "Structured workflow-model values can be incorrectly accepted as omitted inputs for release callers."

Fixed in 3426a19. A mapping or a sequence now reports as "passes a mapping or a sequence" for either model, rather than reading as the omitted default, and test_a_structured_value_is_not_the_default covers it.

@ptr727
ptr727 merged commit 003075d into develop Oct 25, 2026
13 checks passed
@ptr727
ptr727 deleted the feature/auto-3332 branch October 25, 2026 05:03
ptr727 added a commit that referenced this pull request Oct 25, 2026
…nd Nine Fixes to Main (#3472)

## Summary

Promotes the pull requests merged into develop since release 2.0.1166.

- [#3444](#3444): keeps
untracked files out of the pre-commit snippet's prose gate.
- [#3446](#3446): exempts
the BuildKit `check=` parser directive in the prose gate.
- [#3448](#3448): compares
intent staleness against the hub's own copy and names the moved
canonical.
- [#3450](#3450): says
only the prose gate is diff-scoped in the commit hook.
- [#3453](#3453):
reclassifies HolidayLights as an operational config repository in the
registry.
- [#3455](#3455):
classifies the remaining subprocess failures by status rather than by
prose.
- [#3464](#3464): splits
the semicolon series in WORKFLOW.md's filesystem output-seam bullet.
- [#3466](#3466): audits a
caller's `workflow-model` input against the registry.
- [#3469](#3469): emits
one snapshot shape from the skills report's cannot-read paths.
- [#3462](#3462): splits
`pr_review.py`'s pure check and round readers into the leaf modules
`pr_review_checks.py` and `pr_review_rounds.py`.
- [#3470](#3470): holds a
covered held `wait` while any workflow run on the head has not
completed.
- [#3473](#3473): answers
this promotion's review. The `wait` help and `scripts/README.md` now say
what a held head's exit `0` does not prove: a `CHECKS UNREADABLE`
payload, the conditions the unregistered-required-check hold reads, and
the reach of the completed-run read. The exit code itself stays for
#3471.

Closes #3370
Closes #3372
Closes #3371
Closes #3369
Closes #3378
Closes #3454
Closes #3332
Closes #1759
Closes #3451

## Constituent coverage

Each change this promotion carries had coverage on its own pull request
into develop, at its merged head:

| Pull request | Merged head | Coverage |
| --- | --- | --- |
| [#3444](#3444) |
`a61d1e2a` | attested local pass at the merged head, base `develop` |
| [#3446](#3446) |
`0a0efea1` | attested local pass at the merged head, base `develop` |
| [#3448](#3448) |
`c2c71ea7` | attested local pass at the merged head, base `develop` |
| [#3450](#3450) |
`4ec37c34` | attested local pass at the merged head, base `develop` |
| [#3453](#3453) |
`f20a543c` | a Copilot round on the merged head, its one changed file,
no open findings |
| [#3455](#3455) |
`85f0ce60` | attested local pass at the merged head, base `develop` |
| [#3464](#3464) |
`fcd48e2d` | attested local pass at the merged head, base `develop` |
| [#3466](#3466) |
`3426a19c` | attested local pass at the merged head, base `develop` |
| [#3469](#3469) |
`ea4b0bb7` | attested local pass at the merged head, base `develop` |
| [#3462](#3462) |
`236eceab` | attested local pass at the merged head, base `develop` |
| [#3470](#3470) |
`3545ba46` | attested local pass at the merged head, base `develop` |
| [#3473](#3473) |
`46010530` | attested local pass at the merged head, base `develop`, and
a Copilot table on an earlier head naming exactly the changed files |

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants