Repository navigation
Package an attested evaluator pilot kit - #28
Merged
Merged
Conversation
Summary by CodeRabbit
WalkthroughThe PR adds a deterministic evaluator pilot kit with strict manifest, checksum, archive, and directory verification. The release workflow builds, attests, uploads, and publishes the kit. Tests and documentation cover creation, validation, onboarding, and release evidence. ChangesEvaluator pilot kit
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ReleaseWorkflow
participant BuiltWheel
participant PilotBundle
participant EvaluatorKit
participant GitHubRelease
ReleaseWorkflow->>BuiltWheel: Build release wheel
ReleaseWorkflow->>PilotBundle: Create kit from wheel and source commit
PilotBundle->>EvaluatorKit: Write manifest, checksums, and deterministic ZIP
ReleaseWorkflow->>PilotBundle: Verify kit archive
PilotBundle-->>ReleaseWorkflow: Return verified summary
ReleaseWorkflow->>GitHubRelease: Upload artifact and release asset
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/release.yml:
- Line 133: Update the release step’s Bash assignment around version so it no
longer interpolates steps.release-version.outputs.version directly into shell
source. Pass the step output through the step’s env configuration, then read the
resulting environment variable inside the script while preserving the existing
version value.
In `@docs/pilot-kit-v1.schema.json`:
- Around line 37-39: Update the path property in the published schema to retain
its non-empty constraint and add a bounded pattern that accepts only safe
canonical relative file paths, rejecting absolute paths, parent-directory
segments, drive prefixes, double slashes, backslashes, control characters, and
explicit directory entries.
In `@docs/PILOT.md`:
- Around line 35-43: Update the native verification command sequences in
docs/PILOT.md lines 35-43 and docs/RELEASING.md lines 70-78: after each gh
attestation command and after python scripts/pilot_bundle.py verify-directory .,
check $LASTEXITCODE and fail immediately when it is non-zero. Keep the existing
extraction and directory validation flow unchanged.
- Around line 238-241: Update the PowerShell wrapper around pilot_check.py to
capture $LASTEXITCODE and explicitly handle 0, 1, and 2, throwing for any
unexpected exit code. Also adjust pilot_check.py’s exception handling so runtime
or invalid failures consistently return the documented failure code rather than
allowing uncaught SystemExit(1), while preserving 1 for “valid but not ready.”
- Line 77: Update the macOS/Linux instructions in PILOT.md to use the
macOS-compatible shasum -a 256 command, or explicitly require GNU Coreutils
before the section; keep the existing hash verification workflow unchanged.
In `@scripts/pilot_bundle.py`:
- Line 83: Update the example record load in the surrounding creation flow to
pass the pilot-record size limit used by _verify_payload, rather than relying on
_load_json_file’s MAX_MANIFEST_BYTES default. Keep the same limit for reading
and verifying this artifact, while preserving existing defaults for other JSON
files.
- Around line 111-113: Move the output filename validation using root_name and
output.name to the beginning of create_pilot_kit, alongside the existing input
checks, before any wheel, repository asset, or example-record reads occur.
Preserve the current PilotKitError message and all existing validation outcomes.
- Around line 114-122: Update the archive.writestr call in the ZipFile creation
block to pass compresslevel=9 explicitly, ensuring entries using the precreated
_zip_info metadata are compressed at the intended deflate level.
In `@tests/test_pilot_bundle.py`:
- Around line 141-150: Add a test alongside
test_verify_directory_rejects_unexpected_file that creates a symlink within the
extracted pilot kit and verifies verify_pilot_kit_directory raises PilotKitError
for it, covering the symbolic-link rejection branch before checksum validation.
Use pytest.mark.skipif with the platform’s symlink-permission condition when
needed for Windows CI.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 76f47c53-12f2-4783-96fe-885737e0fa90
📒 Files selected for processing (13)
.github/workflows/release.ymlCHANGELOG.mdREADME.mdROADMAP.mddocs/COMPETITIVE_STRATEGY.mddocs/PILOT.mddocs/PRODUCTIZATION.mddocs/RELEASING.mddocs/pilot-kit-v1.schema.jsonexamples/README.mdscripts/pilot_bundle.pyscripts/release_check.pytests/test_pilot_bundle.py
Owner
Author
|
Addressed all nine review findings in |
Owner
Author
|
Post-merge verification passed on
Workflow-produced SHA-256:
This remains nonpublishing evidence; it does not claim that the three-developer pilot passed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
python -m ruff check .python -m ruff format --check .python -m mypy src scripts/pilot_bundle.pypython scripts/release_check.py source --version 0.2.0 --allow-unreleasedpython -m pytest -ra— 391 passed--require-cleanpip check, and installedself-check --format jsond0b7e35and all threads resolvedExact reviewed-branch artifacts
Commit:
d0b7e35c47ebff024153d770269d41ebfaf26211af86f34c4a7d895d0de60c4c4f77806d445485c767f761ff05fe68a1086f0959145a81caabbfd70f87ed01c7d659b27740555504d7db974113c5a0ed1e2315192e06ab5c803295e59164ada75375d672bc28e27f9a9b79e3878ae6d2daea3130These hashes identify one fresh-clone build of the final reviewed branch. The kit is byte-deterministic for an identical wheel and source commit; Python distribution builds are not claimed to be reproducible across separate build invocations. The post-merge manual workflow will rebuild and attest default-branch artifacts, whose hashes may differ because the merge commit and distribution bytes are kit inputs.
Boundaries
This does not publish to PyPI, call a live provider, collect participant content, or claim that the three-developer pilot passed. GitHub provenance and internal checksum verification establish build identity/consistency, not software safety or user adoption.