Skip to content

fix(release): the post-push integrity check could never pass, and ran after the push - #672

Merged
IanFrelinger merged 3 commits into
masterfrom
fix/release-post-push-verification
Sep 29, 2026
Merged

IanFrelinger merged 3 commits into
masterfrom
fix/release-post-push-verification

Conversation

@IanFrelinger

Copy link
Copy Markdown
Owner

The post-push integrity check could never pass, and it runs after the push

reusable-release-nuget.yml hashes every .nupkg before pushing (line 87), pushes (line 187), then downloads what nuget.org serves and asserts the SHA-256 matches (line 251).

Those bytes are never equal. nuget.org repository-signs on publish, appending a .signature.p7s entry. Measured on the real feed:

$ curl -s https://api.nuget.org/v3-flatcontainer/ashlar.sdk/0.1.2/ashlar.sdk.0.1.2.nupkg
served bytes : 42664
sha256       : eb2111dcaa00867068186ca61fbb0533366bf329e823800fc69ff7ebf45c21d3
entries      : ... , .signature.p7s (12987 bytes)

That is not the hash of the file that was pushed. And it fails at the worst moment: dotnet nuget push has already run, and nuget.org permits unlisting but not deletion, so the release is public and unrecallable by the time the step reports.

Why nobody noticed

It has never executed. The v0.1.2 release (run 33867752688) failed one step earlier, on flat-container index lag — 12 attempts × 15s against an index that had not caught up. The byte comparison was never reached.

That earlier step has since had its budget raised to 40 × 15s. So the next release is the first time this one runs, and it will fail. A check that cannot pass, sitting behind a check that failed, is indistinguishable from a check that works.

What it compares now

Every archive entry except the signature, by name and uncompressed content (scripts/lib/nupkg_content_digest.py) — the part publication must not alter, and the part a consumer restores. Verified against the served package: signing appends .signature.p7s and touches nothing else; [Content_Types].xml does not even declare the p7s extension.

The signature's presence is now asserted positively and separately, so excluding it from the digest loses nothing. The manifest keeps raw sha256 as provenance and adds contentSha256; a manifest written by the old script has no contentSha256, and the verifier refuses it by name rather than silently passing. Both sides import the same module, so they cannot drift into computing different things and agreeing anyway. A run that verified zero packages is also refused.

The test holds two properties that must be true at once

appending a repository signature must not change the digest or every release fails
changing any packaged byte must change it or nothing is caught

It also asserts that the raw file hash does move when a signature is appended — the fact that made the original check impossible — so the reason stays visible.

Mutation-checked, all three caught: removing the signature exclusion, ignoring entry content, dropping the length prefix. The length prefix initially survived its mutation, so frame_entry was extracted as a pure function and the injectivity property is now asserted on chosen bytes rather than left as a claim no test could reach.

Second hole, same publish path

release-nuget.yml is a workflow_dispatch taking an arbitrary version and handing it straight to the same reusable publish job — with none of release.yml's guards. A dispatch could publish packages whose version disagreed with the assemblies inside them, or leave consumer-template pinning a different release.

It now runs the same validate job first: SDK pin, assert_dispatch_version_allowed, assert_consumer_pin_matches. The guards are duplicated rather than shared because a reusable workflow cannot gate its own caller, and a guard the caller may skip is not a guard.

Not changed

ci/published-version reads 0.1.2 and that is correct — it names what is on nuget.org, not what is being cut (consumer-template/CONSUMING.md:5). release.yml's guard deliberately requires it to move on the release commit. The gap is that the only written tag procedure (CHANGELOG.md:7) does not mention that step; that is a changelog fix, kept out of this PR so the two do not conflict.

🤖 Generated with Claude Code

… after the push

WHAT WAS WRONG

reusable-release-nuget.yml hashes every .nupkg before pushing (step "Render NuGet release
manifest", line 87), pushes (line 187), then downloads what nuget.org serves and asserts
the SHA-256 matches (line 251).

Those bytes are never equal. nuget.org repository-signs on publish, appending a
`.signature.p7s` entry to the archive. Measured on the real feed: ashlar.sdk 0.1.2 as
served carries that entry (12987 bytes) and hashes to eb2111dc..., which is not the hash
of the file that was pushed. The check could only fail.

It fails at the worst possible moment. `dotnet nuget push` has already run, and nuget.org
permits unlisting but not deletion, so the release is public and unrecallable by the time
the step reports.

WHY NOBODY NOTICED

It has never executed. The v0.1.2 release (run 33867752688) failed one step earlier, on
flat-container index lag - 12 attempts x 15s against an index that had not caught up - so
the byte comparison was never reached. That earlier step has since had its budget raised
to 40 x 15s, which means the next release is the first time this one runs, and it will
fail.

A check that cannot pass, sitting behind a check that failed, is indistinguishable from a
check that works.

WHAT IT COMPARES NOW

Every archive entry except the signature, by name and uncompressed content
(scripts/lib/nupkg_content_digest.py). That is the part publication must not alter and the
part a consumer restores. Verified against the served package: signing appends
`.signature.p7s` and touches nothing else - `[Content_Types].xml` does not even declare the
p7s extension. The signature's presence is now asserted positively and separately, so
excluding it from the digest loses nothing.

The manifest keeps the raw `sha256` as provenance for the artifact we produced, and adds
`contentSha256`. A manifest written by the old script has no `contentSha256`, and the
verifier refuses it by name rather than silently passing.

Both sides import the same module, so they cannot drift into computing different things
and agreeing anyway. The verifier also refuses a run that verified zero packages, which is
what an empty manifest or an all-skipped version match would otherwise look like.

tests/scripts/nupkg-content-digest.test.sh holds the two properties that have to be true at
once: appending a repository signature must NOT change the digest, or every release fails;
and changing any packaged byte MUST change it, or nothing is caught. It also asserts that
the raw file hash DOES move when a signature is appended - the fact that made the original
check impossible - so the reason stays visible. Mutation-checked: removing the signature
exclusion, ignoring entry content, and dropping the length prefix are each caught.

SECOND HOLE, SAME PUBLISH PATH

release-nuget.yml is a workflow_dispatch taking an arbitrary `version` and handing it
straight to the same reusable publish job, with none of release.yml's guards. A dispatch
could publish packages whose version disagreed with the assemblies inside them, or leave
consumer-template pinning a different release. It now runs the same validate job first
(SDK pin, assert_dispatch_version_allowed, assert_consumer_pin_matches). The guards are
duplicated rather than shared because a reusable workflow cannot gate its own caller.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@IanFrelinger
IanFrelinger force-pushed the fix/release-post-push-verification branch from 8b2d66e to 621c85b Compare September 29, 2026 15:35
IanFrelinger and others added 2 commits September 29, 2026 12:21
`lychee (README + docs)` is a required check with NO path filter on pull_request, so it
runs on every PR. It fails on two links in docs/handoff/HANDOFF.md:

  [403] https://claude.ai/code/artifact/a1b2328c-... (at 38:4)
  [403] https://claude.ai/code/artifact/bcfebaf4-... (at 40:4)

They are PRIVATE artifacts from the session that originated the handoff notes. claude.ai
answers an unauthenticated fetch with 403 by design, so a link checker can never verify
them - it is measuring claude.ai's auth behaviour, not anything about this repository.

This is not a regression from this PR. The links have been on master since 2026-08-22
(#362). The workflow's `push` trigger had not re-run over that file since 2026-09-10, and
its `pull_request` trigger has no path filter, so the first PR to reach the check inherited
a defect that had been sitting on master for five weeks - and it blocks every PR until it
is excluded.

Excluded rather than deleted: the links are the only pointer to that reference material
for whoever can open them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@IanFrelinger

Copy link
Copy Markdown
Owner Author

Added one commit unrelated to the release machinery, because it was blocking this PR and every other one.

lychee (README + docs) is a required check with no path filter on pull_request, so it runs on every PR. It fails on two links in docs/handoff/HANDOFF.md:

[403] https://claude.ai/code/artifact/a1b2328c-... (at 38:4)
[403] https://claude.ai/code/artifact/bcfebaf4-... (at 40:4)

Those are private Claude artifacts from the session that originated the handoff notes. claude.ai answers an unauthenticated fetch with 403 by design, so a link checker can never verify them — it is measuring claude.ai's auth behaviour, not anything about this repository.

Not a regression from this PR. The links have been on master since 2026-08-22 (#362). The workflow's push trigger had not re-run over that file since 2026-09-10, and its pull_request trigger has no path filter — so the first PR to reach the check inherited a defect that had been on master for five weeks.

Excluded in .lycheeignore rather than deleted: the links are the only pointer to that reference material for whoever can open them.

@IanFrelinger
IanFrelinger merged commit 2a416c8 into master Sep 29, 2026
14 checks passed
@IanFrelinger
IanFrelinger deleted the fix/release-post-push-verification branch September 29, 2026 16:34
IanFrelinger added a commit that referenced this pull request Sep 29, 2026
…se, and I made it worse (#682)

An expert panel reviewing the release before the v0.2.0 tag found that the post-push chain I
rewrote in #672 was not fixed, only half fixed - and that three structural faults around it
would turn any post-push failure into a release that is public but unrecorded. The tag was held
on this. All four are fixed here, each proved by a test or a structural check.

1. MY VERIFIER DIED ON THE FIRST 404 - the same index lag that failed v0.1.2.

#672 fixed the byte comparison and kept the download line as it was: a bare
urllib.request.urlretrieve, no timeout, no retry, no HTTPError handling. nuget.org indexes
packages one at a time after a push, and the visibility poll before this step waited for only
four ids, so the other ~18 could still 404 on the first request. That raised an uncaught
HTTPError after every package was already public. My own header comment named the lag.

It now retries retryable statuses (404, 408, 429, 5xx) and transport errors with a timeout,
records a NAMED failure instead of raising, and continues so every failure is reported at
once. A non-retryable status (403) fails fast. The feed URL is now overridable, both so this
can be tested and so a staging feed can be verified by the same code rather than a copy.

tests/scripts/nuget-verify-retry.test.sh drives the REAL script against a local HTTP server
that serves 404 for a while and then the package. Eight assertions: lag is ridden out, a
package that never appears is a named failure and not a traceback, it is retried the
configured number of times, the loop continues past it, a 403 fails fast. With the original
bare urlretrieve restored the test reproduces exactly the predicted failure -
"urllib.error.HTTPError: HTTP Error 404: Not Found" - and fails.

Getting the test to run at all surfaced the friction it is about: python's
subprocess.run(["bash", ...]) on Windows resolves bare `bash` to System32\bash.exe, the WSL
launcher, which fails before the verifier starts. The server received nothing and two
assertions "passed" having measured nothing. It now hands python the bash running the test.

2. THE VISIBILITY POLL WAITED FOR FOUR OF ~22 PACKAGES.

NUGET_POST_PUSH_VERIFY_PACKAGE_IDS is unset, so it fell back to a hard-coded
Hosting.Bundle, Hosting, Sdk, CLI. It now defaults to every id in the release manifest, which
already exists at that point; an explicit operator list still overrides it.

3. GHCR AND NUGET PUBLISHED IN PARALLEL, WITH NO EDGE BETWEEN TWO IRREVERSIBLE ACTS.

Both `images` and `nuget` needed only `validate`. In the v0.1.2 run the nexo-cli semver retag
completed while the NuGet job was still running, and then NuGet failed. `images` now needs
`nuget` to have SUCCEEDED. The order is deliberate: GHCR tags can be re-pointed or deleted,
nuget.org packages can only be unlisted, so the irreversible side goes first and the second
promise is made only once the first is kept.

4. A POST-PUSH FAILURE SKIPPED THE ONLY DURABLE COPY OF WHAT SHIPPED.

`github-release` was `if: success()` over `needs: [validate, images, nuget]`. When any post-push
step failed it was skipped - empirically `skipped` in run 33867752688 - so the nupkgs and
nuget-publish-manifest.json survived only as a 90-day Actions artifact, for packages that are
permanent. That is why the v0.1.2 release page was assembled by hand.

A step now records that irreversible publication BEGAN, immediately before the first push -
not after the last, since the push loops run under -e and a failure half way still leaves some
packages public. It is plumbed out as a job and workflow_call output, and `github-release` runs
under always() gated on it. The release is created as a DRAFT, so running after a failure
publishes nothing, and the draft now opens with a warning when the NuGet job did not succeed,
and says plainly when images were not published rather than printing empty tags.

The image tags moved from inline ${{ }} in the run: script into env:, which is the safer shape.

5. THE REHEARSAL EXISTED, AND THE RUNBOOK NEVER SAID SO.

I told the maintainer no rehearsal was possible because NUGET_PUBLISH_MODE=oidc means staging
would publish for real. That was wrong. `0.2.0-rc1` passes both guards - checked against
master - and runs the entire post-push chain against the real feed, at the cost of one
permanent prerelease no 0.2.0 pin resolves to. It is now step 8 of "Before you tag".

WHAT THIS DOES NOT PROVE. Items 3 and 4 change the job graph of the release workflow, which
cannot be run locally. They are checked structurally - both workflows parse, the needs and if
expressions are the intended ones, the marker precedes both push paths - but a workflow graph
is only proved by running it. That is what the rc1 rehearsal is for.

Co-authored-by: t <t@t>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant