Repository navigation
fix(release): the post-push chain could strand a half-published release, and I made it worse - #682
Merged
Merged
Conversation
…se, and I made it worse An expert panel reviewing the release before the v0.2.0 tag found that the post-push chain I rewrote in #672 was not fixed, only half fixed - and that three structural faults around it would turn any post-push failure into a release that is public but unrecorded. The tag was held on this. All four are fixed here, each proved by a test or a structural check. 1. MY VERIFIER DIED ON THE FIRST 404 - the same index lag that failed v0.1.2. #672 fixed the byte comparison and kept the download line as it was: a bare urllib.request.urlretrieve, no timeout, no retry, no HTTPError handling. nuget.org indexes packages one at a time after a push, and the visibility poll before this step waited for only four ids, so the other ~18 could still 404 on the first request. That raised an uncaught HTTPError after every package was already public. My own header comment named the lag. It now retries retryable statuses (404, 408, 429, 5xx) and transport errors with a timeout, records a NAMED failure instead of raising, and continues so every failure is reported at once. A non-retryable status (403) fails fast. The feed URL is now overridable, both so this can be tested and so a staging feed can be verified by the same code rather than a copy. tests/scripts/nuget-verify-retry.test.sh drives the REAL script against a local HTTP server that serves 404 for a while and then the package. Eight assertions: lag is ridden out, a package that never appears is a named failure and not a traceback, it is retried the configured number of times, the loop continues past it, a 403 fails fast. With the original bare urlretrieve restored the test reproduces exactly the predicted failure - "urllib.error.HTTPError: HTTP Error 404: Not Found" - and fails. Getting the test to run at all surfaced the friction it is about: python's subprocess.run(["bash", ...]) on Windows resolves bare `bash` to System32\bash.exe, the WSL launcher, which fails before the verifier starts. The server received nothing and two assertions "passed" having measured nothing. It now hands python the bash running the test. 2. THE VISIBILITY POLL WAITED FOR FOUR OF ~22 PACKAGES. NUGET_POST_PUSH_VERIFY_PACKAGE_IDS is unset, so it fell back to a hard-coded Hosting.Bundle, Hosting, Sdk, CLI. It now defaults to every id in the release manifest, which already exists at that point; an explicit operator list still overrides it. 3. GHCR AND NUGET PUBLISHED IN PARALLEL, WITH NO EDGE BETWEEN TWO IRREVERSIBLE ACTS. Both `images` and `nuget` needed only `validate`. In the v0.1.2 run the nexo-cli semver retag completed while the NuGet job was still running, and then NuGet failed. `images` now needs `nuget` to have SUCCEEDED. The order is deliberate: GHCR tags can be re-pointed or deleted, nuget.org packages can only be unlisted, so the irreversible side goes first and the second promise is made only once the first is kept. 4. A POST-PUSH FAILURE SKIPPED THE ONLY DURABLE COPY OF WHAT SHIPPED. `github-release` was `if: success()` over `needs: [validate, images, nuget]`. When any post-push step failed it was skipped - empirically `skipped` in run 33867752688 - so the nupkgs and nuget-publish-manifest.json survived only as a 90-day Actions artifact, for packages that are permanent. That is why the v0.1.2 release page was assembled by hand. A step now records that irreversible publication BEGAN, immediately before the first push - not after the last, since the push loops run under -e and a failure half way still leaves some packages public. It is plumbed out as a job and workflow_call output, and `github-release` runs under always() gated on it. The release is created as a DRAFT, so running after a failure publishes nothing, and the draft now opens with a warning when the NuGet job did not succeed, and says plainly when images were not published rather than printing empty tags. The image tags moved from inline ${{ }} in the run: script into env:, which is the safer shape. 5. THE REHEARSAL EXISTED, AND THE RUNBOOK NEVER SAID SO. I told the maintainer no rehearsal was possible because NUGET_PUBLISH_MODE=oidc means staging would publish for real. That was wrong. `0.2.0-rc1` passes both guards - checked against master - and runs the entire post-push chain against the real feed, at the cost of one permanent prerelease no 0.2.0 pin resolves to. It is now step 8 of "Before you tag". WHAT THIS DOES NOT PROVE. Items 3 and 4 change the job graph of the release workflow, which cannot be run locally. They are checked structurally - both workflows parse, the needs and if expressions are the intended ones, the marker precedes both push paths - but a workflow graph is only proved by running it. That is what the rc1 rehearsal is for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An expert panel reviewing the release before the v0.2.0 tag found that the post-push chain I rewrote in #672 was only half fixed, and that three structural faults around it would turn any post-push failure into a release that is public but unrecorded. The tag is held on this PR.
1. My verifier died on the first 404 — the same lag that failed v0.1.2
#672 fixed the byte comparison and kept the download as a bare
urlretrieve— no timeout, no retry, noHTTPErrorhandling. nuget.org indexes packages one at a time; the poll before this step waited for 4 ids; the other ~18 could still 404. Result: an uncaught traceback after every package was already public. My own header comment named the lag.Now: retries retryable statuses and transport errors with a timeout, records a named failure instead of raising, continues so every failure is reported together, and fails fast on a 403.
tests/scripts/nuget-verify-retry.test.shdrives the real script against a local server that 404s for a while then serves. With the originalurlretrieverestored it reproduces exactly the predicted failure:Getting it to run surfaced the friction it's about: on Windows, python's
subprocess.run(["bash", ...])resolves toSystem32\bash.exe— the WSL launcher — so the server received nothing and two assertions "passed" having measured nothing.2. The visibility poll waited for 4 of ~22 packages
It now defaults to every id in the release manifest; an explicit operator list still overrides.
3. GHCR and NuGet published in parallel — two irreversible acts, no edge
imagesnow needsnugetto have succeeded. Deliberate order: GHCR tags can be re-pointed or deleted, nuget.org packages can only be unlisted — so the irreversible side goes first.4. A post-push failure skipped the only durable copy of what shipped
github-releasewasif: success(), so on any post-push failure the nupkgs and manifest survived only as a 90-day artifact — for permanent packages. That's why the v0.1.2 release page was assembled by hand. A step now records that publication began (before the first push, since the loops run under-e);github-releaseruns underalways()gated on it. It's a draft, so nothing publishes, and it now opens with a warning when NuGet didn't succeed.5. The rehearsal existed; the runbook never said so
I told the maintainer no rehearsal was possible. Wrong:
0.2.0-rc1passes both guards (checked against master) and runs the whole post-push chain against the real feed, for one permanent prerelease no0.2.0pin resolves to. Now step 8 of "Before you tag".What this does not prove
Items 3 and 4 change the release job graph, which can't be run locally. They're checked structurally — both workflows parse, the
needs/ifare as intended, the marker precedes both push paths — but a workflow graph is only proved by running it. That is what the rc1 rehearsal is for.Gates:
repo-gates: all 14 gate(s) passed(the new test was auto-discovered), interpolation linter clean.🤖 Generated with Claude Code