Skip to content

The nightly release publishes only after the Darling PostgreSQL tests pass - #4448

Merged
erikdarlingdata merged 3 commits into
devfrom
ci/n473-nightly-publish-waits-for-pg-tests
Sep 26, 2026
Merged

erikdarlingdata merged 3 commits into
devfrom
ci/n473-nightly-publish-waits-for-pg-tests

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Refs #4336. The 2026-09-26 nightly published its release although its Darling PostgreSQL tests failed.

Why

The 2026-09-26 nightly published a release while the Darling PostgreSQL tests job (darling-pg) failed. darling-pg and the windows build job were siblings under check, so a red darling-pg never blocked build's gh release create or linux's gh release upload/ghcr push — the release went out regardless of test result.

What changes

.github/workflows/nightly.yml only.

Before:

  • check → build (needs: check) publishes the windows artifacts, packs the Velopack Setup.exe, deletes and recreates the nightly GitHub release, and uploads assets into it.
  • darling-pg (needs: check) runs the gated live-PostgreSQL tests as a sibling of build — its result has no effect on the release.
  • linux (needs: [check, build]) publishes the linux tarball, uploads it into the nightly release, and builds + pushes the ghcr container image, signs it, and attests it.

After:

  • build and linux keep building everything they did before, but no longer talk to GitHub Releases or ghcr. build uploads the windows zips/Setup.exe/checksum file and the derived version string as workflow artifacts. linux builds the linux tarball/checksum and the container image (via docker save, not docker push) and uploads those as workflow artifacts too.
  • A new publish job (needs: [build, linux, darling-pg]) runs only when needs.build.result == 'success' && needs.linux.result == 'success' && needs.darling-pg.result == 'success', downloads those artifacts, and does exactly what build/linux used to do: delete + recreate the nightly release, upload the windows and linux assets into it, push the container image to ghcr, sign it with cosign, and attest the linux tarball. contents: write, packages: write, id-token: write and attestations: write moved from the workflow-level permissions: block (now contents: read) to publish's own block — no other job needs write access.
  • A new workflow_dispatch boolean input, publish (default true), lets a manual re-run build and test without shipping a release; publish's if: also requires inputs.publish != false on a workflow_dispatch run.
  • darling-pg's existing failure artifact (darling-pg-failure, if: failure()) now also captures the gated upgrade tests' own temp-cluster server logs — those tests call Directory.CreateTempSubdirectory("darling-pg17boot-"), "darling-managedconf-", etc. under the Windows per-user temp directory (%TEMP%), which is a different directory from the runner's own temp directory (RUNNER_TEMP / ${{ runner.temp }}) that the upload step can reach. A new step, Collect the gated tests' own cluster logs, runs before the upload (if: failure()) and copies every logfile, *.log, and any file under a log\ or pg_log\ directory beneath %TEMP%\darling-* into RUNNER_TEMP\darling-cluster-logs\, preserving relative paths, skipping files over 50 MB, and printing a one-line count; the upload step's path list now points at ${{ runner.temp }}/darling-cluster-logs/** instead of the old ${{ runner.temp }}/darling-*/** globs, still with if-no-files-found: ignore.

No code changed — workflow-only, and every literal string, project path and command the jobs run is unchanged from before, just relocated.

Test plan

  • python3 -c "import yaml; yaml.safe_load(open('.github/workflows/nightly.yml'))" — parses clean.
  • actionlint .github/workflows/nightly.yml — no new findings (two pre-existing shellcheck info-level hints in the untouched check job's "new commits" step, unrelated to this change).
  • Manual job-graph check: build → check, darling-pg → check, linux → [check, build], publish → [build, linux, darling-pg] — matches the intended graph.
  • git grep -n "nightly.yml" -- '*.cs' '*.py' found the tests that parse this file's text: NightlyVersionInjectionTests, CiClusterWorkerSizingTests, CiClusterWorkerSizingLiveTests, LockedModeRestoreCoverageTests, ProductVersionDeclarationTests, ReadmeDerivedCountPinTests. None of them assert on job names, needs:, or step location — they pin the dotnet publish/docker build command lines, the conf-append blocks, and locked-mode restore coverage, none of which this change touches. Ran on macOS (in-process xUnit v3 runner, WPF framework entry stripped from the runtimeconfig):
    • NightlyVersionInjectionTests: 3/3 passed (still finds exactly one docker build line — in the new publish job — and every nightly publish still carries -p:Version=${{ steps.version.outputs.VERSION }}).
    • CiClusterWorkerSizingTests, LockedModeRestoreCoverageTests, ProductVersionDeclarationTests, ReadmeDerivedCountPinTests: 60/60 passed together.
  • dotnet build Darling/Darling.Tests/Darling.Tests.csproj -p:EnableWindowsTargeting=true and dotnet build Lite.Tests/Lite.Tests.csproj -p:EnableWindowsTargeting=true: both 0 errors.
  • No CrossAppGuardCiGateTests exists in this tree pinning nightly.yml's structure — a git grep for it found nothing; the six tests above are the actual set that reads this file.
  • The gated live-PostgreSQL suite, the real publish gating decision, and the new failure-artifact globs cannot run on macOS (windows-only runner, real PostgreSQL runtimes). The next dispatched nightly proves the gate: a darling-pg failure should leave publish skipped and the nightly release untouched, and a green run should publish exactly as before.

Follow-up fixes (same PR, same file)

  • The release now names the commit that was actually built, not dev's tip at publish time. build writes git rev-parse HEAD (the full SHA) to its own handoff file alongside the version string, uploaded in the same windows-version artifact. publish reads that file, uses the short form for the release notes' Commit line, and passes the full SHA to gh release create ... --target <sha> — replacing --target dev and git rev-parse --short HEAD, both of which named whatever dev's tip was 20–60 minutes later, after darling-pg finished. publish echoes the SHA it's using before creating the release.
  • publish now has its own concurrency group (nightly-publish, cancel-in-progress: false), so two close dispatches can no longer race on the same delete-then-create of the nightly release.
  • A cron run always publishes. inputs.publish reads as an empty string on a scheduled dispatch, not false, so the gate is now format('{0}', inputs.publish) != 'false' instead of a bare != false comparison; the three needs.X.result == 'success' terms are unchanged.
  • Artifact hygiene: the four handoff artifact uploads (windows-release-assets, windows-version, linux-release-assets, darling-container-image) now set retention-days: 3 — these run nightly and cost about 450 MB/run with no reason to keep them past the next publish. darling-pg-failure's upload sets overwrite: true, so re-running a failed darling-pg job doesn't collide with the first attempt's artifact name.
  • linux no longer waits on build. It reads no output of build's beyond the check job's shared stamp, and downloads no windows artifact, so needs: [check, build] became needs: check and the two jobs now run in parallel. The stale comments above linux and above darling-pg's checkout step — both still describing the old shape where build itself published the release and linux uploaded into it — are corrected to describe the current publish job instead.
  • Windows/Linux version equality is now checked and enforced. linux writes its own derived VERSION into releases/linux-version.txt, included in the linux-release-assets artifact. publish reads it, compares it against the windows version it already reads from windows-version.txt, and exit 1 with both values printed on a mismatch, before either the windows or linux assets are shipped.

CHANGELOG entry

SECTION: None
ENTRY: None: CI-only change (nightly workflow job graph), no user-visible effect.

… pass

Moves every release-publishing step (the GitHub release, the linux
tarball upload, the ghcr container push, cosign signing, the build
provenance attestation) out of the build and linux jobs and into one
new publish job that needs build, linux and the Darling PostgreSQL
tests job (darling-pg) and runs only when all three succeed. build
and linux keep building and now upload their outputs as workflow
artifacts; publish downloads them and publishes exactly what those
jobs built. Adds a workflow_dispatch publish boolean input (default
true) so a diagnostic re-run can skip publishing. darling-pg's
failure artifact now also collects the gated upgrade tests' own
temp-cluster server logs, not just the fixed cluster's log.

Refs #4336
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 20:46
@erikdarlingdata
erikdarlingdata merged commit 1b67f3a into dev Sep 26, 2026
17 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the ci/n473-nightly-publish-waits-for-pg-tests branch September 26, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant