Skip to content

The in-place upgrade test reports the upgraded server's own log and any restart when a post-upgrade read fails - #4445

Merged
erikdarlingdata merged 1 commit into
devfrom
test/n473-upgrade-in-place-diagnostics
Sep 26, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
test/n473-upgrade-in-place-diagnostics

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Refs #4336. Diagnostics for a gated upgrade-path failure in the nightly.

Why

The gated in-place upgrade test builds a real store on the old PostgreSQL
major, runs the production bootstrap, and then measures the upgraded store
through a pooled connection right after the bootstrap returns. In a recent
nightly run that pooled read failed with a "forcibly closed" connection
exception. The failure artifact carried only the shared CI rig's own server
log, not the upgraded cluster's, so there was no way to tell whether the
upgraded server had actually restarted underneath the test, or something
else was going on.

What changes

Test-only, in Darling/Darling.Tests/DarlingStoreUpgradeTests.cs:

  • Right after the bootstrap (EnsureRunningAsync) returns, the test reads
    pg_postmaster_start_time() on a Pooling=false connection and records
    it. This mirrors the shape the product's own cluster-identity read
    already uses, so the probe itself can't be the thing that dies.
  • Any exception raised after the bootstrap returns — the pooled read, or
    any of the assertions that follow it — is now caught once and rethrown
    with a diagnosis: the server's start time is read again, and if it moved,
    the message says so directly ("the upgraded server restarted between the
    bootstrap and this read"); if the second read itself fails, that's
    reported too.
  • The same diagnosis always appends the upgraded cluster's OWN server log
    tail (the newest of its pg.log and logging-collector ring files, the
    same file the product picks for its own diagnostics), with any
    FATAL/PANIC/terminated by exception/"was terminated"/"database
    system is shut down" lines surfaced first, ahead of the raw tail.

No product code changed. The pooled read at the point of failure is left
exactly as it was — it's the product-shaped read this test exists to
exercise — only the failure handling around it is new.

Test plan

  • dotnet build on both Darling.Tests and Lite.Tests: 0 errors on
    each.
  • DocCommentHygieneTests: green (77 total, 0 failed).
  • The full DarlingStoreUpgradeTests class run in-process on macOS: 146
    total, 6 failed, 13 skipped — identical counts and identical failing
    test names before and after this change (those 6 fail on macOS today for
    unrelated Windows-path reasons and are gated out of scope here; verified
    by running the same class against the unmodified dev tip).
  • The gated test itself (UpgradeInPlace_OldMajorStoreWithRealData_…)
    can't run here: it needs real Windows PostgreSQL runtimes
    (DARLING_TEST_PGRUNTIME_OLD/_NEWZIP), so it's build-verified only and
    left for the next nightly to prove — that run is the actual pin: either
    the failure message now names a restart with the upgraded server's own
    log, or the test goes green and the diagnostics were unused.

CHANGELOG entry

SECTION: None
ENTRY: None: test-only diagnostics for a gated CI test; no user-visible effect.

…ade read fails

The gated in-place upgrade test measures the store through a pooled
connection right after the bootstrap returns. When that pooled read
fails, the failure carried only the shared rig's own server log and no
way to tell whether the upgraded server had restarted underneath it.

Record pg_postmaster_start_time() on a Pooling=false connection right
after the bootstrap returns and again before diagnosing any later
failure. If it moved, the message says so plainly. Either way, the
message now includes the upgraded cluster's own server log tail (the
newest of its pg.log and logging-collector ring files), with
FATAL/PANIC/terminated lines surfaced first.

Test-only: no product code changed.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 20:01
@erikdarlingdata
erikdarlingdata merged commit 5e4acd8 into dev Sep 26, 2026
17 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the test/n473-upgrade-in-place-diagnostics branch September 26, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant