Repository navigation
The in-place upgrade test reports the upgraded server's own log and any restart when a post-upgrade read fails - #4445
Merged
Conversation
…ade read fails The gated in-place upgrade test measures the store through a pooled connection right after the bootstrap returns. When that pooled read fails, the failure carried only the shared rig's own server log and no way to tell whether the upgraded server had restarted underneath it. Record pg_postmaster_start_time() on a Pooling=false connection right after the bootstrap returns and again before diagnosing any later failure. If it moved, the message says so plainly. Either way, the message now includes the upgraded cluster's own server log tail (the newest of its pg.log and logging-collector ring files), with FATAL/PANIC/terminated lines surfaced first. Test-only: no product code changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4336. Diagnostics for a gated upgrade-path failure in the nightly.
Why
The gated in-place upgrade test builds a real store on the old PostgreSQL
major, runs the production bootstrap, and then measures the upgraded store
through a pooled connection right after the bootstrap returns. In a recent
nightly run that pooled read failed with a "forcibly closed" connection
exception. The failure artifact carried only the shared CI rig's own server
log, not the upgraded cluster's, so there was no way to tell whether the
upgraded server had actually restarted underneath the test, or something
else was going on.
What changes
Test-only, in
Darling/Darling.Tests/DarlingStoreUpgradeTests.cs:EnsureRunningAsync) returns, the test readspg_postmaster_start_time()on aPooling=falseconnection and recordsit. This mirrors the shape the product's own cluster-identity read
already uses, so the probe itself can't be the thing that dies.
any of the assertions that follow it — is now caught once and rethrown
with a diagnosis: the server's start time is read again, and if it moved,
the message says so directly ("the upgraded server restarted between the
bootstrap and this read"); if the second read itself fails, that's
reported too.
tail (the newest of its
pg.logand logging-collector ring files, thesame file the product picks for its own diagnostics), with any
FATAL/PANIC/terminated by exception/"was terminated"/"databasesystem is shut down" lines surfaced first, ahead of the raw tail.
No product code changed. The pooled read at the point of failure is left
exactly as it was — it's the product-shaped read this test exists to
exercise — only the failure handling around it is new.
Test plan
dotnet buildon bothDarling.TestsandLite.Tests: 0 errors oneach.
DocCommentHygieneTests: green (77 total, 0 failed).DarlingStoreUpgradeTestsclass run in-process on macOS: 146total, 6 failed, 13 skipped — identical counts and identical failing
test names before and after this change (those 6 fail on macOS today for
unrelated Windows-path reasons and are gated out of scope here; verified
by running the same class against the unmodified
devtip).UpgradeInPlace_OldMajorStoreWithRealData_…)can't run here: it needs real Windows PostgreSQL runtimes
(
DARLING_TEST_PGRUNTIME_OLD/_NEWZIP), so it's build-verified only andleft for the next nightly to prove — that run is the actual pin: either
the failure message now names a restart with the upgraded server's own
log, or the test goes green and the diagnostics were unused.
CHANGELOG entry
SECTION: None
ENTRY: None: test-only diagnostics for a gated CI test; no user-visible effect.