The store's TimescaleDB moves to 2.30.1 before the store opens, closing GHSA-hcfx-29v5-2rcw, with nothing ever reverted and its own alert (#3908)

[erikdarlingdata]

Closes #3908. Closes #3951: CompleteAfterStartAsync now returns any outcome that is not a successful major upgrade untouched, so a Failed outcome can no longer become ExtensionUpdated and lose its alert (that status is gone).

Problem

Every release since 3.2 shipped TimescaleDB 2.28.1, and GHSA-hcfx-29v5-2rcw covers it. Bumping the bundle was blocked because nothing could move an existing store off 2.28.1 safely:

• The post-start update ran on a live store. The same-major ALTER EXTENSION timescaledb UPDATE in CompleteAfterStartAsync ran after the web host, MCP host, Viewer seats and the job scheduler had all connected. Measured: once the extension moved, every session that had loaded the old library failed its next statement, parallel queries failed, and the update's pre-step killed running jobs, which the scheduler then held in crash backoff for up to an hour.
• The 17-to-18 bridge could not reach a newer version. It probed and updated on one session, so the probe loaded the old library and TimescaleDB refused the update (0A000). A fresh session would still fail, because the old runtime has no update path to 2.30.1 (22023).
• Rolling back after a move bricks the store. Releases 3.3 to 3.8 swap in their own runtime whenever the stamp disagrees, and that runtime cannot load a 2.30.1 store.

What changed

• The bundle moves to TimescaleDB 2.30.1 and carries 2.28.1's two versioned libraries. fetch-pg-runtime.ps1 adds $tsCarried with pinned SHA256s and sanity-checks the result. A version counts as carried only when both its DLLs are present, because the TSL one holds compression and continuous aggregates.

• The quiesced update runs before the store opens. It is DarlingStoreUpgrade.UpdateTimescaleQuiescedAsync, called from EnsureRunningAsync after the conf append and before the network plan and the start. For each database that allows connections, it:

  • starts the cluster on a private loopback port with timescaledb.max_background_workers=0 and autovacuum=off;
  • reads the version on a session that loads nothing (timescaledb.disable_load=on);
  • runs the ALTER as the first statement of a fresh session with Npgsql type loading off;
  • proves the new library loads on an ordinary session.

  The cluster is then stopped, and the stop is confirmed (fast, then immediate, then pg_ctl status must answer 3). Only the connection is ever retried; the ALTER is sent once.

• Nothing is ever reverted. Every store this service has shipped is on 2.28.1, which the runtime carries, so a failed update leaves the store opening on its own version. That removes the reviewer's blocking concerns about reverting past a commit point.

• The gate is a record in the data directory. darling-timescaledb.version holds either the version as last read after start, or pending <from> <to> written before the ALTER. The update runs when the record is missing, pending, or not the runtime's version. There is deliberately no "absent" record: the first start of a new store reads it seconds before the worker creates the extension, and that stale record skipped the update after a runtime change. The gated tests caught it.

• The post-start ALTER is retired. CompleteAfterStartAsync is verify-and-analyze only. The new VerifyTimescaleAfterStartAsync reads the version, records it, and settles the outcome: Updated, Failed, Behind, or none.

• A separate alert. StoreTimescaleReport / EvaluateStoreTimescaleAsync fires Critical under the same Store Runtime Upgrade metric with its own key (storetimescale). A start that upgrades the major and then cannot move the extension therefore raises both alerts. An update that landed stays log-only, like the old extension-only outcome, and the ExtensionUpdated status is gone. The major-upgrade alert's TimescaleDB versions now say where the extension ended up, and leave it out when it did not move, so the two alerts cannot disagree.

• The pg_upgrade bridge skips what the new runtime carries. PlanTimescaleBridge makes that decision, so every shipped PostgreSQL 17 store is restored at 2.28.1. The quiesced update moves it after the directory swap commits, so a failure can no longer cost the rollback copy; that is the reviewer's link-mode finding. For a version the new runtime does not carry, the bridge targets the newest version both runtimes carry, ALTER-first.

• Rollback guards:

  • PinLegacyRuntimeStamp keeps pg-runtime.stamp at the 3.3-3.8 package hash, so a rollback to those releases keeps this runtime. This release's own stamp is now pg-runtime.sha256.
  • PackageCannotLoadStore refuses to swap in a package with no libraries for the recorded version. That is a rollback past this release.

• Orphan recognition. A server the update left on its private port is stopped before the runtime advance and before the normal start. It is identified by darling-timescaledb-update.port matching the port in postmaster.pid; any other running server is adopted as before.

• StartClusterAsync takes a wait and quotes the server log tail on failure.

• check-runtime-currency.ps1 drops the GHSA-hcfx-29v5-2rcw acknowledgement. The check now reports 2.30.1 clear of all four published advisories, and demanded the deletion.

Tests

• Real servers, run locally against the assembled runtimes:
  • Same-major 18.4 + 2.28.1 to 18.6 + 2.30.1, from the field's legacy-stamp state and from no stamp at all (that run deletes both stamps and asserts the no-stamp branch's log line): every row intact, outcome Updated, record written, no marker left.
  • A package missing its 2.28.1--2.30.1 script:
    • the store opens on 2.28.1 through the carried library, on the NEW binaries;
    • nothing is blocked and nothing reverted;
    • the outcome is Failed and the alert report is built;
    • the next start stops a server left on the private port with its marker.
  • The 17.10 to 18.6 pg_upgrade: bridge skipped, restored at 2.28.1, moved to 2.30.1 after the commit, both reports agree.
  • The downgrade refusal.
• All runtime-gated classes: 285 tests, 0 failed (DarlingManagedPostgresTests, StoreStatementStatsLive, CiClusterWorkerSizingLive, DarlingLiveStoreExtensionParity, DarlingPgRuntimeVersionPin).
• The full non-live suite: 12,564 tests, 0 failed.
• Unit tests: the gate, the record, the version parser, the bridge plan, TryReadPostmasterPort, the orphan rule without a server, the rollback guard both ways, the report mapping, the alert text, and a wiring-order pin.
• Mutation red-watch, 6 of 6 caught: the rollback guard off, the TSL requirement off, the orphan port check off, the gate always running, a loose version parse, and a bridge plan ignoring the old runtime. The loose parse was missed first; I added rows that only the digit check rejects.
• The pg_upgrade test ran on ports 56432/56433 locally, because a Docker container holds 55432. That was a temporary local edit, never committed. The nightly runs it on the real ports.

Adversarial review round (after the first push)

• BLOCKING, CI's PG shard 1 was red: CaptureDownChunkOrderTests assumed ChunkAppend's plan. TimescaleDB 2.30 plans the same read as DeferredChunkAppend by default, which lists only the chunks it visited ("Chunks Visited: 1"). The pin now accepts either shape: one visited chunk per arm, or eight or more scans with at most one executed per arm. Verified on throwaway 18.6 + 2.30.1 and 18.4 + 2.28.1 clusters.
• MAJOR: the "unstamped" same-major run was stamped by the legacy pin, so it never exercised the no-stamp branch. Fixed as described under Tests.
• Smaller fixes, each tested or pinned:
  • A bootstrap retry keeps the outcome of the attempt that ran the update, so a Failed report can no longer turn into a false Behind.
  • An absent extension is recorded under the runtime version it was read under, so a store without the extension no longer pays a start and stop on every start, and a fresh install's second start pays none.
  • A first extraction (a clean reinstall) refuses a package that cannot open the recorded TimescaleDB, sharing MissingTimescaleLibraries with the swap guard. The README says to keep pg-runtime when rolling back.
  • The quiesced step now:
    • runs only when the data directory's major matches the runtime's, which covers a A link-mode store upgrade that fails before its commit point leaves the store unable to start: pg_control.old is never restored, and RevertRuntime moves runtimes under a live postmaster #3927 revert that could not run;
    • keeps FindFreeLoopbackPort inside its never-throws contract;
    • logs a failed start, and says when a reported failure had in fact landed or had nothing to move;
    • quotes the newest server log.
  • Wording: the non-store database message and the Behind alert text.
  • The pg_upgrade old-cluster start runs quiesced, and the bridge targets only the old runtime's own default version, the one it has update scripts for.

Guard edits (the kind each guard asks to have named)

• StartupCommandTimeoutTests:
  • BridgeTimescaleAsync is now 1/0/0/0 and CompleteAfterStartAsync 2/0/0/0.
  • New members: UpdateTimescaleAsFirstStatementAsync 0/0/0/1, ReadExtversionAsync 1/0/0/0, ListConnectableDatabasesAsync 1/0/0/0.
  • The total is 32 after merging dev's 33.
• AlertReadFailureSurfaceTests: one exemption, the TimescaleDB self-alert's catch, whose evidence is a parameter. The evaluator's exempt count is now 13 and the tree total 27.

Related, found while building this

• A link-mode store upgrade that fails before its commit point leaves the store unable to start: pg_control.old is never restored, and RevertRuntime moves runtimes under a live postmaster #3927, link-mode pre-commit recovery: fixed separately by A store upgrade that fails before its commit point puts the old cluster back as it found it, and only claims a revert that happened (#3927) #3954, which is merged into this branch. All runtime-gated tests, including the PostgreSQL 17 pg_upgrade and A store upgrade that fails before its commit point puts the old cluster back as it found it, and only claims a revert that happened (#3927) #3954's revert tests, pass on the combined code.
• Every service restart raises a false Store Checkpointer Pressure warning: PostgreSQL counts the shutdown checkpoint as requested, and a monitored target's restart reads as a WAL-forced checkpoint the same way #3955, a false "Store Checkpointer Pressure" warning on every restart, found on DARLING01: fixed separately.

Field validation

DARLING01, a production-shaped store with 1,024 chunks, 26 continuous aggregates and 148 jobs: 18.4 + 2.28.1 moved to 18.6 + 2.30.1. The extension update took about 4 s on a private port, and collection resumed 31 s after the runtime swap began. Every count is identical, all TimescaleDB jobs succeeded on 2.30.1, and the stamps, record and marker are exactly as designed.

No CHANGELOG edit here; the entry goes through the batched splice.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SFctxdyLSdZbycGtvwXk8j