The startup sweeps stop a continuous aggregate's jobs before they drop it

[erikdarlingdata]

Fixes #5551.

Why

Three startup sweeps drop a continuous aggregate with DROP MATERIALIZED VIEW ... CASCADE without stopping its jobs or waiting for a refresh that is already running. A refresh running during the drop gave XX000: tuple concurrently deleted (#5416, already caught and recovered). It is the inferred cause of a PostgreSQL backend crash in CI (#5549).

What changes

Before each drop of a continuous aggregate, QuiesceContinuousAggregateJobsAsync (in TimescaleSupport.cs):

1. Finds the aggregate's jobs (refresh, compression, retention) in timescaledb_information.jobs, joined to continuous_aggregates on either identity a job can report (the user view or the materialization hypertable, the join ContinuousAggregateRefreshStateSql already uses). The view name is a bound parameter. It also follows aggregates built on the one being dropped (the daily tier), because the CASCADE drops those and a refresh of the daily tier is the same race. Calls are unqualified alter_job, like the rest of the product code.
2. Stops only the jobs that are scheduled now (alter_job(job_id, scheduled => false), one statement) and remembers which.
3. Waits until two looks in a row at pg_stat_activity (this database, backend_type ending in [<job_id>]) show no worker of any of the aggregate's jobs, stopped or not.
4. At the cap, if the FINAL look still lists a worker: schedules the stopped jobs again, logs a warning, and skips this drop. The next pass retries. If the final look is empty, the drop goes ahead.
5. If the drop fails: the jobs are scheduled again, best effort and logged, in the existing per-relation catch, after ReopenBrokenConnectionAsync. A cancellation (shutdown, or the hourly pass's budget) schedules them again too. Every resume runs on its own 10 second grace token, never the caller's.
6. A plain-PostgreSQL store (no timescaledb_information), a plain view and an absent relation have no jobs, so this is a no-op, as before.

The sweep itself cancels and terminates no backend, but stopping a job makes TimescaleDB's scheduler end that job's running worker (see Review round 1, finding 7).

A job that was already stopped before the sweep is never in the remembered set, so it stays stopped.

Sites (every DROP MATERIALIZED VIEW in product code)

git grep -n "DROP MATERIALIZED VIEW" -- Darling/ finds two product statements, both in TimescaleSupport.cs, covering three call sites. All three now quiesce:

• DropRetiredBaselineAggregatesAsync, superseded pass (Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653), through DropRetiredBaselineRelationSql.
• DropRetiredBaselineAggregatesAsync, retired pass (Retire the orphaned cpu_utilization_baseline / file_io_baseline continuous aggregates (#1995 cleanup) #2007), through DropRetiredBaselineRelationSql.
• DropStaleContinuousAggregatesAsync, the composer-dimension reshape (four views, CASCADE takes the daily tier).

The rest of the hits are test code or comments. A census test (below) fails if a new product drop appears.

The cap: 30 seconds, polled every 250 ms

Both sweeps run on the start path (serially, before the collectors start) and again on the hourly store-maintenance pass, whose whole budget for every step is 5 minutes. A drop happens once per store. Because the jobs are stopped first, only a refresh already running is waited for, so the wait is a few seconds for these small or day-old aggregates: TimescaleDB's scheduler usually ends the worker itself within about 3-4 s of the stop (measured), and a worker that finishes first is just as good. Worst case, every one of the eight relations (two superseded, two retired, four reshaped) is stuck behind a worker: 8 x 30 s = 4 minutes, still inside the hourly 5. A longer cap would let a stuck worker stall startup or eat the hourly budget; a refresh that really takes longer costs nothing to wait out, because the aggregate is skipped and retried, not dropped under a worker. A pin test holds the number and the 8 x cap <= 5 minutes arithmetic.

Small defects fixed in code I touched

• DropStaleContinuousAggregatesAsync had no connection reopen after a failed drop. An XX000 closes the Npgsql connection (Flaky test: the retired baseline sweep test finds its connection closed after the first attempt #5416), so the next view's stale check threw "Connection is not open" and logged a second, misleading failure. It reopens now, like the retired-baseline sweep.

Both sweeps gained an overload that takes AggregateJobQuiesceOptions before the token (the production callers and signatures are unchanged); the tests use it to reach the cap in 2 seconds.

Review round 1

• Finding 1 (the RetiredBaselineSweepRecoveryTests sweeper's search path): fixed in Darling tests: a scratch database drop no longer hands its backend to the next test, and the retired-baseline race no longer runs a refresh beside the drop #5550, which owns that test.
• Finding 2, test (b) passed with the wait deleted: it now watches the sweeper's own backend in pg_stat_activity for the DROP while the same worker pid is alive on both sides of each look, requires at least one look that saw the hold, and skips (with the reason) when the scheduler ended the worker first. RED-checked by deleting the wait loop: (b) fails.
• Finding 3, test (c) could flake: it uses a zero cap, so the skip and the resume follow the first look within milliseconds.
• Finding 4, paths that leave jobs stopped: the unschedule statement is now inside the try, so a stop that committed and then lost its connection is undone (reopen, then resume). Every resume (after the cap, in the three callers' catch blocks, after a cancellation, and when the quiesce itself throws) goes through one method that runs on its own 10 second grace token, the bound the cancellation resume already had. The old cancellation-only variant is gone. A resume that cannot happen logs the job ids and the view.
• Finding 5, the cap path: the drop is skipped only if the FINAL look still lists a worker. An empty final look lets the drop go ahead. The warning now says what happened (a worker was still listed at the last look, after N s), not that a worker "was still running" at the cap.
• Finding 6, reopen failure: when the caller's reopen fails, or the resume's reopen fails, the warning names the stopped job ids and the view and gives the alter_job statement. The three catches share one helper for the reopen and the resume.
• Finding 7, "no backend is cancelled or terminated" was wrong, and every comment, doc and log text that said so is corrected (the quiesce doc, the cap doc, the cap warning, the PR body, the changelog, test comments). Measured on TimescaleDB 2.30.1 with a worker held at an advisory lock: after alter_job(scheduled => false) the scheduler ended the worker about 3-4 s later; the server log shows job N threw an error and an ERROR (invalid transaction termination) from the worker, and the job's stats went to total_failures + 1 with last_run_status = Failed (job_status Paused while stopped).
  • Does the Background-job self-heal covers compression jobs only - a dead CAGG refresh or retention job is never detected or re-armed #3816 job-failure alert fire on it? After a successful drop: no. The job, its timescaledb_information.job_stats row and the raw bgw_job_stat row are all gone with the aggregate (measured: 0 rows after the DROP), so the arm never sees it. After a skip or a failed drop: the job stays with one more failure and Failed as its last status until its next run succeeds, which is exactly the arm's condition (failures grew since the previous hourly sample AND last run Failed), so it can fire once at INFORMATION ("N new failure(s)", cooldown applies) if its hourly sample lands before the job's next successful run. It needs a worker actually running at the moment of the stop AND a kept aggregate, so it is rare. I did not change the arm: telling this failure from a real one would need the arm to know about the sweep, which is a design decision, not a small fix. I documented it on the arm and on the quiesce. Options for the seat: leave it (INFORMATION, rare, truthful: a run did fail), or have the resume path run the job once right after resuming so its last status becomes Success.
• Also fixed: the old resume warning said the jobs stay stopped "until ... the next start", which is not true (a later pass leaves a job it finds stopped alone); the text is corrected.

Review round 2

• Finding 1 (Medium, from round 1's own change): the grace-token reopen could throw out of the sweeps' catch. After a failed drop that holds stopped jobs, ReopenAndResumeAfterFailedDropAsync reopens the connection on the 10 s resume grace. ReopenBrokenConnectionAsync does not catch OperationCanceledException, so a reopen that took longer than the grace (a pool wait, or a host that stopped answering; Npgsql's connect timeout is 15 s) threw it past the per-relation catch while the caller's token was still live. The jobs stayed stopped with no log line, the hourly pass skipped every later convergence step, and at startup the exception reached the host and stopped the service.

• Fix: the grace reopen is now inside a try that catches OperationCanceledException. It logs the reopen-failed template plus the stopped job ids and the view ("the connection did not reopen within 10 s"), and returns false, which ends the sweep like any failed reopen. It rethrows only when the caller's own token is cancelled, after logging. The reschedule that follows cannot throw past the caller: ResumeContinuousAggregateJobsAsync catches everything, including its own grace running out. The helper is now internal with an optional resumeGrace argument as the test seam (production leaves it unset).

• Audit of every other use of the grace token and every exit of the quiesce and resume paths. Only this one escaped.

  • s_resumeGrace is used in two places: the resume (reopen and reschedule both inside a catch (Exception), logged with the job ids) and this reopen (now fixed).
  • QuiesceContinuousAggregateJobsAsync: its only grace use is the resume in its catch-all and in the cap branch, which cannot throw, so the original exception is what the caller sees.
  • The three callers' catch (OperationCanceledException) when (hold.Unscheduled.Count > 0) blocks resume on the grace (cannot throw) and rethrow the caller's own cancellation. A cancellation from the quiesce comes only from the caller's token (the grace is never passed to it), and the hold is empty then, so those blocks are not entered a second time.
  • The three callers' reopen-and-resume goes through the fixed helper; the empty-hold branch still reopens on the caller's token, where a cancellation is the caller's own.

• Tests: new AggregateJobGraceReopenTests (no database; a never-opened connection and a zero grace, so the reopen's token is cancelled before it starts). One test pins that a grace-expired reopen returns false and logs the job ids, the view and the alter_job statement, with one warning. The other pins that with the caller's token also cancelled it logs the jobs and then throws. A source pin was added to the quiesce pin class (the reopen sits in that try/catch with the caller-token rethrow, and there are exactly two grace tokens).

• RED-check: with the catch disabled (when (false)), both new tests FAIL (the OperationCanceledException escapes, or the log line is missing). With the fix, both pass.

• Also fixed (a CI failure on this PR): EveryProductDropOfAContinuousAggregate_IsPrecededByTheQuiesce filtered comment lines with StartsWith("//"), which CommentFilterAdoptionTests.EveryLinePrefixCommentFilter_StatesTheBoundItRestsOn flags as a hand-rolled comment filter (the Guard tests job was red). It now uses an anchored regex for a whole-line comment.

• Run for this round (rig PostgreSQL 18.6 + TimescaleDB 2.30.1 on port 55512, UTC, CI's worker settings; stopped by PID afterwards): build 0 warnings; AggregateJobQuiesce* (pins and 8 live) plus AggregateJobGraceReopenTests, RetiredBaselineAggregate* and StoreObjectConvergence*: 37/37; every Stage=Guard test: 1401/1401 (this includes the comment-filter census); *Census*, *Ratchet*, *Hygiene*, *Pin*, *Guard*, *StartupCommandTimeout*, *StorageCommandTimeout*: 1531 run, 0 failed, 2 skipped (pre-existing skips). Full suite not run locally.

• Also fixed (a second CI failure on this PR, Darling PG shard 5): TimescaleEnableConnectionSafetyTests.NoLiveTest_KeepsUsingAConnectionItHandedToTryEnableAsync flagged AggregateJobQuiesceTests.cs:676: Assert.SkipWhen(!await TimescaleSupport.TryEnableAsync(connection, null, ct), ...). The census accepts only an inline Assert.True(await ...), a result captured and checked in the very next statement, or a route through LiveTimescaleProbe.TryEnableAsync (LivePostgresStoreFixture keeps using the connection after CREATE EXTENSION timescaledb kills the backend, masking the real cause #1922: CREATE EXTENSION can kill the backend it runs on). That was the file's only TryEnableAsync call. OpenStoreAsync now migrates the store, then enables TimescaleDB through LiveTimescaleProbe.TryEnableAsync on the probe's own connection and skips when it is unavailable, so the skip-without-TimescaleDB behaviour is kept. Small defect fixed in the same method: a skip (or a failed setup) used to leak the connection it had opened, because the caller never received it to dispose; it is disposed on that path now. The census is untouched. Earlier lanes ran only the Stage=Guard classes and missed this one; this time I ran every source-scanning class (any class in a file that uses EnumerateFiles, GetFiles(...*.cs) or File.ReadAllText) plus every class named Census, Ratchet, Hygiene, Guard or Pin: 580 classes in 6 batches, 8863 tests, 0 failed except SensitiveStatementAutoParamTests.Measure_20MbPlan_WithACut_ThreeShapes_StayUnderTheCeilings, a timing-ceiling test that reads no source and passed alone (44 of 44) on re-run. TimescaleEnableConnectionSafetyTests is 2 of 2. Live: AggregateJobQuiescePinTests, AggregateJobGraceReopenTests and AggregateJobQuiesceLiveTests against a throwaway PostgreSQL (TimescaleDB preloaded, UTC, port 55513, stopped afterwards): 17 of 17, none skipped. Build 0 warnings. No full suite run locally; CI decides.

Test plan

Rig: PostgreSQL 18.6 + TimescaleDB 2.30.1 on port 55510, UTC, CI's worker settings.

• Build, 0 warnings.
• AggregateJobQuiescePinTests (7): SQL text of the job lookup, stop, start and worker count; the probe; the cap; a census that every product drop is preceded by the quiesce, that each catch reopens and resumes through one helper, and (new) that the stop is inside the try, no resume takes a caller token, and the cap decides on the final look.
• AggregateJobQuiesceLiveTests (8, own scratch store each), 15/15 with the pins, run 3 times in a row on a fresh rig (port 55510, UTC, CI's worker settings), all green with 0 skipped:
  • (a) the jobs are stopped before the drop (an event trigger records their state at the start of the DROP), and the drop still removes the aggregate and all three jobs;
  • (b) a real running job worker holds the drop until it exits. The refresh selects through an IMMUTABLE-labelled function that takes a shared advisory lock, the test holds the exclusive one, and the scheduler's worker sits in pg_stat_activity as Refresh Continuous Aggregate Policy [id]. The test asserts the jobs are stopped while it runs; and on every look, while that same pid is alive on both sides of the look, the sweeper's OWN backend never shows the DROP in pg_stat_activity.query (matched as DROP MATERIALIZED VIEW or the start of the DO $do$ block, because track_activity_query_size cuts the long text), the aggregate exists, and the sweep has not finished. It passes only if at least one look saw the worker alive and the sweeper inside its wait loop; if the scheduler ended the worker before that, the test SKIPS and says so. Then the drop goes ahead once the worker exits;
  • (c) at the cap (now a ZERO cap, so the stop-to-resume window is milliseconds and the scheduler cannot end the worker first) the jobs are scheduled again, the aggregate stays, the same backend pid is still running, one warning, and the next pass drops it;
  • (new) cancelled while waiting: the cancellation goes through, the aggregate stays, and every job it stopped is scheduled again (the resume ran on its own token);
  • (new) a zero cap with no worker running: the empty final look lets the drop go ahead, with no warning;
  • (d) a failed drop (planted XX000, which breaks the connection) schedules the jobs again after the reopen;
  • (e) a job stopped by hand before the sweep stays stopped (same test as d);
  • reshape: the daily tier's jobs are stopped with the hourly's, a failed drop schedules both tiers again, a clean drop removes both;
  • plain view, absent relation, and a store without TimescaleDB are no-ops, and the plain view still drops.
• RED-check (round 1), the wait loop deleted (stop kept, return right after it): (b) FAILS, (c) FAILS (the drop ran: expected 0 dropped, saw 1), and the cancellation test FAILS. (b) failed in one of two ways across two RED runs: "the sweeper reached its DROP while the job worker (pid N) was still alive" (the new assertion, caught on pg_stat_activity), and, when the DROP finished too fast to catch, "the aggregate's jobs were dropped before its worker exited". Wait loop restored, 15/15 green after.
• Earlier RED-checks (round 0): quiesce made a no-op fails 4 of 6 live tests; quiesce on with resume a no-op fails d, c and reshape.
• Also green after this round, in one run: RetiredBaselineAggregate*, StoreObjectConvergence*, and every non-live *Census*, *Ratchet*, *Hygiene*, *Guard*, *Pin* class (111 classes, 1311 tests, 0 failed, 2 skipped), plus LiveCleanupConversionRatchetTests, LivePostgresCollectionHygieneTests and LivePlanShapeIsolationCensusTests. The *LiveTests Guard/Pin classes were not run (no CPU load on the laptop; CI decides).
• Full suite not run locally (CI decides).

Things to double-check

• Seam note on (b): TimescaleDB's scheduler ends a worker whose job was unscheduled, measured at about 3-4 s after the stop. (b) looks for two seconds after the stop and now only passes when it observed the hold; if the scheduler wins that race on a very loaded cluster the test skips (and says why) rather than passing without proof.
• A process killed between the stop and the drop leaves the aggregate's jobs stopped. The next start retries the same drop (the verdicts are re-evaluated each pass), so the usual outcome is the aggregate going away anyway. If the next pass decides to keep the aggregate instead, it reads those jobs as stopped by hand and leaves them stopped (an earlier version of the resume warning said "or the next start", which was wrong, and is gone). Every log line that says jobs stayed stopped now names the ids and the view and gives SELECT alter_job(<job id>, scheduled => true).
• The existing RetiredBaselineSweepRecoveryLiveTests race test drives the job with CALL run_job, which runs in a client backend rather than a job worker, so the quiesce does not see it. That file belongs to Darling tests: a scratch database drop no longer hands its backend to the next test, and the retired-baseline race no longer runs a refresh beside the drop #5550 and is untouched; its assertions still pass.
• The job-worker look needs pg_stat_activity.backend_type for the job owner's backends, which the service login (the owner of the aggregates) can read.

Proposed changelog entry

Fixed:

• The startup sweeps stop a continuous aggregate's jobs before they drop it ([The startup sweeps stop a continuous aggregate's jobs before they drop it #5552]). When Darling retired or rebuilt a rollup on start, its background refresh could be running at that moment, which failed the drop with "tuple concurrently deleted" and could take the PostgreSQL backend down with it. Darling now stops the rollup's jobs first and waits up to 30 seconds for a refresh that is already running to exit. TimescaleDB ends such a refresh itself within a few seconds of the stop; it rolls back and counts as one failed run, which can show up once as an information-level job-failure alert if the rollup is kept. If a refresh is still running at the end of the wait, the rollup is left alone and the next pass tries again; if the drop fails, the jobs start again so the rollup keeps refreshing, and if they cannot be started the log names them and the view.