Skip to content

The scratch-database exit drain returns at its budget even with a drop in flight - #5547

Merged
erikdarlingdata merged 1 commit into
fix/nightly-foreground-threadfrom
fix/nightly-foreground-thread-bounded-drain
Oct 8, 2026
Merged

erikdarlingdata merged 1 commit into
fix/nightly-foreground-threadfrom
fix/nightly-foreground-thread-bounded-drain

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Refs #5480.

Stacked on fix/nightly-foreground-thread (c1eba1e). Another lane worked the same brief and pushed that branch at 07:51Z, with a nightly in flight (run 37746048228). This PR has one commit on top of it, so its base is that branch, not dev. It replaces that branch's exit-drain budget with one that holds even when a drop is already running. The seven-class leak fix and ScratchDatabaseDisposalCensusTests come from c1eba1e and are unchanged here.

Cause

  • Nightly 37737551739, job 113180523202 (dev 2186662): Total: 28763, Failed: 0, then the summary at 07:18:35.7Z, then Waiting 10 seconds for foreground threads to exit... at 07:18:36.71Z, then [FATAL ERROR] Foreground threads were left running at 07:18:46.72Z, and exit code 1.
  • The message comes from xunit.v3 4.0.1, xunit.v3.mtp-v2.dll (src/common/MicrosoftTestingPlatform/TestPlatformTestFramework.cs). After app.RunAsync() returns, it queues a thread-pool item that waits 1 s, prints "Waiting", waits shutdownForegroundThreadWaitSeconds (default 10), then calls Environment.Exit(1). It never checks for threads. Anything that keeps the process alive past 11 s trips it, and that includes a ProcessExit handler.
  • No thread was left running. The ScratchPostgres ProcessExit drain was still dropping leaked scratch databases, one at a time, each with the quiesce Four Darling live tests wait for the state they assert #5480 added. On Oct 8 that took more than 11 s (39 lines, about 0.36 s each). On Oct 7 (run 37581130191, passed) it took about 4.7 s.
  • Bisect: dev push runs are sharded (six processes), so each drain is short, and none of the 18 dev Build runs since Oct 7 06:00Z shows the line. Only the single-process nightly-workflow runs show it: 37737551739, 37723838729 and 37728953153 (all FATAL, 39 leaked). 37581130191 is clean (38 leaked, short drain).

What this commit changes (ScratchPostgres.cs, ScratchPostgresExitDrainTests.cs)

c1eba1e's drain stops starting drops after 6 s, but a drop it already started can still run its quiesce (up to 10 s) and its DROP DATABASE (up to 30 s). On a local rig, one DROP DATABASE waited 26 to 30 s for its immediate checkpoint (pg.log: checkpoint complete: wrote 8922 buffers ... sync=16.786 s, total=26.405 s). That drain can still overrun the 11 s window.

This commit:

  • runs clusters side by side, and up to 16 drops at once within a cluster, under one 6 s deadline (ExitDrainBudgetSeconds, same name and value);
  • returns at the deadline whatever is in flight. The deadline cancels connects and quiesces. A drop already sent is never cancelled, because an interrupted DROP DATABASE can leave an invalid database. A database the drain did not finish is named on stderr and counted in the new ExitDrainOutcome.Unfinished;
  • keeps the dead-cluster probe (the first database on a cluster goes alone, so a dead cluster costs one connect) and the quiesce before each drop (ScratchPostgresQuiesceCensusTests still passes);
  • lets QuiesceTimescaleJobsAsync take an optional caller token, linked with its 10 s cap.

Evidence

Check Base drain (dev) This drain
4 hanging clusters (TCP accepted, no answer, 3 s connect timeout) 12.0 s, RED (past the 11 s window) 3.01 s
1 hanging cluster, 3 databases, 1 s budget n/a (no budget) 1.04 s, Unfinished 3
Live rig, 6 to 10 leaked databases n/a returns by the budget, every database counted

Test plan

  • Darling.Tests builds with 0 warnings and 0 errors.
  • Local run, rig on 55504: ScratchPostgresExitDrainTests (9, live ones included), ScratchDatabaseDisposalCensusTests, ScratchPostgresQuiesceCensusTests, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, DocCommentHygiene* and RawGatePerSlotSourceFloorLiveTests: Total 124, Failed 0.
  • An earlier local run of my own version of the seven-class fix (the same change as c1eba1e) passed RawRepairEpochTriggerLiveTests, RawPurgeTriggerGateErrorTests and RawGatePerSlotSourceFloorLiveTests on the rig, with no "outlived" line naming them.
  • Nightly on this branch, run 37746531776 (publish=false, test_map=false; publish is skipped when publish=false): expect Failed: 0, no FATAL ERROR line, and 0 outlived its test lines (only the deliberate 20010101 drain-test names).
  • The other lane's nightly on c1eba1e, run 37746048228, for comparison.

For the coordinator

  • Two lanes worked this brief. Pick one: merge this PR into fix/nightly-foreground-thread and send that branch to dev, or take c1eba1e alone and accept the unbounded in-flight drop.
  • One test was rewritten: c1eba1e's TheDrain_OutOfBudget_StartsNoDrop... now expects Unfinished == 3 and Skipped == 0. Skipped now means only "its cluster did not answer".
  • The mixed dead/live live test gets a long budget, because it pins the mix, not the deadline.

No changelog entry (test infrastructure).

…in flight

The drain stopped starting drops after 6 seconds, but a drop already started
could still run its quiesce (up to 10 s) and its DROP DATABASE (up to 30 s),
and one DROP DATABASE waited 26 seconds for its immediate checkpoint on a
local rig, so the exit could still pass xUnit's 11-second window.

The drain now runs clusters side by side and up to 16 drops at once under one
deadline, returns when it passes whatever is in flight, cancels connects and
quiesces at the deadline (never a sent drop), and counts what it did not
finish as Unfinished. The one-connect probe of a dead cluster and the quiesce
before each drop are kept.

Tests: four hanging clusters drain in 3.0 s (12.0 s on the old drain, past the
window); a 1-second budget returns in 1.04 s with all three unfinished; a live
leak of six databases returns by the budget and accounts for each.

Refs #5480
@erikdarlingdata
erikdarlingdata merged commit c29d8b8 into fix/nightly-foreground-thread Oct 8, 2026
9 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/nightly-foreground-thread-bounded-drain branch October 8, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant