Repository navigation
The scratch-database exit drain returns at its budget even with a drop in flight - #5547
Merged
erikdarlingdata merged 1 commit intoOct 8, 2026
Conversation
…in flight The drain stopped starting drops after 6 seconds, but a drop already started could still run its quiesce (up to 10 s) and its DROP DATABASE (up to 30 s), and one DROP DATABASE waited 26 seconds for its immediate checkpoint on a local rig, so the exit could still pass xUnit's 11-second window. The drain now runs clusters side by side and up to 16 drops at once under one deadline, returns when it passes whatever is in flight, cancels connects and quiesces at the deadline (never a sent drop), and counts what it did not finish as Unfinished. The one-connect probe of a dead cluster and the quiesce before each drop are kept. Tests: four hanging clusters drain in 3.0 s (12.0 s on the old drain, past the window); a 1-second budget returns in 1.04 s with all three unfinished; a live leak of six databases returns by the budget and accounts for each. Refs #5480
erikdarlingdata
merged commit Oct 8, 2026
c29d8b8
into
fix/nightly-foreground-thread
9 checks passed
erikdarlingdata
deleted the
fix/nightly-foreground-thread-bounded-drain
branch
October 8, 2026 07:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #5480.
Stacked on
fix/nightly-foreground-thread(c1eba1e). Another lane worked the same brief and pushed that branch at 07:51Z, with a nightly in flight (run 37746048228). This PR has one commit on top of it, so its base is that branch, not dev. It replaces that branch's exit-drain budget with one that holds even when a drop is already running. The seven-class leak fix andScratchDatabaseDisposalCensusTestscome from c1eba1e and are unchanged here.Cause
Total: 28763, Failed: 0, then the summary at 07:18:35.7Z, thenWaiting 10 seconds for foreground threads to exit...at 07:18:36.71Z, then[FATAL ERROR] Foreground threads were left runningat 07:18:46.72Z, and exit code 1.xunit.v3.mtp-v2.dll(src/common/MicrosoftTestingPlatform/TestPlatformTestFramework.cs). Afterapp.RunAsync()returns, it queues a thread-pool item that waits 1 s, prints "Waiting", waitsshutdownForegroundThreadWaitSeconds(default 10), then callsEnvironment.Exit(1). It never checks for threads. Anything that keeps the process alive past 11 s trips it, and that includes a ProcessExit handler.ScratchPostgresProcessExit drain was still dropping leaked scratch databases, one at a time, each with the quiesce Four Darling live tests wait for the state they assert #5480 added. On Oct 8 that took more than 11 s (39 lines, about 0.36 s each). On Oct 7 (run 37581130191, passed) it took about 4.7 s.What this commit changes (ScratchPostgres.cs, ScratchPostgresExitDrainTests.cs)
c1eba1e's drain stops starting drops after 6 s, but a drop it already started can still run its quiesce (up to 10 s) and its
DROP DATABASE(up to 30 s). On a local rig, oneDROP DATABASEwaited 26 to 30 s for its immediate checkpoint (pg.log:checkpoint complete: wrote 8922 buffers ... sync=16.786 s, total=26.405 s). That drain can still overrun the 11 s window.This commit:
ExitDrainBudgetSeconds, same name and value);ExitDrainOutcome.Unfinished;ScratchPostgresQuiesceCensusTestsstill passes);QuiesceTimescaleJobsAsynctake an optional caller token, linked with its 10 s cap.Evidence
Test plan
Darling.Testsbuilds with 0 warnings and 0 errors.ScratchPostgresExitDrainTests(9, live ones included),ScratchDatabaseDisposalCensusTests,ScratchPostgresQuiesceCensusTests,LivePostgresCollectionHygieneTests,LiveCleanupConversionRatchetTests,DocCommentHygiene*andRawGatePerSlotSourceFloorLiveTests: Total 124, Failed 0.RawRepairEpochTriggerLiveTests,RawPurgeTriggerGateErrorTestsandRawGatePerSlotSourceFloorLiveTestson the rig, with no "outlived" line naming them.publish=false,test_map=false; publish is skipped whenpublish=false): expectFailed: 0, noFATAL ERRORline, and 0outlived its testlines (only the deliberate20010101drain-test names).For the coordinator
fix/nightly-foreground-threadand send that branch to dev, or take c1eba1e alone and accept the unbounded in-flight drop.TheDrain_OutOfBudget_StartsNoDrop...now expectsUnfinished == 3andSkipped == 0. Skipped now means only "its cluster did not answer".No changelog entry (test infrastructure).