Flaky test classes: the statement filter tests no longer depend on the runner's speed (Refs #5459)

[erikdarlingdata]

Refs #5459

Four lanes work this branch. A takes the classes that need no database, B the PostgreSQL live classes, C the Lite classes, and D the timing flakes the others left. Each lane adds its rows to the class table. Users see no change. The product code gains two test seams with the same default.

SECTION: None

Class table

Class	Cause	Fix	Proof
StatementFilterAlertTests	The statement filter judges text under a wall-clock budget. That is 1.5 s plus 0.5 s per MB of each distinct document. A slow runner spent it on the 4 MB report tests, so the document came back withheld whole.	AlertStatementFilter gets an internal UseBudget seam (AsyncLocal). The default is unchanged. The 4 MB tests pin a one-hour budget. Two new tests hand the filter a spent budget and expect everything withheld unjudged.	18/0 at the time. Lane D added more rows for this class below.
DarlingStoreUpgradeRevertTests	A child that exits at once made WaitForExitAsync return without looking at an already-cancelled token.	Already fixed by #4867 (6f2f170). The three managed-PostgreSQL tool runners check the token first.	With that check commented out, the two pins for it fail (2 of 251). Restored, 0 failed.
DarlingManagedPostgresTests	A just-stopped store still maps a file for a moment, so Directory.Move met a sharing violation on Windows.	Already fixed by #4864 (ad79be9). MoveDirectoryOnceReleased retries for up to 10 s.	MoveDirectoryOnceReleasedTests 3 of 3. The real-runtime test passed on an extracted runtime.
DarlingUpgradeInstallTreeWaitTests	Two causes. A cold PowerShell start outlasted 60 s, which #4911 (7dcd3fe) fixed with 180 s. Also, the process the wait must refuse on lived only 30 s.	The holder now lives the child's limit plus two minutes. The test kills it when it is done with it.	Delaying the child's start by 35 s with the old holder fails 1 of 2. The same delay with the fix passes 2 of 2.
DarlingDeployStaleFileTests	The same cold PowerShell start. The probe's own work is small and bounded.	Already fixed by #4866 (545881f): a 180 s limit, pinned by DeployPowerShellProbeLimitTests.	The class and the pin pass.
LocalClockBucketKeyTests (Lite)	Not a class flake. Two tests failed on every run of one stacked series of branches that was also editing the file.	None needed.	The two names never existed on the base. 10/0 at head.
ScenarioTests (Lite)	The CPU-spike scenario seeded its period from the raw clock. A run in the hour before 04:00 UTC saw an almost empty baseline.	Already fixed by #4385 (6516e28). The period is anchored to a UTC midnight.	With the clock at 03:55 UTC and the old period, the same 3 tests fail. With the anchor, 41/0 at five clock times.
AnomalyDetectorTileBehaviourTests (Lite)	At 21:xx UTC the hours +2 and +3 are 23 and 0, so InRange(x, 23, 0) can never pass.	Already fixed by #4183 (64e9911). The assertion compares the hour offset.	The old assertion at 21:05 UTC fails 2 of 2. The new one passes at each of the 24 hours.
StatusBarSizeReadLockTests (Lite)	The writer's 5 s wait for the process-wide database lock ran out under load.	Already fixed by #4670 (003980e). The wait is a 30 s hang backstop.	No failure since 21 Sep. 1/0 at head.
McpPageContractTests (Lite)	The test seeded today's run two minutes back, which lands on yesterday just after midnight UTC.	Already fixed by #3963 (bbfa65d), with the same guard in the Darling twin by c01e1d9.	No failure since 22 Sep. 29/0 at head.
PerformanceCalendarDataTests (Lite)	Failed on its own PR branch before that PR merged. Not a base flake.	Fixed inside that PR (f681802).	3/0 at head.
MemoryTrendGrantJoinToolTests (Lite)	Seeds a few seconds apart straddled a trend bucket boundary on about one run in ten.	Already fixed by c01e1d9. Seeds start 10 s past a five-minute floor.	No failure since 23 Sep. 3/0 at head.
QueryStoreRegressionsToolTests, SignificantWaitsToolTests (Lite)	Failed on a merge-train branch whose empty-answer wording differed from the pin. Deterministic, not a flake.	Already fixed by #5250 (785c433).	3/0 each at head.
FinOpsTests (Lite)	See the "already fixed" list below.	#4558 and #4891.	35/0 at head.
DuckDbSentinelConnectionTests (Lite)	RunMemoryTrimCycle skips silently when another test holds the process-wide lock. The test re-ran it for one second only.	Test only. The test waits on CompletedTrimCycleCount, which moves only after a cycle really ran, with a 30 s hang backstop.	A planted 3 s hold fails the old loop with the CI numbers. The new loop passes. 14/0 at head.
TestCpuAccountantTests (Lite), burn	Burn() spun for 40 ms of wall clock, but the accountant charges thread CPU time. A descheduled thread charges less.	Test only. Burn() spins until the thread's own CPU counter moves by the burn.	A duty-cycle plant fails the old burn 4 of 4 and passes the new one 4 of 4.
TestCpuAccountantTests (Lite), idle side (lane D)	GetThreadTimes moves in ticks of about 15.6 ms. The idle side was bounded under 10 ms, below one tick. One clock interrupt landing on a waiting thread read as 15 ms and failed Two_tests_in_parallel_are_charged_separately. The unflowed-work test has the same shape.	Test only. The test measures the counter's tick. The idle side must stay under two ticks and the busy side burns at least four. A new test plants one stray tick on the idle side.	The old bound with a one-tick plant on the idle side fails with the CI message "the idle test was charged 15 ms". It fails the same way at the base. The new bound passes the plant. 6/0, three runs.
StatementFilterAlertTests, three incidents (lane D)	The test read the filtered result under the real 1.5 s plus 0.5 s per MB budget. Dev failed with "Expected 3, Actual 2". Lane A's pinned budget already covers that assertion at head.	Test only. A new stepped clock charges each judge call a fixed time. A new test runs the three-incident case with a budget that runs out part way.	The old assertion under a 50 ms step reads "Expected: 3, Actual: 0". A 0.26 ms step reads "Actual: 1". The new test passes with 3 distinct passes and an unchanged limit.
StatementFilterAlertTests, slow judge (lane D)	The test asserted took < 1950 ms on the real clock. A runner that stalls for a moment fails it though the budget logic is fine.	Test only. The budget uses the stepped clock, 50 ms per judge call. The test asserts the budget ran out, values were withheld, and the charged time passed the limit by at most one call. The wall bound stays at 19.5 s as a hang backstop.	A planted 2 s stall fails the old test ("took 2551 ms") and passes the new one. 19/0 for the class.
StatementCollectionCensusTests, snapshot worst case (lane D)	The session budget is 15 s of wall time. The worst-case read has 200 plans and one 27 MB plan. The only assertion that depends on load was < 20 s.	CollectorContext gets an internal ScrubSessionFactory seam, null by default. The test pins an hour-long session and expects all 201 plans judged and none withheld. A new test runs a spent session and expects all 201 withheld. The wall bound is a 2 minute hang backstop.	A planted 21 s hold fails the old test ("the read took 00:00:23") and passes the new one. The read takes about 1.8 s alone. 46/0 for the class.
DarlingEncryptPasswordStreamTests (lane D)	Neither child waits on anything of its own. The verb reads one closed stdin line and makes one DPAPI call. What a busy runner stretches is the start of a fresh executable and of a cold PowerShell 5.1. The waits were 60 s and 120 s.	Test only. Both waits use a named 180 s limit, as #4866 and #4911 did. DeployPowerShellProbeLimitTests pins it for this file too.	The pin fails against the old file and passes now. A real slow start cannot be planted without a hold of over a minute, so the cause stands by elimination. 7/0.
RollupBackfillLiveTests (mid-slice cancellation)	The failing message was the old one, from before the cancel was held. That test cancelled after polling the coverage floor, and a fast runner committed every batch between two polls.	Already fixed by 75bbb21 (#4856). It holds a raw chunk lock mid-range, waits for the refresh to be seen waiting, and only then cancels.	9/0 on a throwaway rig. A theory about a frozen postmaster was tested and refuted.
QueryStoreIntervalWideGridLiveTests (round trips 1, expected 0)	The scan counters were read before a backend's throttled stats report landed.	Already fixed by 33252ff (#4449). It flushes the seeding connections' pending stats first.	4/0 on the rig. A planted stray pooled backend did not make the head fail, so a speculative change was dropped.
ComposeStoreRolesLiveTests	Two things. PostgreSQL prints a 60 s statement_timeout as 1min. The read timeouts were not reproduced.	The first is already fixed by e254521 (#4447). The helper compares pg_settings.setting in milliseconds. The second has no failing sample to trace.	9/0, nothing skipped, on a rig with the bundled runtime.
PgLogEventsLivePostgresTests	At the midnight rotation pg_ls_logdir() reports modification in whole seconds. The two files tied and the tail read the outgoing one.	Already fixed by dba249f (#4738). The newest file is ordered by modification, then name.	With three sibling classes, 19 total, 0 failed, 2 skipped (they need their own clusters).
DarlingMcpStoreHostToolsLiveTests	The tool's chunk count was compared with one owner read taken after the call, while other classes changed chunks.	Already fixed by dd54e57 (#4815). The count is compared only when the owner's reads before and after agree.	Same run as above, 0 failed.

Already fixed on the base (the failing message matches the fixed cause)

• PgTargetAnomalyTests (window shrank on a slow runner): Darling worker: no signal when the fleet gate skips slots, and Extended Events sessions left on removed servers #4732 (a9e5da9).
• McpToolsListBudgetTests (a reflection race leaked a DI service into tools/list): MCP tool schemas exclude DI services by type, closing a reflection race that leaked them into tools/list (issue-4075) #4110 (b1a7def).
• RetiredBaselineAggregateLiveTests (connection lost to a refresh-job race): The retired-baseline sweep reopens its connection after a drop loses a race with a refresh job #5439 (82678f5).
• Lite FinOpsTests (CPU samples outside the 24-hour read at some hours): Lite tests: keep the high-impact query tests' read window over their seeded rows at any hour #4558 (834e596), Lite FinOps tests: CPU samples sit inside the 24-hour utilization read at any time of day #4891 (8eeea9c).
• PgServerLogTailRotationLiveTests, PgLogEventsCsvlogLiveTests, PlanSync4512CancellationTests (exact count 1 where PostgreSQL logs a lock wait twice): Four Darling live tests wait for the state they assert #5480 (483bb7d).
• DarlingCustomViewsTests, ReadLatencyWebRecordingTests (a test-only route leaked across classes): Darling tests: a test-only read route leaks into other test classes running at the same time #4782 (9f4f971), 6179609.
• DarlingCloudIdentityProbeTests (wall-clock bound on a starved pool): Cloud probe budget test fails when a starved thread pool fires the budget timer late #4741.
• PgLogBurstAndRotationLiveTests: Tests: the two-rotation log-tail test allows PostgreSQL 18's repeated lock-wait line #4919.

Not class flakes

• A fixture failure where timescaledb_information.continuous_aggregates was missing (7 classes). It was one PR's own migration defect, fixed in that PR.
• McpToolsListBudgetTests and AlertReadFailureSurfaceTests failing on a PR. Those are census gates, deterministic when a PR grows a description or adds an uncounted catch.
• Infra: two shard failures that lost events, and one PostgreSQL shard that exited non-zero with "Failed: 0, Not Run: 1" and no per-test result.

Product code changed

• PerformanceMonitor.Alerting/AlertStatementFilter.cs: NewBudget() replaces the four new JudgeBudget(ReadBudget) sites. UseBudget (internal, AsyncLocal) is the test seam.
• PerformanceMonitor.Collectors/CollectorContext.cs (lane D): BeginStatementScrub() uses ScrubSessionFactory?.Invoke() ?? new SensitiveStatements.Session(). The property is internal and null in production.
• Nothing is set in production, so both defaults are what they were. No user-visible change.

Test plan

• StatementFilterAlertTests 19/0, with StatementFilterDeliveryTests, SensitiveStatementRawCleanTests and StatementFilterWarmUpWaitTests: 81/0.
• DarlingUpgradeInstallTreeWaitTests, DarlingDeployStaleFileTests, DeployPowerShellProbeLimitTests, MoveDirectoryOnceReleasedTests, DarlingStoreUpgradeRevertTests, DarlingManagedPostgresTests: 278/0, 9 skipped (they need a PostgreSQL runtime).
• Lane D: TestCpuAccountantTests 6/0, StatementCollectionCensusTests 46/0, DeployPowerShellProbeLimitTests with DarlingEncryptPasswordStreamTests 7/0.
• -trait Stage=Guard -failSkips: Darling.Tests 1372/0, Lite.Tests 372/0.
• Census 454/0 (1 skipped), Ratchet 15/0, DocCommentHygiene 78/0.
• Lane C, Lite: LocalClockBucketKeyTests 10/0, ScenarioTests 41/0, AnomalyDetectorTileBehaviourTests 2/0, StatusBarSizeReadLockTests 1/0.
• Lane C, Lite, continued: SignificantWaitsToolTests 3/0, QueryStoreRegressionsToolTests 3/0, PerformanceCalendarDataTests 3/0, MemoryTrendGrantJoinToolTests 3/0, McpPageContractTests 29/0, DuckDbSentinelConnectionTests 14/0, FinOpsTests 35/0.
• Lane B: rollup backfill and wide-grid 13/0, store roles 9/0, log events and siblings 19/0 with 2 skipped, on a throwaway rig.
• Build: 0 warnings on both test projects.
• Full suite (CI decides).

Handoff

Lane D finished all six items. One thing is left alone. DarlingEncryptPasswordStreamTests has no planted slow start, because that needs a hold of over a minute. The pin proves the limit, and the cause stands by elimination.

Lane A and lane B had nothing left. Lane B dropped a speculative change to the wide-grid class because a planted stray backend did not fail the head. If that class fails again, check for a pooled backend with a pending scan.