Repository navigation
Tests share WPF threads, Postgres roles and runner CPU by design: give each test its own, and guard it #5602
Copy link
Copy link
Closed
Description
Claude posting for Erik Darling.
What happens
The Darling.Tests and Lite.Tests suites fail CI for reasons that have nothing to do with the change. Sometimes no test fails at all. Each failure blocks every merge until a lane fixes it. Today there were four:
| Issue | Shared thing | What collided |
|---|---|---|
| #5559 | WPF state | Two test classes used a WPF dictionary from two threads at the same time. |
| #5560 | Postgres cluster | Two tests changed the same grants at the same time, and Postgres refused one with "tuple concurrently updated". |
| #5561 | Runner CPU | A timing test on a 27 MB plan went over its limit on a busy runner. |
| #5596 | WPF state | All tests passed, but the process took longer than 11 seconds to exit. Each STA test thread left a WPF exit handler that waits 300 ms. |
The fixes (#5567, #5566, #5563, #5598) each remove one collision. The design that causes the collisions stays, so the next test that uses one of these shared things can fail the same way.
Cause
The tests share three things by design.
- WPF threads. 55 test files call
SetApartmentState(ApartmentState.STA)and start their own thread. Several files carry their own private copy of anOnStaThreadhelper.WpfStaGate(Lite and Darling WPF tests take one gate so parallel STA classes stop racing a shared WPF cache (#5559) #5567) now makes these tests take turns, but each call still starts a new thread. So the WPF state per thread and the exit handlers per thread grow with the number of tests. - The Postgres cluster.
ScratchPostgresgives a test its own database. Roles and the grants on a database are stored for the whole cluster. So tests that run at the same time can still change the same rows. - Runner CPU. About 45 test files compare an elapsed time with a limit, and they run in parallel with other tests. This count comes from a grep, not a full audit.
No open issue covers this.
What to fix
- Run every WPF test on one STA thread for each test process. The test host starts that thread with a running
Dispatcherat the first use. One shared helper runs a test body on it. Remove the privateOnStaThreadcopies. RemoveWpfStaGatewhen nothing uses it. Some tests must have their own thread, for example a test of dispatcher shutdown. Put each of them on an allow list with the reason. - Make each test change only Postgres objects that no other test can see. Audit every test that creates or changes a role or a grant. Give each such object a name that includes the scratch database name. If product code changes an object that every test shares, put those tests in one collection that runs alone.
- Stop timing tests from competing for CPU. Some tests can assert on the work done: rows read, plan nodes walked, or calls made. Change those tests to do that. Put each test that must check an elapsed time in one collection with
DisableParallelization = true. Judge the fastest of three runs, as The 27 MB plan walk and 20 MB cut-walk timing pins judge the fastest of three runs (#5561) #5563 does. - Add a census test like the existing Census and Ratchet classes. It fails when a test file starts its own STA thread outside the allow list. It also fails when a test checks an elapsed time outside the timing collection.
- Put two numbers in the PR, before and after the change. The first is the time of the "Darling whole-tree guards" job. The second is the exit time of each test shard after its last test.
Order
Lanes can start now on one branch. The open PRs #5570, #5579, #5588, #5591, #5595 and #5599 change test files too, so rebase onto dev after they merge. Then convert any new test that the census test catches.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
Timing tests run alone in one collection, and a census keeps them the…
WPF tests run on one shared STA thread per test process (#5602)
StaThreadCensusTests reads code only, through the shared source walker (
StaTestThreadTests join the timing collection (#5602)
Live Postgres tests that write the cluster's fixed roles or settings …
WpfExitSwitchTests takes the shared STA exclusive gate and runs in th…
The time range picker's WPF tests run on the shared STA thread (#5602)
Live-postgres timing leftovers: the lock acquire moves to timing, the…
erikdarlingdata commentedon Oct 9, 2026
Fixed by #5604, merged to dev as 0d7e76f. WPF tests now run on one shared STA thread per test process. Postgres tests that change the fixed roles or server-wide settings run alone. Tests that check elapsed time run alone in one timing collection. A census test for each keeps new tests from sharing these again. Claude posting for Erik Darling.
Read-budget tests: floor the timed-out slice at half the budget, with…