Skip to content

Test processes no longer pay 0.3 s per WPF STA thread at exit, which failed green Darling runs (#5596) - #5598

Merged
erikdarlingdata merged 5 commits into
devfrom
fix/5596-foreground-threads
Oct 8, 2026
Merged

erikdarlingdata merged 5 commits into
devfrom
fix/5596-foreground-threads

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Closes #5596. A CI-only fix: no product code changed.

Why

The "Darling whole-tree guards" job runs the whole Darling.Tests suite in one process. After every test passed it sometimes printed "Waiting 10 seconds for foreground threads to exit..." and then "[FATAL ERROR] Foreground threads were left running, forcing process exit", exit code 1, 0 failed tests. Three of four Lite.Tests shards printed the wait line in green runs too and exited 1 to 3.6 seconds later.

No thread was left running. The runner only reports "the process was still alive 1 second after the run returned" and forces exit 10 seconds after that, so anything slow in the exit stage trips it.

What held the process

WPF's own weak-event exit handlers.

  • Every STA test thread that uses a WPF WeakEventManager (a bound DataGrid, say) makes WPF create a WeakEventTable for that thread's dispatcher. The table registers a ProcessExit handler (ShutDownListener.HandleShutDown, target WeakEventTableShutDownListener) that nothing removes when the thread ends.
  • At exit the handler runs on the main thread. WeakEventTable.OnShutDown sees that the table's own thread is gone, and calls Dispatcher.Invoke(..., TimeSpan.FromMilliseconds(300)) on a dispatcher nobody will ever run. It waits the full 300 ms, once per table.
  • About fifty test files start their own STA thread, so the whole-tree run ends with a long list of these handlers. The exit stage passed the 11 seconds xUnit v3 allows after the run returns. The "Pipe is broken" from the diagnostics tool in the first checkpoint fits: the runtime was already in its exit stage.

Evidence (the diagnostic workflow and module initializer were temporary and are deleted):

  • On a laptop, one test class (ColumnFilterPersistenceTests, 38 tests, 1.1 s) left 13 weak-event-table handlers. Timing each handler showed 13 x ~300 ms (about 4.0 s) in WeakEventTableShutDownListener, nothing else slow (the data-binding listeners, the column-filter flush and the module uninitializer each took 0 to 1 ms). The process printed the wait line and exited about 4 s after the summary.
  • With the switch below the same class exits at once (no wait line, exit within 50 ms of the summary), measured three times.
  • The runner's ProcessExit handler list (read through the runtime's event field) names the same handlers: 13 pairs of WeakEventTable and DataBindEngine listeners for that one class.
  • Ruled out: a generic host with ConsoleLifetime (.NET 10 uses signal handlers, not a ProcessExit hook). Also ColumnFilterStore's exit flush (0 ms), ScratchPostgres's exit drain (no database in the guards job), and every new Thread( (all background or joined).
  • Removing the handlers from inside a ProcessExit handler does not work: the runtime has already taken its snapshot of the list. The fix has to be in place before the process starts, which is why it is a runtime configuration switch.

What changes

  • Darling.Tests.csproj and Lite.Tests.csproj set the WPF runtime switch Switch.MS.Internal.DoNotInvokeInWeakEventTableShutdownListener to true. The handler then purges the table directly. Both test projects start WPF STA threads, so both get it (the same cost showed in the Lite shards).
  • WpfExitSwitchTests (new, in Darling.Tests, linked into Lite.Tests so each process checks itself) has three checks. The switch is on in this process. The installed WindowsBase.dll still contains the switch name, because a renamed switch would be a silent no-op. A table's own exit handler, picked out by the table it holds, returns in under 250 ms for a table whose STA thread ended. The assertion takes the best of three runs, and the old shape is 300 ms or more every time.
  • WpfExitSwitchPinTests (new, guard stage): both project files declare the switch. Messages name The Darling whole-tree guards job fails when a foreground thread outlives the suite #5596 and the rule.
  • build.yml: the new linked file is added to the lite_linked_shard filter, as DarlingReadsTraitGuardTests requires for every file Lite.Tests compiles from Darling.Tests.

Test plan

  • RED on the old shape: with the switch taken out of the built runtimeconfig.json, WpfExitSwitchTests fails 2 of 3 (the switch check and the timing check). With the project files as on origin/dev, WpfExitSwitchPinTests fails 2 of 2.
  • WpfExitSwitch* in Darling.Tests: Total 5 (3 + 2), Failed 0. WpfStaGateCensusTests with them: Total 7, Failed 0.
  • Lite.Tests, WpfExitSwitch*, DarlingReadsTraitGuardTests, GuardStage*, WpfStaGateCensus*: Total 20, Failed 0.
  • Every *Census* and *Ratchet* class in Darling.Tests, plus WpfExitSwitch*: Total 505, Failed 0, Skipped 1.
  • -trait Stage=Guard -failSkips on Darling.Tests: Total 1434, Failed 0, Skipped 0.
  • 0 build warnings in Darling.Tests and Lite.Tests.
  • Proof at suite scale, on a GitHub runner with the switch on (the temporary diagnostic workflow, which ran the whole Darling.Tests suite exactly as the guards job does): Total 29398, Failed 0, process exit code 0, and no "Waiting 10 seconds" line. The last ProcessExit handler ran 0.17 s before the process was gone.
  • Baseline at suite scale, same runner image and same commit, switch removed from the built runtimeconfig.json (job diag (WPF switch off) of the temporary Diag 5596 workflow): Total 29398, Failed 2 (the two switch checks in WpfExitSwitchTests, the expected RED), the runner printed "Waiting 10 seconds for foreground threads to exit..." and then "[FATAL ERROR] Foreground threads were left running, forcing process exit", exit code 1. The 118 ProcessExit handlers took 17.9 s from the first to the last. With the switch on (job diag (WPF switch on)) the same 118 handlers took 0.03 s, exit code 0, Failed 0, no wait line.
  • The "Darling whole-tree guards" job on this PR: draft PRs skip CI, so it runs once the PR is ready.

Handoff

Done: cause found, fix, tests, diagnostic workflow and module initializer deleted, squash body written. Left: the "Darling whole-tree guards" job result on this PR now that it is ready.

CHANGELOG entry (CI-only fix):
SECTION: Fixed
ENTRY: - The Darling whole-tree guards CI job no longer fails after every test passed ([#5598]) - WPF left a process-exit handler for every test thread that used a bound control, each waiting out a 300 millisecond timeout, so the test process outlived the runner's 11 second exit window and the runner forced exit code 1 with no failed test; both test projects now set the WPF switch that skips the wait. This is a CI-only fix: no product code changed.

Each STA test thread that touches a WPF weak-event manager leaves a ProcessExit
handler that waits out a 300 ms dispatcher invoke on a dead thread. The WPF switch
that makes the handler purge directly is now set in both test projects, with a test
that times the handler and a pin on the project files. The diag workflow and module
initializer stay for one more run.
@erikdarlingdata erikdarlingdata changed the title Checkpoint: finding what keeps the Darling test process alive after the run (#5596) Test processes no longer pay 0.3 s per WPF STA thread at exit, which failed green Darling runs (#5596) Oct 8, 2026
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 8, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant