Repository navigation
Test failure: System.Diagnostics.Tests.ProcessWaitingTests.WaitAsyncChain #118977
Description
Activity
Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.The synchronous version of the same test also failed for the first time in years: #86416 (comment)
Let's wait and see if it was a one time thing.
Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.I looked into this after it hit the CI of PRs I'm involved in this week (#135597, #135601, #134749, #134883). What I find is a test-budget shape in the two chain tests, which the class-level parallelism this assembly runs with since #135160 (it enabled class parallelism on this lane; #135323 then removed the remaining macOS-only serialisation) appears to have brought close to its limit on the x86 Checked lane. I do not see a product defect, and I have a small test-only change validated on three platforms (PR to follow).
What fails.
WaitChainandWaitAsyncChainstart a four-deep chain ofRemoteExecutorprocesses, and every level waits with the sameWaitInMS. The root's wait has to cover the start-up, work and exit of all four processes while the leaf's covers one, so the root has the leaf's budget while covering four lifetimes. In every failure stack I have (four CI runs and the issue's original) the exception is at the root wait, and the four failing durations across the three runs below are 30.6–30.9 s.SlowRuntimeTimeoutModifieris 1 on a Checked runtime (only a Debug runtime, or RISC-V, gets more), so the budget on this lane is 30 s. The synchronousWaitChainfails the same way, and on Windows that is a kernel wait on the process handle (WaitForExitCore→WaitOne), so at least in that case the root genuinely had not exited at 30 s; the asynchronous failures have the same durations and the same root line.Why now. With
parallel test collections = on [4 threads], several of the single- and two-processRemoteExecutortests in these runs take about 3–10.5 s each on this lane, and the chains that pass are already at 20–28 s. The pre-merge Helix experiment on #135160 shows the same picture (four-thread class parallelism, x86 Checked runtime from build 1621730, per its description):run WaitChainWaitAsyncChain#135160 pre-merge experiment, Helix job e2b3d201, 2026-10-0526.10 s pass 28.35 s pass build 1630016 (#135517), 2026-10-09 28.09 s pass 30.90 s fail build 1631224 (#135594), 2026-10-11 30.63 s fail 30.59 s fail build 1631242 (#135597), 2026-10-11 20.42 s pass 30.83 s fail (Durations are the
timeattribute of each test in the work item'stestResults.xml.) The 2025 hit in this issue was a Release x64 JitStress lane, so the sensitivity is older than the parallelism change; the current burst (12 hits in 7 days) started after it.Paired measurement. On my own Windows machine, main
bb146503eebbuiltclr+libs -rc Checked -lc Debug -a x86, the Helix payload run throughRunTests.cmdwith the process pinned to 4 logical CPUs so xunit uses 4 threads as on Helix, three runs each:configuration WaitChainWaitAsyncChainassembly run unfixed, parallel collections (4 threads) ×3 10.9 / 11.2 / 11.2 s 12.1 / 11.8 / 11.9 s 228–242 s unfixed, same binaries, -parallel none×37.0 / 6.9 / 6.9 s 7.1 / 7.4 / 7.4 s 505 s ×3 fixed, parallel collections (4 threads) ×3 11.6 / 11.5 / 11.2 s 12.1 / 11.8 / 11.8 s 217–241 s On this (much faster) hardware the parallel runs measured the chain about 1.6× longer than the serial runs of the same binaries. It does not reach 30 s here, so I could not reproduce the failure itself; these runs associate parallel execution with longer chain durations on this machine.
Change. Give each waiter
WaitInMSper process beneath it: leaf 1×, then 2×, 3× and 4× at the root, in both tests, through a one-line helper. The product wait paths and the other waiting tests are untouched. I considered serialising just the two tests but did not measure it; it runs against what #135160 was after (its description measured the assembly at 763 s serial against 312 s parallel on the same x86 Checked runtime), and the current budgets give the outer waits no allowance for the nested work in any case. Both chains pass on Windows x86 (the three fixed runs above), linux-x64 and osx-arm64, each on a Checked runtime; the per-platform table is in the PR. Whether the lane stops hitting this is something only CI history after the change can show.Note
This comment was prepared with AI assistance (Anthropic Claude, checked by OpenAI Codex) at my request. I reviewed it before posting.
Failed in: runtime-coreclr libraries-jitstress 20250821.1
Failed tests:
Error message:
Stack trace:
Known Issue Error Message
DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.
Fill the error message using step by step known issues guidance.
{ "ErrorMessage": ["System.Diagnostics.Tests.ProcessWaitingTests.WaitAsyncChain [FAIL]","A task was canceled."], "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false }Known issue validation
Build: 🔎⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.
Result validation:
Validation performed at: 10/9/2026 8:25:52 PM UTC
Report
Summary