Repository navigation
Give the nested process chain in ProcessWaitingTests depth-proportional wait budgets - #135608
Merged
jkotas merged 2 commits intoOct 12, 2026
Merged
Conversation
…al wait budgets WaitChain and WaitAsyncChain start a four-deep chain of RemoteExecutor processes and every level waits with the same WaitInMS. The root's wait has to cover the start-up, work and exit of all four processes while the leaf's covers one, so the root has the leaf's budget. In the failure stacks I have for dotnet#118977 the exception is at the root wait, and the recent failing durations are 30.6-30.9 s, the 30 s budget of the x86 Checked lane. With this assembly's test classes running in parallel (dotnet#135160, dotnet#135323), several of the single- and two-process waiting tests on that lane take about 3-10.5 s each and passing chains already take 20-28 s, which leaves the root's budget little margin. Give each waiter WaitInMS per process beneath it: leaf 1x, then 2x, 3x and 4x at the root. Test-only; the product wait paths are unchanged, and the single-process waiting tests keep their budgets. Fixes dotnet#118977
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
dotnet-policy-service
Bot
added
the
community-contribution
Contributor
|
Tagging subscribers to this area: @dotnet/area-system-diagnostics-process |
jkotas
reviewed
Oct 12, 2026
Member
|
Thanks a lot for providing the fix @pcshrosbree ! |
Member
|
/backport to release/9.0 |
Contributor
|
Started backporting to |
4 tasks
Member
|
/backport to release/9.0-staging |
Contributor
|
Started backporting to |
2 of 4 tasks
Member
|
/backport to release/11.0 |
Contributor
|
Started backporting to |
2 of 4 tasks
ViveliDuCh
pushed a commit
that referenced
this pull request
Oct 12, 2026
…Tests depth-proportional wait budgets (#135632) Fixes #118977 <!-- Issue Number --> main PR #135608 <!-- Link to PR if any that fixed this in the main branch. --> Backport to `release/9.0-staging`. /cc @ViveliDuCh @pcshrosbree # Description <!-- Give a brief summary of the issue and how the pull request is fixing it. --> The same `WaitChain`/`WaitAsyncChain` timeout failure seen in main also occurred in `release/9.0`. Scale each wait budget by the number of nested processes it covers. # Customer Impact <!-- What is the impact to customers of not taking this fix? --> - [ ] Customer reported - [x] Found internally Intermittent CI test failures; no product behavior change. # Regression <!-- Is this fixing a problem that was introduced in the most recent release, ie., fixing a regression? --> - [ ] Yes - [x] No Existing test timeout issue; no known product regression. # Testing <!-- What kind of testing has been done with the fix. --> #135608 reports both chain tests passing on Windows x86, Linux x64, and macOS arm64. No additional local tests run for this backport; CI is still in progress and currently has failures. # Risk <!-- Please assess the risk of taking this fix. Provide details backing up your assessment. --> Low: test-only timeout adjustment. Assertions and product code are unchanged. Co-authored-by: Peter Shrosbree <peter.shrosbree@gmail.com>
ViveliDuCh
pushed a commit
that referenced
this pull request
Oct 13, 2026
…epth-proportional wait budgets (#135643) Backport of #135608 to release/11.0 /cc @ViveliDuCh @pcshrosbree The same `WaitChain`/`WaitAsyncChain` timeout failure seen in main also occurred in `release/11.0`. Scale each wait budget by the number of nested processes it covers. # Customer Impact <!-- What is the impact to customers of not taking this fix? --> - [ ] Customer reported - [x] Found internally Intermittent CI test failures; no product behavior change. # Regression <!-- Is this fixing a problem that was introduced in the most recent release, ie., fixing a regression? --> - [ ] Yes - [x] No Existing test timeout issue; no known product regression. # Testing <!-- What kind of testing has been done with the fix. --> #135608 reports both chain tests passing on Windows x86, Linux x64, and macOS arm64. No additional local tests run for this backport; CI is still in progress and currently has failures. # Risk <!-- Please assess the risk of taking this fix. Provide details backing up your assessment. --> Low: test-only timeout adjustment. Assertions and product code are unchanged. Co-authored-by: Peter Shrosbree <peter.shrosbree@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #118977
Problem
ProcessWaitingTests.WaitChainandWaitAsyncChainstart a four-deep chain ofRemoteExecutorprocesses, and every level waits with the sameWaitInMS. The root's wait has to cover the start-up, work and exit of all four processes while the leaf's covers one, so the root has the leaf's budget while covering four lifetimes. In every failure stack I have for #118977 the exception is at the root wait, and the four failing durations across three CI runs are 30.6–30.9 s, the 30 s budget of this x86 Checked lane (SlowRuntimeTimeoutModifieris 1 on a Checked runtime; only a Debug runtime, or RISC-V, gets more).With this assembly's test classes running in parallel (#135160 enabled it on this lane; #135323 removed the remaining macOS-only serialisation), several of the single- and two-process
RemoteExecutortests on thenet11.0-windows-Debug-x86-coreclr_checkedlane take about 3–10.5 s each and the chains that pass are already at 20–28 s (26.1 / 28.4 s in #135160's own pre-merge Helix experiment), which leaves the root's budget little margin. The synchronousWaitChainfails the same way through a kernel wait on the process handle, so at least in that case the root genuinely had not exited; the asynchronous failures have the same durations and root line. The per-run numbers are in the issue comment.Change
Give each waiter
WaitInMSper process beneath it: leaf 1×, then 2×, 3× and 4× at the root, in both tests, throughChainWaitInMS(int processesBelow). Test-only; the product wait paths and the single-process waiting tests are unchanged. Serialising only the two tests was considered and not measured; it runs against what #135160 was after, and the current budgets give the outer waits no allowance for the nested work in any case.Validation
Same source (
bb146503eeb+ this change) on each platform,clr+libs -rc Checked -lc Debug, the inner-loop run ofSystem.Diagnostics.Process.Testswith parallel collections, durations fromtestResults.xml:WaitChainWaitAsyncChain† The same six POSIX-signal tests (
SignalHandler_CanDisposeInHandler×2,TestCreateNewProcessGroup_HandlerReceivesExpectedSignal×4) fail identically in the unfixed and fixed runs on both Unix machines, so they predate this change. Both platforms' runs were launched undernohupfrom a non-interactive shell; a probe of that launch pattern on the Linux machine shows the child process with SIGHUP, SIGINT and SIGQUIT ignored, which points at the launch environment.The chain durations stay the same within run-to-run noise (the budget grows, the work does not). On the Windows machine the unfixed chain also passes at 11–12 s, so the Helix failure itself did not reproduce locally; whether the lane stops hitting it is for CI history after the change to show.
Note
This PR was prepared with AI assistance (Anthropic Claude, checked by OpenAI Codex) at my request. I reviewed the change and the measurements before opening it.