Enable class-level parallelism in Process tests

[steveisok]

Motivation

Mitigate #135001's Process-suite execution-budget exhaustion by isolating ambient-state-sensitive tests and restoring default xUnit class-level parallelism on Windows/Linux. This follows the review suggestion and replaces the previous sharding approach without rewriting history.

Desktop macOS retains the original assembly-wide serialization. Recent Mono x64 and Checked CoreCLR arm64 runs timed out under class parallelism. This is a compatibility safeguard against the unresolved concurrency problem, not a root-cause fix or a claim that macOS CI is now green.

Changes

• Run the three parent-environment tests wholly inside RemoteExecutor, preserving nested child inheritance, overrides, null removal, and fixture disposal. Process exit handles environment cleanup.
• Run the Windows encoding test inside a remote process that detaches from the inherited console and allocates a private console. Preserve native/code-page checks and the Unix/container UTF-8 path; process exit frees the private console.
• Isolate the process-wide first-chance-exception counter with RemoteExecutor.
• Use a unique directory and remote working directory for the Unix executable-resolution test. Its standard RemoteExecutor.IsSupported condition intentionally excludes that one outer-loop test when RemoteExecutor is unsupported; the bespoke single-file fallback stays removed.
• Define TARGET_OSX only for TargetOS=osx with the Unix target framework, then apply the original CollectionPerAssembly attribute. This uses the actual test build target, not the build host, and applies to both Mono and CoreCLR. Windows/Linux keep class-level collections; Android, MacCatalyst, and Browser behavior is unchanged. The Browser assembly skip is preserved.

The existing single project and packaging remain. No sharding, thread caps, timeout increases, product-code changes, or PR-specific CI gates. The macOS safeguard changes scheduling only, not test eligibility.

Validation

Supplemental compilation and actual xUnit discovery passed for eight target configurations using cached CI dependencies. The macOS Mono x64, CoreCLR arm64, and single-file source configurations each place all 765 locally discovered rows in one shared collection. Windows/Linux retain class collections; Android/MacCatalyst/Browser remain class-based. Cross-target evaluation on Windows verified that the gate follows TargetOS, and an osx build with a MacCatalyst target framework does not enable it. Local execution of two inexpensive tests from different classes confirmed one executed collection for the macOS-targeted assembly versus two for Windows/Linux. These are cached-runtime metadata/smoke checks, not actual macOS or NativeAOT execution; collection membership restores serialization even when the runner reports multiple available threads.

The prior cleanup revision passed all four affected Windows tests and ten concurrent parent-state probe calls with zero violations across 3,747 samples. Supported discovery/traits remained 724 Windows rows and 765 Unix rows, with 689 ordinary Windows cases; the only unsupported-runner eligibility change was the accepted Unix outer-loop condition.

Earlier local serial/parallel full-suite times were 763.201s and 311.975s on the same saved x86 Checked runtime. Both had the same job-breakaway access-denied failure. These measurements are indicative, not controlled performance proof or a promised CI speedup.

The earlier Windows Server 2016 experiment, on test source a083867ca24, ran 691 cases: 687 passed, one job-breakaway failure, three skips; 518.706 test seconds and 586.995 full work-item seconds. All five isolation cases passed with four-thread class parallelism. It used the intact historical x86 Checked/Debug-libraries runtime from build 1621730, source dda144147efa0a3669f11b283253da3ddf0bb9c1, not a current-head runtime build. Its reporting script also encountered an XML naming collision. It was not a green run.

Remaining limits

The local repository build is blocked by the missing shared-framework targeting pack; actual NativeAOT evaluation is blocked by missing ILCompiler targets. Supplemental checks do not establish a successful repository, bundled single-file, or native-platform build. The macOS concurrency problem and Windows job-breakaway failure remain unresolved. This revision restores the old macOS scheduling only; passing macOS Mono/CoreCLR execution still needs confirmation from the new automatic CI run. No manual jobs, new baselines, reruns, or monitoring were started.

Note

This pull request description and changes were generated with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
12 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

[steveisok]

I suspect we'll want to make the sharding infra more generic. Keeping it local to the test suite for now to get reactions ;-).

[jkotas]

632 cases totaling 883 seconds before being killed

What are the top 10 or 20 tests cases that take the most time? The simplest fix may be to skip these tests on Windows x86 CoreCLR checked builds.

Example of prior art:

runtime/src/libraries/System.Runtime.Serialization.Formatters/tests/BinaryFormatterTests.cs

Line 51 in 6f1d933

[SkipOnCoreClr("Takes too long on Checked", ~RuntimeConfiguration.Release)]

[jkotas]

I suspect we'll want to make the sharding infra more generic.

I do not think this we want this sharding infra in the first place. I do not think this is an infrastructure problem. I think it is a problem with the test authoring - tests should not take this long.

[steveisok]

632 cases totaling 883 seconds before being killed

What are the top 10 or 20 tests cases that take the most time? The simplest fix may be to skip these tests on Windows x86 CoreCLR checked builds.

I compared the last timeout run versus the last passing run. My first reaction makes me think this is more infra slowing us down than something in the runtime.

Matched-case measurement	Passing run	Timeout run
Total execution time	470 s	707 s
Same 20 slowest cases, ranked by timeout duration	125 s	160 s
125 cases that shifted from 1–2 s to 3–5 s	228 s	390 s

The timeout run accumulated 238 additional seconds across those matched cases. 68% of that increase came from the 125 ordinary cases shifting from roughly two seconds to roughly three seconds. The 20 slowest cases contributed only 15% of the increase.

So the pattern isn’t primarily “a few exceptionally expensive tests.” It’s many otherwise modest tests getting slower, with the extra time accumulating in a sequential suite.

The fact that this suite is sequential makes me lean towards partitioning than just excluding tests.

[steveisok]

We compared the exact old and new x86 Checked runtime bundles on three Windows Server 2016 Helix machines. Each machine alternated both versions using the same probe for process startup, redirected output, and a four-process chain.

The newer runtime was not consistently slower. Startup differences within each machine ranged from about -5% to +4%. In contrast, one machine was roughly 2.7 times slower with both versions. The other scenarios also showed no consistent newer-runtime penalty.

This points toward host/environment variability for these operations, but does not prove an infrastructure change or rule out a regression specific to the full test suite.

[jkotas]

The fact that this suite is sequential makes me lean towards partitioning than just excluding tests.

https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.Process/tests/AssemblyInfo.cs#L6-L7 says the whole suite is sequential since some tests modify ambient state like the console code page and environment variables. There is only a small number of tests that do this. The standard way to deal with these types of tests is to author them using remote executor so that their modifications do not impact the main process. This is the simplest thing to address to improve the runtime of the System.Diagnostics.Process tests.

The default CI machines run the tests on 4 threads in parallel, so assuming less than ideal parallelization, this should result into ~3x improvement of the test time.

[steveisok]

The default CI machines run the tests on 4 threads in parallel, so assuming less than ideal parallelization, this should result into ~3x improvement of the test time.

OK, I'll give that a try and see what the data shows.

[jkotas]

What is the typical total test execution time improvement that you see with these changes?

[steveisok]

What is the typical total test execution time improvement that you see with these changes?

A little over 9m. That's workable.

I'm fighting the mac legs now for whatever reason. May have to make this exclusive to win-x86 if this churns longer.

[steveisok]

/backport to release/11.0

[github-actions]

Started backporting to release/11.0 (link to workflow run)

[steveisok]

/backport to release/10.0

[github-actions]

Started backporting to release/10.0 (link to workflow run)

[github-actions]

@steveisok backporting to release/10.0 failed, the patch most likely resulted in conflicts. Please backport manually!

git am output

$ git cherry-pick 780c55a27764cb5729babfcfef8de849cdf3fcaf

Auto-merging src/libraries/System.Diagnostics.Process/tests/Interop.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessStandardConsoleTests.cs
CONFLICT (content): Merge conflict in src/libraries/System.Diagnostics.Process/tests/ProcessStandardConsoleTests.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessStartInfoTests.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessTests.Unix.cs
CONFLICT (content): Merge conflict in src/libraries/System.Diagnostics.Process/tests/ProcessTests.Unix.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessTests.Windows.cs
CONFLICT (content): Merge conflict in src/libraries/System.Diagnostics.Process/tests/ProcessTests.Windows.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj
error: could not apply 780c55a2776... Enable class-level parallelism in Process tests (#135160)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"


$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: Shard Windows x86 Process tests into balanced Helix work items
Applying: Enable class-level parallelism in Process tests
Using index info to reconstruct a base tree...
M	src/libraries/System.Diagnostics.Process/README.md
M	src/libraries/System.Diagnostics.Process/tests/Interop.cs
M	src/libraries/System.Diagnostics.Process/tests/ProcessStandardConsoleTests.cs
M	src/libraries/System.Diagnostics.Process/tests/ProcessStartInfoTests.cs
M	src/libraries/System.Diagnostics.Process/tests/ProcessTests.Unix.cs
M	src/libraries/System.Diagnostics.Process/tests/ProcessTests.Windows.cs
M	src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj
Falling back to patching base and 3-way merge...
Auto-merging src/libraries/System.Diagnostics.Process/README.md
CONFLICT (content): Merge conflict in src/libraries/System.Diagnostics.Process/README.md
Auto-merging src/libraries/System.Diagnostics.Process/tests/Interop.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessStandardConsoleTests.cs
CONFLICT (content): Merge conflict in src/libraries/System.Diagnostics.Process/tests/ProcessStandardConsoleTests.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessStartInfoTests.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessTests.Unix.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/ProcessTests.Windows.cs
CONFLICT (content): Merge conflict in src/libraries/System.Diagnostics.Process/tests/ProcessTests.Windows.cs
Auto-merging src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj
CONFLICT (content): Merge conflict in src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0002 Enable class-level parallelism in Process tests
Error: The process '/usr/bin/git' failed with exit code 128

Link to workflow output