Repository navigation
test: stop the AOT [EngineTest=Pass] run hitting its 5 minute budget - #7125
Conversation
The ExpectedStateTests.Pass AOT run on Windows CI failed with what looked like a NotInParallelLock deadlock: every remaining global [NotInParallel] test logged a TaskCanceledException from EnterExclusiveAsync. It was not a deadlock. The AOT invocation passes --timeout 5m, a whole-session budget, and healthy runs take 3.5-5 minutes on Windows CI. When the budget ran out, the sequential global NotInParallel phase (which runs last) hit the cancelled token at the writer gate for each remaining test. The JIT run in the same job took 5m15s and passed because it has no session timeout. - Raise the AOT session budget to 10 minutes. - DedicatedThreadExecutorCleanUpOrderTests (#6885) took about 16 seconds per test: the executor also runs the project's ~50 global test hooks, each with the 300 ms CleanUp sleep, all on the serial NotInParallel path. Only the test body's CleanUp now sleeps, so the class takes about 4 seconds instead of 52 and the full Pass run drops from about 2.5-3 minutes to under 2 locally. This also makes the regression sharper, since the hooks no longer put seconds between consecutive test bodies. Co-Authored-By: Claude <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
ReviewTest-only change, and it looks good. The diagnosis in the description is convincing: the AOT
Suggestions (non-blocking):
Approving from a code standpoint. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a798693c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
[Low impact] The PR appears safe to merge. SummaryRaises the AOT test-session budget and removes unnecessary delays from the cleanup-order regression tests.
Reviews (1) · Last reviewed commit: "test: stop the AOT [EngineTest=Pass] run..." · Reviewed by Greptile |
Updated [TUnit](https://github.com/thomhurst/TUnit) from 1.75.0 to 1.76.0. <details> <summary>Release notes</summary> _Sourced from [TUnit's releases](https://github.com/thomhurst/TUnit/releases)._ ## 1.76.0 <!-- Release notes generated using configuration in .github/release.yml at v1.76.0 --> ## What's Changed ### Other Changes * chore: add OSS scanner config, threat model and SECURITY.md by @thomhurst in thomhurst/TUnit#7080 * fix(deps): patch MessagePack for Aspire and use per-TFM Microsoft.Extensions.Http by @Quattordici in thomhurst/TUnit#7072 * fix: skip tests marked with an empty skip reason by @thomhurst in thomhurst/TUnit#7083 * fix(engine): report the engine mode actually selected in the banner by @thomhurst in thomhurst/TUnit#7085 * fix(engine): order keyed NotInParallel tests by priority, not hash code by @thomhurst in thomhurst/TUnit#7090 * fix: remove root causes of flaky tests and the broken Apple smoke jobs by @thomhurst in thomhurst/TUnit#7095 * fix(engine): run synchronous keyed NotInParallel tests concurrently by @thomhurst in thomhurst/TUnit#7087 * fix(sourcegen): run every [Arguments] row that infers the same generic type by @thomhurst in thomhurst/TUnit#7100 * fix: same filter path and display name for generic tests in both engine modes by @thomhurst in thomhurst/TUnit#7110 * fix(engine): apply max parallelism cap to keyed NotInParallel tests by @thomhurst in thomhurst/TUnit#7084 * fix(ci): install only Chromium for Playwright and wait on dpkg lock by @thomhurst in thomhurst/TUnit#6994 * fix(core): keep the engine-selected execution mode when late assemblies load by @thomhurst in thomhurst/TUnit#7091 * fix(engine): end fail-fast runs without crashing the test host by @thomhurst in thomhurst/TUnit#7115 * test: stop the AOT [EngineTest=Pass] run hitting its 5 minute budget by @thomhurst in thomhurst/TUnit#7125 * fix(engine): align reflection-mode data sources and hooks with source-gen by @thomhurst in thomhurst/TUnit#7094 * fix(pipeline): stop skipping the NuGet package tester modules by @thomhurst in thomhurst/TUnit#7127 * ci: take AOT publishes and doc snippets off the Ubuntu critical path by @thomhurst in thomhurst/TUnit#7128 * fix(analyzers): TUnit0001 false positive for MethodDataSource on generic type-parameter parameters by @thomhurst in thomhurst/TUnit#7109 * fix: report generic class constraint violations as failed tests in both modes by @thomhurst in thomhurst/TUnit#7112 * ci: cut runner contention and the AOT publish off the critical path by @thomhurst in thomhurst/TUnit#7131 * fix(assertions): make cross-type IsEqualTo work for numeric and nullable arguments on C# 12 by @thomhurst in thomhurst/TUnit#7086 * fix(engine): substitute class type arguments and array element types in build-time constraint checks by @thomhurst in thomhurst/TUnit#7102 * ci: test the NuGet packages on Windows in their own job by @thomhurst in thomhurst/TUnit#7130 * fix(sourcegen): bind params T[] on generic test methods by @thomhurst in thomhurst/TUnit#7101 * fix(engine): run every [Arguments] row on generic test classes in both modes by @thomhurst in thomhurst/TUnit#7111 * fix(engine): use last inferred type for repeated generic parameters by @theluckystrike in thomhurst/TUnit#7134 * Make the test suite pass in every environment and run fully in CI by @Quattordici in thomhurst/TUnit#7129 * fix(assertions): skip using directive for global-namespace assertion types by @thomhurst in thomhurst/TUnit#7139 * fix(core): format numeric display-name arguments with the invariant culture by @thomhurst in thomhurst/TUnit#7138 * fix(engine): dispose data-source objects of rows that end as failed tests by @thomhurst in thomhurst/TUnit#7150 * Fix net472 compile gate failure in assertions source generator tests by @thomhurst in thomhurst/TUnit#7170 * fix(sourcegen): close generic methods of generic classes on every class source by @thomhurst in thomhurst/TUnit#7152 * fix(engine): infer object for null rows on generic test methods in reflection mode by @thomhurst in thomhurst/TUnit#7142 * fix(engine): report every test once on fail-fast runs by @thomhurst in thomhurst/TUnit#7145 * test(engine): stop whole-process budgets killing healthy scheduling engine tests by @thomhurst in thomhurst/TUnit#7177 * feat(analyzers): report MethodDataSource Arguments that fit two overloads equally (TUnit0076) by @thomhurst in thomhurst/TUnit#7181 * fix(sourcegen): render top-level dynamic as object so typeof compiles by @thomhurst in thomhurst/TUnit#7182 * fix(sourcegen): pair each GenerateGenericTest type only with its typed ClassDataSource<T> by @thomhurst in thomhurst/TUnit#7188 * fix(engine): run untyped [Arguments(null)] rows on generic classes instead of dropping them by @thomhurst in thomhurst/TUnit#7203 * fix(engine): report generic test classes by their generic type definition in TRX by @thomhurst in thomhurst/TUnit#7200 * test(sourcegen): parse snapshot fixtures as a real build does by @thomhurst in thomhurst/TUnit#7189 * refactor(engine): dedupe reflection generic discovery by @thomhurst in thomhurst/TUnit#7183 * test: make TimeoutHelperTests deterministic under thread-pool load by @thomhurst in thomhurst/TUnit#7180 * fix(analyzers): accept null MethodDataSource arguments for nullable parameters by @thomhurst in thomhurst/TUnit#7179 * test(assertions): make WaitsFor cancellation tests robust to slow CI runners by @thomhurst in thomhurst/TUnit#7176 * fix(assertions): support [AssertionFrom] with unbound generic types by @thomhurst in thomhurst/TUnit#7144 ... (truncated) Commits viewable in [compare view](thomhurst/TUnit@v1.75.0...v1.76.0). </details> Updated [TUnit.Core](https://github.com/thomhurst/TUnit) from 1.75.0 to 1.76.0. <details> <summary>Release notes</summary> _Sourced from [TUnit.Core's releases](https://github.com/thomhurst/TUnit/releases)._ ## 1.76.0 <!-- Release notes generated using configuration in .github/release.yml at v1.76.0 --> ## What's Changed ### Other Changes * chore: add OSS scanner config, threat model and SECURITY.md by @thomhurst in thomhurst/TUnit#7080 * fix(deps): patch MessagePack for Aspire and use per-TFM Microsoft.Extensions.Http by @Quattordici in thomhurst/TUnit#7072 * fix: skip tests marked with an empty skip reason by @thomhurst in thomhurst/TUnit#7083 * fix(engine): report the engine mode actually selected in the banner by @thomhurst in thomhurst/TUnit#7085 * fix(engine): order keyed NotInParallel tests by priority, not hash code by @thomhurst in thomhurst/TUnit#7090 * fix: remove root causes of flaky tests and the broken Apple smoke jobs by @thomhurst in thomhurst/TUnit#7095 * fix(engine): run synchronous keyed NotInParallel tests concurrently by @thomhurst in thomhurst/TUnit#7087 * fix(sourcegen): run every [Arguments] row that infers the same generic type by @thomhurst in thomhurst/TUnit#7100 * fix: same filter path and display name for generic tests in both engine modes by @thomhurst in thomhurst/TUnit#7110 * fix(engine): apply max parallelism cap to keyed NotInParallel tests by @thomhurst in thomhurst/TUnit#7084 * fix(ci): install only Chromium for Playwright and wait on dpkg lock by @thomhurst in thomhurst/TUnit#6994 * fix(core): keep the engine-selected execution mode when late assemblies load by @thomhurst in thomhurst/TUnit#7091 * fix(engine): end fail-fast runs without crashing the test host by @thomhurst in thomhurst/TUnit#7115 * test: stop the AOT [EngineTest=Pass] run hitting its 5 minute budget by @thomhurst in thomhurst/TUnit#7125 * fix(engine): align reflection-mode data sources and hooks with source-gen by @thomhurst in thomhurst/TUnit#7094 * fix(pipeline): stop skipping the NuGet package tester modules by @thomhurst in thomhurst/TUnit#7127 * ci: take AOT publishes and doc snippets off the Ubuntu critical path by @thomhurst in thomhurst/TUnit#7128 * fix(analyzers): TUnit0001 false positive for MethodDataSource on generic type-parameter parameters by @thomhurst in thomhurst/TUnit#7109 * fix: report generic class constraint violations as failed tests in both modes by @thomhurst in thomhurst/TUnit#7112 * ci: cut runner contention and the AOT publish off the critical path by @thomhurst in thomhurst/TUnit#7131 * fix(assertions): make cross-type IsEqualTo work for numeric and nullable arguments on C# 12 by @thomhurst in thomhurst/TUnit#7086 * fix(engine): substitute class type arguments and array element types in build-time constraint checks by @thomhurst in thomhurst/TUnit#7102 * ci: test the NuGet packages on Windows in their own job by @thomhurst in thomhurst/TUnit#7130 * fix(sourcegen): bind params T[] on generic test methods by @thomhurst in thomhurst/TUnit#7101 * fix(engine): run every [Arguments] row on generic test classes in both modes by @thomhurst in thomhurst/TUnit#7111 * fix(engine): use last inferred type for repeated generic parameters by @theluckystrike in thomhurst/TUnit#7134 * Make the test suite pass in every environment and run fully in CI by @Quattordici in thomhurst/TUnit#7129 * fix(assertions): skip using directive for global-namespace assertion types by @thomhurst in thomhurst/TUnit#7139 * fix(core): format numeric display-name arguments with the invariant culture by @thomhurst in thomhurst/TUnit#7138 * fix(engine): dispose data-source objects of rows that end as failed tests by @thomhurst in thomhurst/TUnit#7150 * Fix net472 compile gate failure in assertions source generator tests by @thomhurst in thomhurst/TUnit#7170 * fix(sourcegen): close generic methods of generic classes on every class source by @thomhurst in thomhurst/TUnit#7152 * fix(engine): infer object for null rows on generic test methods in reflection mode by @thomhurst in thomhurst/TUnit#7142 * fix(engine): report every test once on fail-fast runs by @thomhurst in thomhurst/TUnit#7145 * test(engine): stop whole-process budgets killing healthy scheduling engine tests by @thomhurst in thomhurst/TUnit#7177 * feat(analyzers): report MethodDataSource Arguments that fit two overloads equally (TUnit0076) by @thomhurst in thomhurst/TUnit#7181 * fix(sourcegen): render top-level dynamic as object so typeof compiles by @thomhurst in thomhurst/TUnit#7182 * fix(sourcegen): pair each GenerateGenericTest type only with its typed ClassDataSource<T> by @thomhurst in thomhurst/TUnit#7188 * fix(engine): run untyped [Arguments(null)] rows on generic classes instead of dropping them by @thomhurst in thomhurst/TUnit#7203 * fix(engine): report generic test classes by their generic type definition in TRX by @thomhurst in thomhurst/TUnit#7200 * test(sourcegen): parse snapshot fixtures as a real build does by @thomhurst in thomhurst/TUnit#7189 * refactor(engine): dedupe reflection generic discovery by @thomhurst in thomhurst/TUnit#7183 * test: make TimeoutHelperTests deterministic under thread-pool load by @thomhurst in thomhurst/TUnit#7180 * fix(analyzers): accept null MethodDataSource arguments for nullable parameters by @thomhurst in thomhurst/TUnit#7179 * test(assertions): make WaitsFor cancellation tests robust to slow CI runners by @thomhurst in thomhurst/TUnit#7176 * fix(assertions): support [AssertionFrom] with unbound generic types by @thomhurst in thomhurst/TUnit#7144 ... (truncated) Commits viewable in [compare view](thomhurst/TUnit@v1.75.0...v1.76.0). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Updated the managed TUnit package versions to 1.76.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
What happened
CI run 39128089083 (PR #7086) failed
ExpectedStateTests.Passin AOT mode on Windows. The output looked like a deadlock: every remaining global[NotInParallel]test (NotInParallelTests.*,TestBuildContextOutputCaptureTests.*) logged aTaskCanceledExceptionfromNotInParallelLock.EnterExclusiveAsync.It is not a deadlock. The run hit its time limit.
InvokableTestBasepasses--timeout 5m. That limit covers the whole session, and healthy[EngineTest=Pass]AOT runs on Windows CI take 3m35s to 4m56s. The diagnostics artifacts of four recent green runs show this.--hangdump-timeout).[NotInParallel]bucket runs last, one test at a time. When the token is cancelled, each remaining test fails atawait _writerGate.WaitAsync(token)(NotInParallelLock.cs line 89). That is the stack in the CI log. Nothing was holding the lock.failed: 0. Across about a dozen uncontended local runs (source-gen and reflection, with default parallelism and withDOTNET_PROCESSOR_COUNT=4), nothing hung.mainon 2026-09-29 (run 36630664644, at db75285). So it predates fix(engine): apply max parallelism cap to keyed NotInParallel tests #7084 and fix(engine): run synchronous keyed NotInParallel tests concurrently #7087, and neither caused it.Changes
DedicatedThreadExecutorCleanUpOrderTests([Bug]: DedicatedThreadExecutor completes the test before CleanUp() runs, so the next test starts while CleanUp() is still running #6885) took about 16 seconds per test, on the serial[NotInParallel]path. The executor also runs every Before/After(Test) hook, and the test project has about 50 global[BeforeEvery]/[AfterEvery]test hooks. Each hook ran on its own thread with the fixture's 300 msCleanUp()sleep. Now only the test body'sCleanUp()sleeps: the body sets a thread-static flag, and the body runs on the executor's dedicated thread. The class drops from 52s to about 4s, and the full Pass run drops from 2.5-3 minutes to 1m51s locally. The regression check also gets sharper, because the hooks no longer put seconds between consecutive test bodies.No engine or scheduling code changes, so this does not interact with #7115.
Follow-up worth considering (not in this PR)
When the session is cancelled, the sequential phase still walks every remaining global
[NotInParallel]test and logs each one as "Unhandled exception in test ...". That output is what made this look like a lock bug. Reporting them as cancelled, or stopping the loop, would give a clearer diagnosis. This is a scheduler change, so it would need to be coordinated with #7115.Tests
DedicatedThreadExecutorCleanUpOrderTestsin source-gen and reflection modes: 3/3 passed in each (4.7s and 3.8s, previously 52s).Issue6885Testsengine test (reflection mode; AOT is CI-only): passed./**[EngineTest=Pass]source-gen run: 6001 total, 0 failed, 1m51s.Summary by CodeRabbit