Skip to content

ci: stop heavy modules overlapping the AOT test-app publish - #7217

Merged
thomhurst merged 2 commits into
mainfrom
ci/macos-aot-publish-contention
Oct 20, 2026
Merged

thomhurst merged 2 commits into
mainfrom
ci/macos-aot-publish-contention

Conversation

@thomhurst

@thomhurst thomhurst commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Problem

Since #7131, PublishTestProjectAOTModule runs at the same time as the reflection-mode engine tests, packing, the RPC tests, the NuGet package tests and PublishNugetTesterAOTModule. That last module runs three more native AOT (ILC) publishes. On every OS this contention slowed the publish, and the publish is the critical path because RunEngineAotTestsModule waits for it. The whole pipeline step got slower:

Pipeline step (first module start to last module end) Before #7131 (runs 39156626077, 39156219240) After (runs 39400274604, 39389278905; macOS job 118552950835)
ubuntu: publish 3.5 min 13.5–18.5 min
ubuntu: pipeline 8.9–9.3 min 15.3–20.4 min
windows: publish 4.3–7.1 min 19 min
windows: pipeline 10.9–19.8 min 22.3–22.8 min
macos: publish 7.0–11.2 min 24.5 min
macos: pipeline 13–18 min 26 min

On macOS the publish now runs close to the default 30-minute module timeout and hit it on 6 PR runs on 2026-10-20. #7216 raises that timeout as a backstop. This PR deals with the contention that causes it.

#7129, which made the suite run fully in CI, landed in the same window and added test modules. That accounts for some of the slowdown, but not a 4–5× slower publish.

Change

On every OS, RunEngineTestsModule and PublishNugetTesterAOTModule declare [DependsOn<PublishTestProjectAOTModule>] again, as they did before #7131:

  • PublishNugetTesterAOTModule no longer runs native AOT compiles at the same time as the main publish.
  • The reflection-mode engine tests, which start many TestProject processes, run after the publish instead of during it.

The Windows NuGet job (--categories Packaging NuGetPackageTests) does not run the publish. There ModularPipelines skips the module, a skipped module counts as complete, and PublishNugetTesterAOTModule does not wait.

Packing and the RPC tests still overlap the publish, because the isolated artifacts tree from #7131 removed the reason they waited. They are the next candidates if the timings stay poor.

Validation

  • TUnit.Pipeline builds.
  • This PR's own modularpipeline jobs are the real test. I will compare their module timings with the table above.

#7131 let the reflection-mode engine tests and the NuGet tester AOT
publishes run alongside PublishTestProjectAOTModule. On the 3-core,
7 GB macOS runner that contention stretched the publish from 7 to
24-27 minutes, and the macOS pipeline from about 13 to about 26 minutes.
It also tripped the 30-minute module timeout.

On macOS only, both modules now wait for the publish again. Linux and
Windows keep the overlap.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T11:17:03.351256Z ab078ec PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 20, 2026

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8d6f1702-b2d8-48fd-b430-e8a0589040ce
📥 Commits

Reviewing files that changed from the base of the PR and between 5523113 and ab078ec.

📒 Files selected for processing: 2
  • tools/TUnit.Pipeline/Modules/PublishNugetTesterAOTModule.cs
  • tools/TUnit.Pipeline/Modules/RunEngineTestsModule.cs
+-----------------------------------------------+
| Optimization hinders evolution. - Alan Perlis |
+-----------------------------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Review

The change is small and targeted. I found no blocking issues.

  • On macOS only, RunEngineTestsModule and PublishNugetTesterAOTModule now declare an optional dependency on PublishTestProjectAOTModule. That serialises the CPU-heavy work that was starving the 3-core runner, and it matches the diagnosis in the PR description. The dependency is optional, so it can't fail the pipeline when the publish is skipped.
  • Linux and Windows are unchanged.

Things to check or consider:

  1. New API in this repo. Neither DeclareDependencies nor DependsOnOptional is used anywhere else here; every other module uses [DependsOn<>] attributes. The PR says TUnit.Pipeline builds, and I didn't build it myself. A green macOS CI run is the real check that the declarations are honoured at runtime and don't cause a cycle or ordering surprise. I'd compare the module timings against the table in the description before merging.
  2. Platform logic is split across two mechanisms. The attributes carry the unconditional dependencies and DeclareDependencies carries the conditional one. The comments explain why, and this is fine for two modules. If a third module needs the same macOS gating, extract a small shared helper (for example an AotPublishOrdering.OrderAfterPublishOnMacOS(deps)) so the rule and its rationale live in one place.
  3. Follow-up. The description already names packing and the RPC tests as the next candidates if macOS timings stay poor. A tracking issue would help, so that follow-up doesn't depend on someone remembering this PR.

Overall this looks good to merge once the macOS job confirms the timings.

The overlap from #7131 slowed the publish on Linux and Windows too. On
ubuntu it went from 3.5 to 13.5-18.5 minutes and the pipeline from
about 9 to 15-20 minutes. The publish is the critical path everywhere,
so the reflection-mode engine tests and the NuGet tester AOT publishes
now wait for it on every OS. Static DependsOn attributes replace the
macOS-only DeclareDependencies overrides.
@thomhurst thomhurst changed the title ci: stop heavy modules overlapping the AOT test-app publish on macOS ci: stop heavy modules overlapping the AOT test-app publish Oct 20, 2026
@thomhurst
thomhurst deployed to Pull Requests October 20, 2026 11:18 — with GitHub Actions Active
@thomhurst
thomhurst merged commit 8e51148 into main Oct 20, 2026
15 of 18 checks passed
@thomhurst
thomhurst deleted the ci/macos-aot-publish-contention branch October 20, 2026 11:18
@github-actions

Copy link
Copy Markdown
Contributor

Review

Small, targeted change. I found no blocking issues.

  • No cycle. PublishTestProjectAOTModule declares no DependsOn of its own, so adding it as a dependency of RunEngineTestsModule and PublishNugetTesterAOTModule can't create one. RunEngineAotTestsModule already depends on it the same way.
  • Windows NuGet job. The skipped-counts-as-complete reasoning in the comment matches how the other conditionally-skipped modules behave. The Windows NuGet job's own timings will confirm it.
  • Earlier review comment. The earlier summary describes a macOS-only DeclareDependencies/DependsOnOptional version. The current diff has plain [DependsOn<>] attributes on all OSes, so those points don't apply.

Considerations:

  1. This trades parallelism for a shorter critical path. Total pipeline time on Linux could still rise if the publish itself is fast, since the reflection-mode engine tests now start later. The before/after table suggests that's acceptable. Compare module timings from this PR's runs, as the description says.
  2. If timings stay poor, gating packing and the RPC tests behind the publish is the next step. A tracking issue would keep that from being forgotten.

Looks good to merge once the modularpipeline jobs confirm the timings.

@greptile-apps

greptile-apps Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable defects were found in the two added dependencies.

Summary

Makes RunEngineTestsModule and PublishNugetTesterAOTModule wait for PublishTestProjectAOTModule to finish, reducing competing work during the main AOT publish.

  • The main AOT publish finishes before reflection tests and NuGet AOT builds start.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    P[PublishTestProjectAOTModule] -->|finishes before| E[RunEngineTestsModule]
    P -->|finishes before| N[PublishNugetTesterAOTModule]
Loading

Reviews (1) · Last reviewed commit: "ci: make every OS wait for the AOT test-..." · Reviewed by Greptile

thomhurst added a commit that referenced this pull request Oct 20, 2026
…ish optional (#7219)

#7217 made PublishNugetTesterAOTModule depend on PublishTestProjectAOTModule.
The Windows NuGet job runs only the Packaging and NuGetPackageTests
categories, so PublishTestProjectAOTModule is not registered there and the
pipeline failed with ModuleNotRegisteredException. An optional dependency
still waits where the module runs and is ignored where it does not.

This branch was successfully deployed

1 active deployment
Pull Requests — adc34d63 Deployed Oct 20, 2026 by thomhurst via modularpipeline (ubuntu-latest) #20338
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant