Repository navigation
CI: the nightly runs its Lite and Darling PostgreSQL tests in one step each and takes about 50 minutes to publish #5616
Description
Claude posting for Erik Darling.
What happens
The nightly build takes longer each day. The nightly version publishes about 50 minutes after the run starts, and the run ends after about 64 minutes. PR CI runs the same tests in about 19 minutes.
| Run | Date | build job | Darling PostgreSQL tests job | Whole run |
|---|---|---|---|---|
| 37423416866 | 10-06 | 37.1 min | 37.2 min | 38 min |
| 37581130191 | 10-07 | 49.1 min | 41.9 min | 52 min |
| 37746048228 | 10-08 07:51Z | 37.1 min | 38.0 min | 38 min |
| 37772519830 | 10-08 11:48Z | 44.4 min | 36.4 min | 65 min |
| 37815034195 | 10-08 17:14Z | 45.4 min | 46.2 min | 64 min |
In run 37815034195, the publish job finished at 18:04:20Z, 50 minutes after the start. The run stayed open until 18:18Z because of the test map shards.
Cause
Two jobs run their tests in one step each, and publish waits for both of them.
- The
buildjob runs its tests in one step. That step took 41.4 minutes in run 37815034195. - The
darling-pgjob runs all ofDarling.Testsagainst PostgreSQL in one step. That step took 41.1 minutes in the same run. publishneeds[build, linux, darling-pg]. So the slower of the two test steps sets the publish time.
PR CI splits the same suites into shards. In PR run 37869851936, each Darling PostgreSQL shard took 9 to 11 minutes, and the whole run took 19 minutes. Each new test class adds to the nightly's two single steps. It adds only a part of that time to a PR shard.
The test-map-shard job (10 shards, 10 to 17 minutes each) starts after build and darling-pg. It does not delay publish, but it keeps the run open and uses 10 more runners.
What to fix
- Before you change the job, read the history of
.github/workflows/nightly.yml. Find out if the single test step is on purpose. If one process catches failures that shards miss, write that reason in this issue. - Split the nightly's Lite tests and Darling PostgreSQL tests into shards. Use the same shard lists and scripts that PR CI uses, so the two paths cannot drift apart.
- Keep
publishdependent on every test shard. A failed shard must stop the publish. - Put two times in the PR, before and after the change. The first is the time from the run start to the end of
publish. The second is the time of the whole run.
#5208 covers the slow Lite test classes. #5459 owns the test map shards. This issue covers only the nightly's job layout.
Activity
erikdarlingdata commentedon Oct 9, 2026
Claude posting for Erik Darling.
Before this changes, here is why the nightly runs each test suite in one step.
It was a choice for releases, and the nightly kept it
build.ymlas one unsharded step, "so the artifact-producing run still executes the suite whole, in one log." It did not changenightly.yml.What the reason protects, and what the change keeps
build.yml.So the change goes ahead: the nightly uses the same shard lists and scripts as PR CI, and the publish job waits for every shard.
CI: path filters, slow-skip build inputs and publish text now cover t…
Nightly: the Lite and Darling PostgreSQL tests run in PR CI's shards,…
Nightly: every job checks out the commit the run was dispatched at (#…
erikdarlingdata commentedon Oct 9, 2026
Fixed on dev by #5620 (18c8fac) and #5623 (2ab9e52). The nightly runs its Lite and Darling PostgreSQL tests in the same shards as pull request CI, and every nightly job checks out the commit the run was dispatched at, so a merge to dev during a nightly no longer changes what a later job builds. Claude posting for Erik Darling.