Repository navigation
ci: take chaos and perf off the self-hosted box, stop that lane discarding jobs, and run the mutation sweep nightly - #366
Conversation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned Files
|
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
No new clones introduced by this PR. ✅ jscpd (language-agnostic)
No new clones introduced by this PR. Powered by astubbs/duplicate-code-cross-check |
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its |
|
astubbs
changed the title
365 is the fix for something this branch hit twice today: a gate was truncating the shared clone's history on every sweep, so `git merge-base` answered exit 1 where it had answered a commit an hour earlier, and half the gates would have judged a truncated graft while reporting green. Taking it here stops that recurring mid-merge. 366 brings the hosted-chaos trial and, with it, #351 - which diagnosed `OffsetEncodingBackPressureTest` (the 4/45 flake, the most frequent tracked one) as a test that froze the offset it then asserted. FOUR CONFLICTS, and the two that needed a decision: - docs/quarantined-tests.md - 366 removes the back-pressure entry because #351 released it. Taken. But the PCMetricsTest entry that stays cited it as "the entry above" for the shared shape of their failures, so that cross-reference is repaired rather than left pointing at nothing: the rhyme is now stated in the past tense and names #351 as the first place to look for whether the two are one phenomenon or two. Registry re-verified, 2 entries. - docs/inflight/test-untracked-ci-flakes.md - 366 retires the whole "NOT diagnosed - quarantined anyway" section. Taken whole rather than re-adding a section its author deliberately removed, after confirming the durable part - why quarantine rule 1 changed - survives in the solutions write-up that replaced it. - docs/ci.md and maven.yml - both sides edited the same paragraph and the same matrix. Kept both edits: the proto-breaking gate this branch gained from master stays listed beside 366's hosted-chaos job, and `continue-on-error` now honours both `advisory` (the execution-mode axis) and `optional` (366's experiment). Two names for one meaning, deliberately not unified - renaming either would edit the other branch's vocabulary and re-conflict when it lands. Also fixes two SpotBugs findings the merge surfaced in SessionEndTest: an ignored `CountDownLatch.await(timeout)` return in each of two user functions. A discarded await cannot be told from a call whose result nobody needs, and false there means the release never came inside the budget. Named per the repo convention rather than thrown from, since throwing on that thread would fail the record for a reason the test is not measuring. Verified on the merged tree: full reactor test-compile green, bin/check-all.sh 18/18 with only check-shell-lint CANNOT (shellcheck is not installed on this box). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019HwDcZdfv1Ne4u3g6ThrG9
astubbs
changed the title
astubbs
changed the title
astubbs
changed the title
|
@claude review this Focus areas, since this PR is all CI scheduling: the risks are in what stops running, not in any code.
Deliberate and known: |
|
Claude finished @astubbs's task in 5m 35s —— View job Review complete
Reviewed by reading the actual workflow YAML at this branch's HEAD ( Finding: the nightly mutation-sweep exception was never actually written into
|
astubbs
left a comment
astubbs
left a comment
There was a problem hiding this comment.
Just waiting on the mutation testing outcome to see what we do with the full mutation pass.
Nothing triggered by a pull request touches the `[self-hosted, highcpu]` runner any more, so an offline box cannot leave a check pending on anybody. The per-PR lane is deleted outright; what remains there is dispatch- and schedule-only. THE MEASUREMENT THAT STARTED IT. Every `box-exclusive` matrix entry sat in a concurrency group keyed on the LITERAL CONSTANT `highcpu-box-exclusive` with `cancel-in-progress: false`. No ref in the key means one repo-wide queue, and GitHub holds one run in progress plus AT MOST ONE pending - so a third arrival evicts whoever was waiting. Measured over 16 runs across 9 branches: 26 of 32 box jobs never executed a single step, while five of six runners sat idle. Classified by `steps == 0`, which separates evicted-while-pending from cancelled-mid-run; the raw `conclusion=cancelled` count cannot tell those apart, and `gh pr checks` renders both as **fail**, so an absent measurement wore the costume of a failing one. `docs/inflight/ci-chaos-lane-serialised-confirm-no-coresidency.md` predicted this exactly - "if it is not zero, the group key is still wrong, reopen with the counts" - and is deleted here, answered. THE PREMISE DID NOT SURVIVE MEASUREMENT, which is why this goes further than fixing the key. Chaos was on the box because it supposedly needs many real cores. Trialled head-to-head on the SAME commit, both green: hosted 13m59s, self-hosted 12m16s. 14% of wall-clock does not buy a shared physical machine, and the sharing is what produced every scheduling problem this lane has had - a hosted runner gives each job its own VM, so co-residency cannot occur at all. Corroborating it: `bin/chaos-test.sh` needed NO change to move, because it passes no forkCount and no `-Dparallel-tests`, so the suite was never configured to exploit those cores. Chaos stays GATING in its new home. A chaos RED is a real finding, and the suite it replaces was not advisory either. Making the check REQUIRED is a separate act on the master ruleset and must wait until the job exists on master, or every PR sits pending on a context no run produces. Performance needed no trial: it was ALREADY running hosted as the required `Performance Tests` check, so the self-hosted copy was a non-gating duplicate of a gating one - the same test that removed Unit and Integration from this lane. A REGRESSION CAUGHT WHILE DELETING IT. The deleted workflow was uploading the failsafe XML artifact; `maven.yml`'s chaos entry only uploaded Codecov coverage. Moving chaos would have silently dropped the test reports that `docs/inflight/bug-857-family.md` and the log-truncation write-up rely on for diagnosis. Restored as a chaos-only upload step. Sixteen files referenced the deleted workflow. Dated `docs/plans/` records keep their original wording behind `file-refs: N/A` markers naming the recovery commit, per `docs/citations.md`; live docs are repointed. The workflows README loses its row entirely rather than describing a file that is gone - it had been left saying "dispatch only" by an earlier revision of this branch, which the deletion two commits later never caught up with, and no gate sees it because a backticked name in a table cell is not a link. What the box still carries is dispatch- and schedule-only, and whether it earns its keep at all is left open in `docs/inflight/ci-highcpu-box-concurrency-is-runner-count.md` rather than settled here. The concurrency group goes from the remaining workflows too: a nightly sweep and a dispatched chaos hunt could still evict each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The whole-repo PIT sweep was `workflow_dispatch` only, and had been dispatched once in three weeks. A lane nobody triggers measures nothing, and its own open note recorded a mutation score that no run has re-tested since. WHY NIGHTLY RATHER THAN PER-MERGE. A sweep is only useful if it finishes, and a push to master while one is running kills it. So the question is how long a push has to arrive within - and 83% of the last 59 master-commit gaps are under this lane's job-elapsed runtime. Per-merge would abort most of the time at this repo's merge rate; a nightly run collides with nothing. THE FIGURE THAT ARGUMENT RESTS ON WAS MISLABELLED, AND IS CORRECTED HERE FIRST. `docs/inflight/ci-mutation-testing.md` recorded 21m55s under the heading "Wall clock". Its own neighbouring line disproves that - the coverage pass is "311s of 1315s", and 1315s IS 21m55s - so the row was always the PIT phase, not the job. The single prior sweep actually ran: run wall, including ~9m22s queueing 40m50s JOB ELAPSED 31m27s PIT/maven phase (`time ./mvnw`) 21m55s Three different readings of one run, all quoted as if interchangeable, because none said which clock it was on. Corrected with the table, and the note now says to use JOB ELAPSED for scheduling questions, since that is what a push must outrun. Recomputed properly the nightly argument gets stronger, not weaker: 83% under 31m27s, where the mislabelled figure gave 80% under ~22 min. `n=1` is stated explicitly. One observation supports order-of-magnitude and nothing tighter, and the mutation SCORE is the sturdier half of that baseline because it does not depend on which clock anyone picked up. A HOSTED TRIAL ARM runs alongside, the same experiment that settled the chaos question in the previous commit: if the sweep does not need the box either, the box has nothing per-PR and nothing scheduled left to justify it. The hosted arm has fewer cores, so extrapolate from job-elapsed on both sides - this baseline has already been misread three times by mixing clocks, and a slow hosted result read against the PIT-phase number would look like a failure that is not one. Also fixes empty `inputs.*` on a non-dispatch trigger: `schedule` and `push` supply no inputs, so every input needed a fallback matching its dispatch default or the sweep would have run with empty arguments. The docs said "there is no scheduled build, deliberately". That rule now has an exception and says so, rather than being quietly contradicted by a workflow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5d47791 to
544b1c7
Compare
…mplate whether the review skills ran Both are process rules with no gate behind them, so they go where every agent session reads them rather than into a topic doc. NEVER PUT PROSE IN A SHELL STRING. This branch corrupted a markdown table with a `|` used as a perl delimiter, truncated a commit message on an apostrophe, and executed backticks out of an unquoted heredoc - three times, in one branch, after the trap was already documented. The reason it kept recurring is the part worth writing down: the breakage is in the FISH layer above bash, so quoting the heredoc body does not save you, and every fix aimed at the bash layer left the real cause untouched. Write prose to a file and pass the file. ASK WHETHER `ce-simplify` AND `ce-code-review` WERE RUN, and accept a reasoned no. The PR template gets the question rather than a requirement, because a documentation-only change legitimately stops `ce-simplify` at its preflight and a box that can only be ticked invites ticking it. A stated "no, because X" is worth more than an unexamined tick, and the checklist gate accepts an `N/A` with a reason for exactly that shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
544b1c7 to
adec55a
Compare
…ely are not (#368) Three ruleset edits were made on 2026-08-26 - `Chaos Pain Suite` promoted once it reached master, `static: infer` promoted, and a bare `spotbugs` context removed. Those are repository SETTINGS, so no commit can carry them and nothing goes red when the list drifts. What a commit can carry is the reasoning, which is the half that would otherwise be re-derived. WHY THE REASONING IS THE DURABLE PART. The list of required contexts is a command away and is deliberately not written down here. What no command answers is why four checks are deliberately NOT required, each of which looks like an obvious promotion until you try it: - `Mutation Tests (PIT, PR-scoped)` is `continue-on-error: true`, so its check-run CONCLUSION is success even when the step fails. The row reddens; a required check reads the conclusion. Requiring it today gates nothing at all. - `Performance (optional)` is on a dispatch-only lane, so the context never appears on a PR - requiring it would block every PR permanently. - `compat: kafka 4.x (experimental)` is `if: false`. - `full build (master)` is push-only. - The three `Analyze (...)` jobs are already covered by the required `CodeQL` aggregate. TWO ORDERING TRAPS, recorded because both have already happened here. A required context that no run produces leaves every PR pending until it merges master, which is why a check is promoted only after its job is on master. And the ruleset keeps the OLD name when a job is renamed: that is how a bare `spotbugs` outlived the job that became `static: spotbugs`, sitting required with no producer - blocking nothing visibly, and passing never. A THIRD, NEWLY RELEVANT: a skip does not satisfy a required check, it waits. So a job that can legitimately have nothing in scope should report success rather than skip before anyone requires it. `static: infer` has skipped on some runs and is now required, so that is a live risk rather than a hypothetical one. `Chaos Pain Suite` was promoted WITHOUT the bake-in period the retired note advised, deliberately and on the owner's call: a red chaos check is a real finding and will be fixed as one. Written down because the advice and the decision disagree, and a future reader finding it red should investigate the bug rather than assume the gate was promoted too early. This retires `docs/inflight/ci-required-status-checks-need-updating.md`, whose three edits are now made. That note never reached master - it was excluded from #366 because it described work that PR was landing, which docs/inflight/AGENTS.md forbids, and it then existed only as an untracked file behind a local `.git/info/exclude` entry. One `rm -rf` of a worktree would have destroyed the only copy. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing else asks it. The tracker is not in the diff, so a PR that finishes an issue without saying so leaves it open for someone to rediscover, and the checklist had no line covering that. THE HALF THAT MATTERS IS THE PARTIAL CASE, because closing on a partial fix is where the damage is - the remaining work goes with it. #41 is the worked example and the reason this rule exists: its reopening comment lists four remaining items, #366 did the first, and "366 closes 41" was a reasonable-sounding read that would have taken the other three - widening the mutation target, verifying excludedGroups, and the licence-blocked history plugin - with it. That issue also shows the second half: #366 did item one AGAINST the issue's stated preference, refuting "prefer push-on-master over a cron" with a measurement of this repo's actual merge rate. The recommendation lives on the issue and the evidence does not, so the disagreement has to be recorded there or the next reader re-derives it. The closing keyword needs the fully qualified form - Fixes #167 closes nothing - which docs/issue-references.md already owns and this only points at. docs/inflight/issue-index.md, added in this PR, is what makes the check cheap: grep it for the area the PR touches. The rule names its staleness in the same breath, because an index that gets treated as state is the failure it was written to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing else asks it. The tracker is not in the diff, so a PR that finishes an issue without saying so leaves it open for someone to rediscover, and the checklist had no line covering that. THE HALF THAT MATTERS IS THE PARTIAL CASE, because closing on a partial fix is where the damage is - the remaining work goes with it. #41 is the worked example and the reason this rule exists: its reopening comment lists four remaining items, #366 did the first, and "366 closes 41" was a reasonable-sounding read that would have taken the other three - widening the mutation target, verifying excludedGroups, and the licence-blocked history plugin - with it. That issue also shows the second half: #366 did item one AGAINST the issue's stated preference, refuting "prefer push-on-master over a cron" with a measurement of this repo's actual merge rate. The recommendation lives on the issue and the evidence does not, so the disagreement has to be recorded there or the next reader re-derives it. The closing keyword needs the fully qualified form - Fixes #167 closes nothing - which docs/issue-references.md already owns and this only points at. docs/inflight/issue-index.md, added in this PR, is what makes the check cheap: grep it for the area the PR touches. The rule names its staleness in the same breath, because an index that gets treated as state is the failure it was written to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing else asks it. The tracker is not in the diff, so a PR that finishes an issue without saying so leaves it open for someone to rediscover, and the checklist had no line covering that. THE HALF THAT MATTERS IS THE PARTIAL CASE, because closing on a partial fix is where the damage is - the remaining work goes with it. #41 is the worked example and the reason this rule exists: its reopening comment lists four remaining items, #366 did the first, and "366 closes 41" was a reasonable-sounding read that would have taken the other three - widening the mutation target, verifying excludedGroups, and the licence-blocked history plugin - with it. That issue also shows the second half: #366 did item one AGAINST the issue's stated preference, refuting "prefer push-on-master over a cron" with a measurement of this repo's actual merge rate. The recommendation lives on the issue and the evidence does not, so the disagreement has to be recorded there or the next reader re-derives it. The closing keyword needs the fully qualified form - Fixes #167 closes nothing - which docs/issue-references.md already owns and this only points at. docs/inflight/issue-index.md, added in this PR, is what makes the check cheap: grep it for the area the PR touches. The rule names its staleness in the same breath, because an index that gets treated as state is the failure it was written to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows astubbs/parallel-consumer#354, which introduced the
highcpu-box-exclusivegroup this PR removes.Description
The
highcpu-box-exclusiveconcurrency group was discarding most of the work it was meant to schedule. This removes it, and with it every workflow-side claim about how much the self-hosted box can do at once.What was wrong
A GitHub
concurrencygroup is a deduplication primitive, not a queue. It keeps one run in progress and at most one pending, and discards whatever arrives behind that. Used as a mutex over a shared machine, a third arrival evicts the run already waiting - so with several branches active, each push deleted somebody else's pending measurement.Measured
The 50 minutes after the group landed - 2026-08-26, 01:03Z-01:53Z, 16 runs across 9 branches, classified by whether the job executed any step at all (
steps == 0means it was cancelled while pending and never started):Five of the six runners sat idle throughout. Runs were cancelled roughly a minute after creation, never having started.
Over a wider window (N=58 completed runs, 2026-08-25T13:06Z-2026-08-26T01:53Z) the never-started rate is 25%, but that window straddles the change - most of it ran the previous per-ref policy, so the post-change window above is the one that measures this defect.
The evicted job is simply whoever queued last, which is as often chaos as Performance. So the suite built to hunt confluentinc/parallel-consumer#857 was the one most often not running - and the workflow's own reasoning only ever accepted displacing Performance ("Performance is advisory ... so it is the right one to displace"). Branch-vs-branch displacement was never covered;
max-parallel: 1fixed only the sibling case within a single run.The key question, and a refuted prediction
The design hinged on how many runners serve the
highcpulabel - if one, the group is redundant, because a single runner already serialises.Six runners are registered and online (
highcpu-1…highcpu-6), all advertisingself-hosted, Linux, X64, highcpu. So the prediction was refuted: the group was not redundant, and removing it does return up to six-way co-residency. That is accepted deliberately rather than by oversight - see below.What replaces it: nothing
Jobs queue on the runners like every other GitHub Actions job, and every queued job eventually runs. How many run at once is the box's own decision, made by how many runner processes it runs. That is not a workflow concern, and encoding it here is what produced this defect; if six is too many, the lever is on the box and needs no change in this repository.
The only cancellation left is the ordinary one every workflow has: a new push to a PR supersedes that PR's previous run.
Why dropping the exclusion is safe
The co-residency reds that motivated the mutex were
~154s lagStagnationagainst a 150s bound - the bound meeting the load, not a defect. That detector was demoted to non-gating in the same PR that added the mutex:ProgressProbe.recordLagStagnationnow callsobserve(...)rather than recording a violation (verified in the code, not inferred from the write-up). The failure the mutex was bought to prevent had already been fixed in the instrument, where it belonged.Alternatives rejected
docs/ci.mdforbids a scheduled lane for anything a PR gate already covers, on the test "does time alone change the answer?" For chaos it does not; the code does.Absent vs. failed vs. passed
A cancelled check still renders as
failingh pr checks, sodocs/ci.mdkeeps the rule that a cancelled chaos check means NOT MEASURED - neither a pass nor a failure, and the job summary still names the commit it measured. What is gone is the case that made this acute: a pending run being discarded so a PR's head was measured by nothing while every check stayed green.Same defect elsewhere
Searched for other uses of a
concurrencygroup as a resource mutex:grep -rn "highcpu-box-exclusive"found the two other members of the group,chaos-pain.ymlandmutation-full-sweep.yml, and both are fixed here. The sweep's own comment already recorded this defect at its widest - a 360-minute timeout that could starve PRs of chaos measurements for hours. No other workflow in the repo pairs a constant (ref-free) group key withcancel-in-progress: false; the remaining groups are all ref-keyed and cancelling, which is dedup and correct.Documentation
docs/ci.md's section is rewritten as "The box decides its own concurrency", including "Why aconcurrencygroup is not a mutex" so the primitive is not reached for again.docs/self-hosted-runner.mdgains the rule at the point an operator would break it. The inflight note is replaced: its premise (confirm the serialised lane works) is gone, and the open question is now a runner-count one -docs/inflight/ci-highcpu-box-concurrency-is-runner-count.md.Six citations to the old note in
bug-857-family.mdandtest-chaos-phase2.mdwere repointed. They were already stale before this PR - they cite a contended-vs-uncontended replay discriminator that had been moved out of that note earlier - so they now point at the solutions note that actually owns it.And finally: the whole lane stopped running per-PR
With chaos gone, the lane's last per-PR entry was
Performance (optional)- which ran the samecommand as
maven.ymlalready runs as the requiredPerformance Testscheck:bin/performance-test.sh, no different arguments.Performance Tests(maven.yml)ubuntu-latesthighcpu / Performance (optional)Two minutes does not justify a second, non-gating copy of a gating check - this lane's own stated
test, and why Unit, Integration and Mutation were removed from it before.
What is kept is the benchmark, which is the one thing the hosted gate cannot produce: solo on
that box the suite is ~1m54 against GitHub's ~3m50. So the workflow becomes
workflow_dispatchonly rather than being deleted.
Consequence worth stating: nothing per-PR depends on the self-hosted hardware any more, so an
offline box can no longer leave a check pending. All three workflows targeting
highcpuare nowdispatch-only.
Security: the same-repo guard went with the
pull_requesttrigger (dispatch requires writeaccess). That guard is the entire trust boundary for a public repo on a self-hosted runner, so the
job and
docs/self-hosted-runner.mdboth now carry an explicit instruction to restore it in thesame commit as any future
pull_request:trigger.And: the full mutation sweep now actually runs
With the box carrying almost nothing, the whole-tree mutation score was the one thing it still
existed for - and it was dispatch-only, dispatched about once in three weeks, so it measured
nothing. The PR-scoped lane in
maven.ymlonly ever mutates classes a PR changed.It now runs nightly, and getting there required overriding this workflow's own recorded advice,
so the reasoning is written down on both sides. The header argued for
push: branches: [master]-a mutation score changes when the code changes, so a nightly recomputes an identical answer on a
night master did not move. Sound in general; it assumes a quiet master.
Measured over the last 60 master commits: up to 32 commits in a day, a median gap of 0
minutes (squash-merges land in bursts), and 83% of gaps shorter than the sweep's own 31m27s
job-elapsed runtime. (An earlier revision of this PR said "80% under ~22 minutes" - that 21m55s
figure is the PIT phase, not job elapsed, mislabelled as "Wall clock" in
docs/inflight/ci-mutation-testing.mdand inherited from there. Corrected in this PR; the argumentgets stronger, not weaker.) Per-push therefore fails both ways - no concurrency group piles dozens of concurrent
sweeps onto one box; a cancelling group has four in five killed before they finish, which is the
never-completes failure (
internal.*) this lane was rebuilt to escape.This is a deliberate exception to
docs/ci.md's "there is no scheduled build" rule, recordedthere rather than left to contradict it silently. What makes it an exception is not convenience but
that the correct per-merge trigger was measured and found unusable.
A hosted arm runs alongside it as a trial -
ubuntu-latest, 4 threads. The thread split is aRAM ceiling, not a preference: PIT minions get
-Xmx2geach, so 16 threads needs 32g and would OOMa hosted runner (4 vCPU / 16 GB). Neither arm is required; both are
continue-on-error. Comparetheir scores first, not wall-clock. This is safe to trial because
bin/ci-mutation-test.shalready detects an all-
MEMORY_ERRORrun and fails rather than reporting a vacuous green.Bug caught in passing:
inputs.*is empty on a non-dispatch event, so the trigger change wouldhave expanded to an empty target glob and a bare
-Dthreads=. Every one now has a fallback.Follow-ups deliberately not done here
Chaos Pain Suitea required check - after merge, because a required context no run hasproduced blocks every PR whose base predates it.
is dispatched far too rarely to measure anything. It needs
docs/ci.md's "there is no scheduledbuild, deliberately" rule amended rather than quietly contradicted - that rule's test is "does
time alone change the answer?", and for mutation it is merges, not time. Recorded as an open
decision in
docs/inflight/ci-highcpu-box-concurrency-is-runner-count.md.Checklist
docs/ci.md,docs/self-hosted-runner.md, the workflow header comments, and the replaceddocs/inflight/notebin/gate to assert it without a script that asserts the box's runner count, which is exactly the workflow-side capacity claim this PR argues against. Verified withactionlint(clean) and by parsing the rendered matrix and concurrency block.N/A - workflow YAML and docs only, no code in the diff.ce-simplifyrefuses a diff with nocode by design, so there is nothing for it to act on.
ce-code-reviewwas not run locally either -the review worth spending on this change is the one that reads CI behaviour rather than source,
so the intent is to ask for
@claude review thison the PR, which is also the only route that canopen inline review threads. Analysis surfaces were checked instead:
bin/check-pr-analysis-surfaces.sh 366exits 0 with no annotations on the diff.Then: chaos left the self-hosted box entirely
The fix above manages contention for one shared machine. This PR then tests whether that machine is
needed at all, and concludes it is not.
Measured head-to-head on the same commit, both green (that commit was
d8beb162fbefore thisbranch was re-cut; the SHA no longer exists, the measurement stands):
highcpuubuntu-latest14% of wall-clock does not buy a shared physical machine - and the sharing is what produced every
scheduling problem this lane has had. A hosted runner gives each job its own VM, so co-residency
cannot occur there at all. This is the root-cause fix that the concurrency change alone could only
mitigate.
The premise that put chaos on the box - that it needs many real cores to provoke anything - did not
survive measurement. Corroborating it:
bin/chaos-test.shneeded no change to move, because itpasses no
forkCountand no-Dparallel-tests, so the suite was never configured to exploit thosecores.
Still gating. The hosted entry is deliberately not
optional- a chaos RED is a real finding,and the suite it replaces was not optional either. It is named plainly
Chaos Pain Suite(not... (hosted)) because it is about to become a required context, where the name is an API.Making it required is a separate act, after merge. A required context that no run has produced
leaves every PR pending until it merges master, so the ruleset change waits until the job is on
master - the ordering
docs/ci.mdalready states.Caveat recorded rather than glossed: the move was settled on one head-to-head run, and the
hosted job's per-scenario counts were not read (the job-log endpoint returned empty). Duration rules
out a zero-scenario run - that is build-only, ~2 minutes - but the standing rule still applies: read
the job's own
Chaos suite timingsummary and its zero-tests-selected warning before trusting agreen.
What the box carries now
Per-PR, only
Performance- which runs the samebin/performance-test.shasmaven.yml'srequired
Performance Testscheck, i.e. a non-gating duplicate of a gating one. Plus twodispatch-only workflows, of which the full mutation sweep has been dispatched once, on
2026-08-04. Left as two open decisions in
docs/inflight/ci-highcpu-box-concurrency-is-runner-count.mdrather than acted on here.