[systemfsoftware-maker]

Every pull request that changes more than documentation now gets a Bench workflow that times a real mutation run on the PR head against the same run on its merge-base, per project and per phase, each project on its own runner. Before this there was no CI evidence of whether a change made a mutation run faster or slower, or which phase the time goes to.

First PR of Stream G (execution engine). The later optimisations in the stream (per-test selection, infection filtering, kill-likelihood ordering, warm module swap) each cite this lane's table as their speed evidence; the instrument share also decides whether Stream F's native schemata emitter goes ahead.

What the lane measures

.github/workflows/bench.yml has a plan job that reads test/bench/corpus.json and starts one bench job per entry. Each job builds side A (the merge-base with the base branch) and side B (the PR head) on its runner, with each side on its own released-tarball pin, and runs both interleaved in the order A B B A B A A B with --full over that entry:

• three repo entries in test/bench/corpus.json, one dogfood file each from three packages, chosen so that every phase does real work and one entry has no checker. Each entry is bounded so its eight runs fit the job budget: the vitest-runner entry mutates lines 85-160 of its file, and the vitest-runner and TypeScript-checker entries run only their workflow's property test (testFiles). Both sides get the same bound, and it shapes timing only, never a verdict;
• the enterprise-monorepo e2e fixture.

PRs that change only **/*.md or docs/** skip the workflow. A full-corpus bench belongs on main and is not part of this lane.

Time budget

No bench job may run over 10 minutes, and the whole workflow must finish within 10. Each bench job has timeout-minutes: 9 and plan has 1. Each job also sets BENCH_DEADLINE_MS 7.5 minutes after it starts. The orchestrator clips each run's timeout to that deadline, and when the deadline stops a run or arrives before one starts, it writes an aborted report with code budget-exceeded, the finished runs' times and the next action, emits ::error and exits 1. An overrun therefore fails with a reason before the job ceiling is reached.

Measured on head 714dae7, run 38037113409 (whole workflow 6 min 18 s, created to last job done):

job	wall	setup inside the bench step	per run (8 runs)
plan	0:06
test/e2e-core	4:00	turbo build 38 s + 37 s, installs 10 s + 6 s	15-16 s
packages/stryker-js-typescript-checker	5:15	turbo build 22 s + 21 s, installs 6 s + 4 s	29-30 s
enterprise-monorepo fixture	5:28	closure build and pack 39 s + 37 s, fixture install 17 s + 10 s	18-35 s
packages/stryker-js-vitest-runner	5:41	turbo build 39 s + 36 s, installs 9 s + 7 s	27-30 s

The released tarballs come from the released-tarballs cache that ci.yml already fills, so neither side runs Nix when the two pins match (side A copies side B's tarballs). Before this change one job ran every entry and took 26 minutes (3.3 minutes of tarball builds, 22.5 minutes of runs).

Every timing comes from the run's own NDJSON verdict line, not from wall-clock wrapped around the CLI. The job summary shows one table per project:

phase	main (A)	PR (B)	Δ	verdict
prepare	median s · share of total	median s · share of total	B − A	improved / regressed / no signal / not measured / workload unverified
instrument	…	…	…	…
check	…	…	…	…
dry-run	…	…	…	…
mutation-test	…	…	…	…
reporting	…	…	…	…
total	…	…	…	…

A cell only counts as a signal when the four samples on each side do not overlap at all and |Δ median| is at least 3% of A. Under the null hypothesis a cell is a false signal in at most 2/70 ≈ 2.9% of orderings, so treat a lone signal as a reason to re-run, not as proof. Each project also reports its workload status (same, changed with the differing entries, or unverified when a digest could not be computed) and both sides' mutant and executed-test counts. A cell with a changed or unverified workload never signals.

Until this PR is on main, side A's verdict lines carry no check or reporting duration. Those cells read "not measured", and so does mutation-test on side A, because the older engine counts reporting time inside it.

Contract change (breaking)

• The verdict line's phaseDurations gains check (measured {ms} | not-run | not-recorded) and reporting (measured {ms} | not-recorded). check is checker-busy wall-clock, the union of the intervals in which any checker was starting or checking. It overlaps the other phases.
• RunPhase gains reporting, and mutation-test now stops where reporting starts, so the five sequential phases add up to the elapsed time. Exhaustive matches over RunPhase have to handle the new value.
• Verdict lines written before this change decode both new keys as { "_tag": "not-recorded" }. The new keys ship in stream schemaVersion 8.0, the version main's unreleased subsumption change already introduced. The changeset bumps stryker-js-cli-contract minor and stryker-js major.

How an agent consumes the result

1. Artifacts. Download the bench-report-<entry slug> artifacts from the Bench workflow run on the PR head, one per corpus entry. Each holds that entry's bench-report.json.
2. Schema. Decode it with BenchReportJson from @systemfsoftware/stryker-e2e-core (test/e2e-core/src/bench-report.schema.ts, schemaVersion: '1.0'). outcome is a tagged union:
   • summarized: projects[], each with corpus, entry, workload, counts, and rows[] of { phase, a, b, verdict }. A measured side cell stores every sample (ms, shareOfTotal) in position order, and median, min and max are derived from those samples. verdict is improved | regressed | no-signal | not-measured | workload-unverified, with deltaMs and deltaShareOfA where measured.
   • failed: invalid[], each a run key plus code (stream-undecodable | stream-invalid | timeout), exitCode and a bounded stderrTail.
   • aborted: code, reason and nextAction for any setup or orchestration failure, including entry-unknown and budget-exceeded.
     runs[] holds every run's outer wall-clock and exit code, and setupSteps[] the time of each setup step.
3. Verdict line. Each bench job's log carries exactly one annotation: ::error title=Bench regression::… when any cell regressed, ::error for a failed or aborted report (the job also exits non-zero), and ::notice otherwise. The markdown table and the annotation are both rendered from the decoded report, so they cannot disagree with the artifact.

The lane is report-only. It is not part of ci.yml, so no required status context changes.

Code the new tests target in main's mutation report

The dogfood mutation lane mutates src/**/*.workflow.ts and src/**/*.schema.ts in each package. The new laws and integration scenarios target:

• test/e2e-core/src/summarize-bench.workflow.ts: per-phase rule table, sample separation, the 3% floor, workload same / changed / unverified, and suppression of mutation-test when side A's reporting was not recorded.
• test/e2e-core/src/read-bench-run.workflow.ts: exactly one verdict line, undecodable or invalid streams, and timeouts with their exit code and stderr tail.
• test/e2e-core/src/render-bench-report.workflow.ts: table, notice or error choice, failed and aborted markdown, and annotation escaping.
• test/e2e-core/src/bench-summary.schema.ts, bench-report.schema.ts, bench-run.schema.ts: round-trips and abort codes with their next actions.
• packages/stryker-js/src/phase-durations.workflow.ts: the five sequential phases sum to the elapsed time, a missing mark yields PhaseMarksMissing, and check is not-run without a checker.
• packages/stryker-js/src/phase-durations.schema.ts: CheckerBusyIntervalSchema refuses endMs < startMs (one clamp, at PhaseClock.recordCheckerBusy), and the checker-busy union is bounded, order-free, split- and duplicate-invariant.
• packages/stryker-js-cli-contract/src/run-event.schema.ts, through the wire-line and contract-version suites: absent keys decode as not-recorded, and the contract-compat check now reports a removed constrained union branch.

Verification

Run locally on the head: pnpm typecheck (58/58 tasks), plus the changed and added suites in test/e2e-core (bench-decisions, read-bench-run and render-bench-report), packages/stryker-js (phase-durations, check-cost-record, readmit-subsumed, run-event-stream: 35) and packages/stryker-js-cli-contract (all 160), all passing. The bench smoke covered the environment-incomplete, corpus-unreadable, side-setup-failed, entry-unknown and budget-exceeded aborts: each wrote the aborted report, emitted ::error and exited 1, and no Stryker run started. No bench run or mutation run was done locally. The lane itself runs only in CI, on this PR.

Plan: docs/plans/2026-10-10-0536-perf-execution-engine-bench-lane-plan.md.

[ryanleecode]

Root verified: all checks green, 0 threads, no suppressions. Over-10m legs inherited (Stream K).