Skip to content

ci(repo): publish main's full kill matrix as an evidence lane - #263

Merged
ryanleecode merged 4 commits into
mainfrom
stryker/kill-matrix
Oct 10, 2026
Merged

ryanleecode merged 4 commits into
mainfrom
stryker/kill-matrix

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Every push to main now also publishes the dogfood corpus's full kill matrix: for each mutant, every test that kills it, not only the first. Today's Mutation run stops at the first failing test, and the merged mutation.json drops killedBy, so nothing on main can show that a mutant-reduction change loses no kills. The mutant-subsumption work (stryker/mutant-subsumption-layer-1) waits for this lane's first main baseline before it merges.

Root-owned wiring (CONST-E9). This PR changes .github/workflows/, which this unit may not change on its own authority, so it is for the root to review and merge.

What runs, and what it never touches

  • kill-matrix.yml is a reusable workflow (workflow_call, plus workflow_dispatch once it is on main). mutation.yml calls it on push to main and on a dispatch with lane: kill-matrix. It has no pull_request trigger.
  • The lane re-verifies every mutant of the existing PROJECTS corpus with the released CLI. Every step that loads a project config runs with STRYKER_KILL_MATRIX=1. With that variable set, sharedConfig adds disableBail: true and mutator.mutantSetPolicy: 'full'. Without it, sharedConfig is unchanged. The plan uses --full, and the lane restores no incremental report, so no verdict is reused.
  • The lane only produces evidence. It adds no required check and saves no cache. A lane: kill-matrix dispatch skips the plan, build, mutation and report jobs, so it cannot write the main incremental cache or publish a budget baseline. A lane: all dispatch behaves as it does today.
  • The collect step fails, and publishes no artifact, in three cases: a project's report was not made under full, a mutant id in the plan is missing from the merged report, or the corpus has killed mutants but none of them names a second killer.

How an agent consumes this

  • Read: artifact kill-matrix-<run_number> from the latest successful main Mutation run, with one file per project at <project>/stryker-incremental.json. Each mutant record has id, status, killedBy (every killing test id), coveredBy and testsCompleted, all under mutantSetPolicy: 'full'.
  • Do: look up a dropped mutant and its dominator by id, then check killers(dominator) ⊆ killers(mutant). When a mutant is missing from the matrix, the check cannot be made for that pair, so treat it as unresolved, never as a pass. The matrix holds the full mutant set, so it does not contain the extra relational operator (a < b to a != b) that the default policy adds in conditions. A Killed mutant with an empty killedBy was stopped by the hit limit (its statusReason starts Hit limit reached), so no test is credited with the kill. Its killer set is unknown, not empty, so treat it as unresolved.
  • On failure: each failed step emits ::error naming the project, shard, or missing ids, the next action (usually: re-run the kill-matrix lane), and where the log or partial artifact is. The step summary has one row per project: mutants, Killed, mutants with ≥2 killers, and Σ testsCompleted.

Review dispositions

Review: ce-code-review, focused depth. Correctness was reviewed in the main session, plus one separate adversarial read. No reviewer from a different model family was available, because no such CLI is installed in this container.

# Sev Finding Disposition
1 P1 A full matrix leaves out the extra operator that default adds in condition positions (Mutator.service.ts:985-988) Confirmed. Not a defect for this consumer: the subsumption audit compares complements with their ordering-operator dominators, and full produces both. A missing id is unresolved, not a pass. The mutant set is stated above.
2 P2 The plan was made under default while the shards ran under full, so planned ids could disappear without any error Fixed: STRYKER_KILL_MATRIX is now set for the whole workflow, and collect fails when a planned id is not reported
3 P2 The guard checked the policy field, not whether more than one killer was recorded Fixed: collect fails when the corpus has Killed mutants and none names a second killer
4 P3 The summary's "tests completed" column counted declared tests Fixed: the column now sums testsCompleted
5 P3 The lane keeps main's Mutation run open until it finishes, so the next main push waits behind it, and an older pending run is replaced Not fixed. The ruling puts the call in mutation.yml. Noted here for the root.

Verification

  • actionlint 1.7.12 reports nothing for kill-matrix.yml. For mutation.yml it reports only SC2086 on $FULL, which is already on main. dprint check passes.

  • Config probe: when lib/base.js is imported without STRYKER_KILL_MATRIX, neither field is set. With STRYKER_KILL_MATRIX=1, it sets disableBail: true and mutator: { mutantSetPolicy: 'full' }.

  • I ran the collect step's summary script on fixtures. A complete matrix exits 0. A missing planned id, a corpus where every kill was a first kill, and a default report each exit 1 with their annotation.

  • Dispatch proof: run 38021709170 (workflow_dispatch of mutation.yml, lane=kill-matrix, ref stryker/kill-matrix, head d28df2083). All 23 kill-matrix jobs passed (plan, build, 20 shards, collect), and the default lane's plan, build, mutation and report jobs were skipped. Artifact: kill-matrix-419. I recomputed these numbers from the artifact. Every project ran under mutantSetPolicy: 'full', and all 8,875 mutant ids in plan.json are reported:

    project mutants Killed Killed with ≥2 killers Σ testsCompleted
    packages/stryker-js 6,585 865 338 13,117
    packages/stryker-js-typescript-checker 843 92 68 3,188
    packages/stryker-js-vitest-runner 220 43 15 341
    test/e2e-core 1,227 792 652 13,641
    corpus 8,875 1,792 1,073 30,287

    One Killed mutant has an empty killedBy: checker f68b6e883cbcc324, src/trace-affected-files.workflow.ts:62, which was stopped by the hit limit (Hit limit reached (15649/11400)).


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

The plan step now loads every project's config with STRYKER_KILL_MATRIX=1, the same as the shards, so it requests the ids the shards emit. The collect step fails with an annotation when a planned mutant id is missing from the merged report, or when the corpus has killed mutants and none names a second killer. The summary's tests column sums testsCompleted.

@ryanleecode ryanleecode left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: CI green on d28df20, diff read, no open threads.

@ryanleecode
ryanleecode merged commit 7d15adc into main Oct 10, 2026
37 checks passed
@ryanleecode
ryanleecode deleted the stryker/kill-matrix branch October 10, 2026 04:15
systemfsoftware-maker added a commit that referenced this pull request Oct 10, 2026
Brings in #262, #263 and #265 (main 4bf5952).

One conflict, in packages/stryker-js/src/mutation-reporting.service.ts.
#262 added `rememberedMutantIds` to MutationReportingInput and a
stampRemembered step to the incremental-report writers. On this branch
the service file declares the port only and the implementation lives in
drivers/mutation-reporting.ts, so the input field goes on the port and
stampRemembered plus its two call sites go into the driver.

#262's check-cost-workspace fixture called Engine.RunEnvironment.stage,
which this branch renamed to Engine.stage; the fixture now uses that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants