Repository navigation
ci(repo): publish main's full kill matrix as an evidence lane - #263
Merged
Merged
Conversation
The plan step now loads every project's config with STRYKER_KILL_MATRIX=1, the same as the shards, so it requests the ids the shards emit. The collect step fails with an annotation when a planned mutant id is missing from the merged report, or when the corpus has killed mutants and none names a second killer. The summary's tests column sums testsCompleted.
ryanleecode
approved these changes
Oct 10, 2026
ryanleecode
left a comment
Contributor
There was a problem hiding this comment.
Verified: CI green on d28df20, diff read, no open threads.
systemfsoftware-maker
added a commit
that referenced
this pull request
Oct 10, 2026
Brings in #262, #263 and #265 (main 4bf5952). One conflict, in packages/stryker-js/src/mutation-reporting.service.ts. #262 added `rememberedMutantIds` to MutationReportingInput and a stampRemembered step to the incremental-report writers. On this branch the service file declares the port only and the implementation lives in drivers/mutation-reporting.ts, so the input field goes on the port and stampRemembered plus its two call sites go into the driver. #262's check-cost-workspace fixture called Engine.RunEnvironment.stage, which this branch renamed to Engine.stage; the fixture now uses that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every push to main now also publishes the dogfood corpus's full kill matrix: for each mutant, every test that kills it, not only the first. Today's Mutation run stops at the first failing test, and the merged
mutation.jsondropskilledBy, so nothing on main can show that a mutant-reduction change loses no kills. The mutant-subsumption work (stryker/mutant-subsumption-layer-1) waits for this lane's first main baseline before it merges.Root-owned wiring (CONST-E9). This PR changes
.github/workflows/, which this unit may not change on its own authority, so it is for the root to review and merge.What runs, and what it never touches
kill-matrix.ymlis a reusable workflow (workflow_call, plusworkflow_dispatchonce it is on main).mutation.ymlcalls it on push to main and on a dispatch withlane: kill-matrix. It has nopull_requesttrigger.PROJECTScorpus with the released CLI. Every step that loads a project config runs withSTRYKER_KILL_MATRIX=1. With that variable set,sharedConfigaddsdisableBail: trueandmutator.mutantSetPolicy: 'full'. Without it,sharedConfigis unchanged. The plan uses--full, and the lane restores no incremental report, so no verdict is reused.lane: kill-matrixdispatch skips the plan, build, mutation and report jobs, so it cannot write the main incremental cache or publish a budget baseline. Alane: alldispatch behaves as it does today.full, a mutant id in the plan is missing from the merged report, or the corpus has killed mutants but none of them names a second killer.How an agent consumes this
kill-matrix-<run_number>from the latest successful main Mutation run, with one file per project at<project>/stryker-incremental.json. Each mutant record hasid,status,killedBy(every killing test id),coveredByandtestsCompleted, all undermutantSetPolicy: 'full'.killers(dominator) ⊆ killers(mutant). When a mutant is missing from the matrix, the check cannot be made for that pair, so treat it as unresolved, never as a pass. The matrix holds thefullmutant set, so it does not contain the extra relational operator (a < btoa != b) that thedefaultpolicy adds in conditions. AKilledmutant with an emptykilledBywas stopped by the hit limit (itsstatusReasonstartsHit limit reached), so no test is credited with the kill. Its killer set is unknown, not empty, so treat it as unresolved.::errornaming the project, shard, or missing ids, the next action (usually: re-run the kill-matrix lane), and where the log or partial artifact is. The step summary has one row per project: mutants, Killed, mutants with ≥2 killers, and ΣtestsCompleted.Review dispositions
Review:
ce-code-review, focused depth. Correctness was reviewed in the main session, plus one separate adversarial read. No reviewer from a different model family was available, because no such CLI is installed in this container.fullmatrix leaves out the extra operator thatdefaultadds in condition positions (Mutator.service.ts:985-988)fullproduces both. A missing id is unresolved, not a pass. The mutant set is stated above.defaultwhile the shards ran underfull, so planned ids could disappear without any errorSTRYKER_KILL_MATRIXis now set for the whole workflow, and collect fails when a planned id is not reportedtestsCompletedmutation.yml. Noted here for the root.Verification
actionlint1.7.12 reports nothing forkill-matrix.yml. Formutation.ymlit reports only SC2086 on$FULL, which is already on main.dprint checkpasses.Config probe: when
lib/base.jsis imported withoutSTRYKER_KILL_MATRIX, neither field is set. WithSTRYKER_KILL_MATRIX=1, it setsdisableBail: trueandmutator: { mutantSetPolicy: 'full' }.I ran the collect step's summary script on fixtures. A complete matrix exits 0. A missing planned id, a corpus where every kill was a first kill, and a
defaultreport each exit 1 with their annotation.Dispatch proof: run 38021709170 (
workflow_dispatchofmutation.yml,lane=kill-matrix, refstryker/kill-matrix, headd28df2083). All 23 kill-matrix jobs passed (plan, build, 20 shards, collect), and the default lane's plan, build, mutation and report jobs were skipped. Artifact:kill-matrix-419. I recomputed these numbers from the artifact. Every project ran undermutantSetPolicy: 'full', and all 8,875 mutant ids inplan.jsonare reported:packages/stryker-jspackages/stryker-js-typescript-checkerpackages/stryker-js-vitest-runnertest/e2e-coreOne Killed mutant has an empty
killedBy: checkerf68b6e883cbcc324,src/trace-affected-files.workflow.ts:62, which was stopped by the hit limit (Hit limit reached (15649/11400)).Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.