Skip to content

perf(stryker-js): enter the instrument phase before instrumenting (T10 U1) - #348

Merged
ryanleecode merged 4 commits into
mainfrom
stryker/t10-u1-instrument-mark
Oct 13, 2026
Merged

ryanleecode merged 4 commits into
mainfrom
stryker/t10-u1-instrument-mark

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

U1 of the T10 infection-filtered execution plan (docs/plans/2026-10-13-1536-perf-infection-filtered-execution-plan.md, landed in this PR): R18, KTD9.

The instrument phase mark (phaseEntered('instrument')) and the instrument phase span ran in the instrument cells' write phase, after instrumentFiles and makeSandbox had already run in the read phase. Instrumentation and the sandbox copy counted under prepare, and every Bench sample read instrument as 0-1 ms (A/A dispatch 38375500452). The instrument share that gates the T9 native-core restart could not be read.

Both cells (run/instrument.cell.ts for runs, run/plan-instrument.cell.ts for plans) now open the phase span and emit the mark at the start of the read phase. The write phase only offers file skips. No verdict, event shape or schema changes; phaseDurations.instrument grows and prepare shrinks by the same amount. Patch changeset for @systemfsoftware/stryker-js.

Evidence

  • New packages/stryker-js/tests/instrument-phase-span.integration.test.ts (one Feature, .live over real dry-run-only runs; OTel InMemorySpanExporter):
    • run and plan rows: a dry-run-only Engine.strykerCell run and an Engine.planRequest plan on a real temp project. The gather span (and, for the run, stryker.sandbox.acquire) lies inside the single instrument span, and the phase-mark span ends before gathering starts.
    • failing build: a project whose buildCommand exits 1. The Engine.mutationTestCell run on the stage event queue fails with StageError { stage: 'instrument', reason: 'Sandbox initialization failed' } after PhaseEntered prepare then instrument. A second Engine.strykerCell run on a fresh project fails the same way, and its failed gather and sandbox work stay inside the one instrument span.
    • Red on main's cells, checked in a scratch worktree at origin/main with this test file copied in: all 3 scenarios fail (insidePhase: [], markEndsBeforeGather: false, phases ['prepare'], phaseSpans: 0). Green on this head.
  • Local: on 63d0a04ed, pnpm --filter @systemfsoftware/stryker-js exec vitest run exited 0 (full package suite, at 1ecca4e2f; the only later change is this test file, rerun green), plus typecheck, dprint check, oxlint, lint:tsgo (0 errors) and pnpm lint:conventions. Head 235059e45 is a conflict-free merge of origin/main (449e9cd53) into 63d0a04ed, and the PR's five-file diff is unchanged by it. On that head: build, typecheck, format:check and the three affected integration suites (15 tests). No mutation run anywhere (dry-run-only).
  • Speed: no speed claim. Pre-U1 baseline (root's median-of-four over the first four green main Bench push runs after ci(bench)!: take the bench off the pull-request path #316: 38429923569, 38430066932, 38436282117, 38436508210) reads instrument at 0-0.5 ms on every leg because of the misplaced mark this PR fixes; see the plan's T9 section for the per-leg table. The T9 decision reads S = instrument / total as the median of the first four green main push runs on or after this PR's merge commit. Skipped share is not applicable to U1.

Code review

Four ce-code-review rounds (mode:agent), the last on the exact head 235059e45. Every finding below is fixed in a commit on this head or rejected by an independent validator with the reason kept.

  • Round 1, full depth, run 20261013-174110-ae429c85, base a83d1b32e: correctness, project-standards, testing, adversarial (in-process) and learnings with the seven enrolled packs. One P3, fixed (below). Plan fix: a stale "ci(bench)!: take the bench off the pull-request path #316 is open" blocker line was removed.
  • Rounds 2-3, delta 03e7e4233..HEAD plus a full-PR cross-model pass, runs 20261013-190823-3280b24d, 20261013-192400-eed6df5a and 20261013-193111-5d19efcd (final, on 63d0a04ed). In-process: correctness and testing on the delta, both with zero findings on the final head. Cross-model adversarial peer, served model MiniMax-M3.1-Flash-Preview (provider minimax-code, attested from the peer transcript's assistant message_end records). Final-head response ids: delta 071dba1bcf195fd40401afcb797b90d2, full 071dba7cbf2d5af419298f831c30866f. Read-only tools read,grep,glob, no extensions.
  • Round 4, exact head: a report-only correctness lens over exactly 63d0a04ed..235059e45 (the conflict-free merge of origin/main 449e9cd53, bringing in build(deps): bump pnpm-release-management from 21fe619 to b9b4a1d #344 and fix(stryker-js)!: stryker gate judges each project of a merged report against its own thresholds.break #299; 16 files, none of them U1's), run 20261013-195252-fe602168. Served model MiniMax-M3.1-Flash-Preview (provider minimax-code, attested from the peer transcript's assistant message_end records, final response id 071dbf84c8cce7b5dcabac3a3c3221bf). Read-only tools read,grep,glob, no extensions. Zero findings. The reviewer found no merged-in file that touches U1's files or references the instrument phase, PhaseClock or the instrument span. Its residual notes (the #299 gate's thresholds.break decoding default, and journeys for --update-budget-baseline and files outside every listed project) concern main's already-merged gate code, not this PR, and are not dispositioned here.
Finding Source Disposition
P3: the instrument-failure path of the moved mark is untested testing (round 1) Fixed in 63d0a04ed: the failing-build scenario above
P2: the failure scenario asserts events only, not the phase span MiniMax delta Fixed in 63d0a04ed: the span run asserts phaseSpans: 1 plus containment
P2: two runs share one mutated project directory MiniMax delta Fixed in 63d0a04ed: the span run writes its own fresh project
P2: the span-run failure is checked only as a boolean, not the stage MiniMax delta Fixed in 63d0a04ed: the span run asserts the same StageError stage and reason
P2: process-global chdir in the harness MiniMax full Rejected (validator): vitest forks + isolate: true gives each file its own process, scenarios run sequentially, and 13 sibling suites use the same pattern. strykerCell resolves basePath from cwd (RunEnvironment.service.ts:84)
P2: bounded queue is drained only after the run, so a regression hangs MiniMax delta Rejected (validator): dryRunOnly never reaches mutation testing, so no per-mutant events can fill the 256-slot queue; a hang still fails on testTimeout
P2: the guard pins span nesting, never phaseDurations (raised twice) MiniMax full Rejected (validator): the instrument boundary that phaseDurations reports is the markAt inside the stryker.phase.entered span (RunEnvironment.service.ts:132-140), so "mark span ends before gather starts" pins it. The repeat on the final head is suppressed under that verdict (unchanged code)
P3: span fileCount comes from the project map, command fileCount from the read list MiniMax full Rejected (validator): ProjectFiles.readAll is 1:1 (project-files.service.ts:84-93), so the counts are equal by construction; the attribute is telemetry-only
P2: a repeated read phase could emit a duplicate instrument mark MiniMax full Rejected (validator): Sandwich runs read exactly once per cell run with no retry loop, every caller runs each cell once per run, and the old write-phase placement had the same multiplicity

Evidence limits (not findings): reviewers suggested proving events and spans from one execution. That was tried this round and fails: an OTel layer on the mutationTestCell stage path exports no spans, so the span half runs through strykerCell on its own fresh project. A configured buildCommand or node_modules symlink counts toward phaseDurations.instrument because makeSandbox runs inside the phase. Whether the Bench corpus configs set either is [unverified]; R18 readings should note it. Instrumenter-parse failures (Instrumenter failed) take the same enteringInstrumentPhase path but have no scenario of their own.

Cross-stream notes

The `instrument` phase mark and its `instrument` span ran in the instrument
cells' write phase, after `instrumentFiles` and `makeSandbox` had already run
in the read phase. Instrumentation and the sandbox copy therefore counted
under `prepare`, and every Bench sample read `instrument` as 0-1 ms, so the
instrument share of wall clock that gates the T9 native-core restart could not
be read.

Both cells (`instrument.cell.ts` for runs, `plan-instrument.cell.ts` for
plans) now open the phase span and emit the mark at the start of the read
phase; the write phase only offers file skips. The new engine integration test
drives a dry-run-only run and a plan through the published `Engine` entry
points and asserts from OTel span intervals that the gather and sandbox
acquisition spans fall inside the single `instrument` span and that the phase
mark ends before gathering starts. It failed on the old placement and passes
on the new one.

Lands the T10 infection-filtered execution plan (U1 of it, R18, KTD9).
@systemfsoftware-maker systemfsoftware-maker changed the title stryker/t10 u1 instrument mark perf(stryker-js): enter the instrument phase before instrumenting (T10 U1) Oct 13, 2026
Root's reading of the first four green main Bench push runs after #316
(38429923569, 38430066932, 38436282117, 38436508210) replaces the plan's
[unverified] baseline run ids, with the per-leg median-of-four phase table.
Those runs read `instrument` as about 0 ms on every leg because the mark is
misplaced, so they are the baseline later T10 layers compare against, not the
T9 reading: R18 now reads `S` from the first four green main push runs after
U1 merges. R17 states Bench's non-overlapping-samples rule, since a single
typescript-checker run took 33.5 s against about 46 s for the other three.
Review finding (P3, testing): the changeset says a run that fails while
instrumenting or building its sandbox now reports the `instrument` phase as
entered, but no scenario drove that failure. The new scenario writes projects
whose `buildCommand` exits 1 and runs dry-run-only mutation runs in them.
Through `Engine.mutationTestCell` and the stage event queue it asserts the run
fails in the `instrument` stage with reason `Sandbox initialization failed`,
after `prepare` then `instrument` were reported entered. Through
`Engine.strykerCell` and the span exporter, in a second fresh project, it
asserts the same stage failure and that the failed gathering and sandbox work
stay inside the one `instrument` phase span. Against main's cells the phases
read `['prepare']` and no phase span exists; on this branch both hold.

Every scenario runs inside its temp project's working directory, so config
discovery cannot pick up the package's own Stryker config.

@ryanleecode ryanleecode left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Root verdict on 235059e: round 1 was full depth with the seven enrolled packs. Rounds 2-3 were delta passes plus a full-PR MiniMax M3.1 cross-model pass. Round 4 was an exact-head correctness lens (MiniMax M3.1) over the conflict-free merge of main 449e9cd, which touches none of U1's files. Every finding is fixed in 63d0a04 or rejected by the validator with the reason kept. CI is green on the head, with 0 threads, and there's no overlap with main since its base. The plan is the accepted T10 plan with my gate changes. U1 moves the instrument mark, so R18 can re-read the instrument share for the T9 gate.

@ryanleecode
ryanleecode merged commit b08379d into main Oct 13, 2026
17 checks passed
@ryanleecode
ryanleecode deleted the stryker/t10-u1-instrument-mark branch October 13, 2026 20:01
systemfsoftware-maker added a commit that referenced this pull request Oct 13, 2026
…t, wire-text stream decode

Takes main through b08379d (#333, #343, #339, #320, #346, #344, #286, #348,
and the gate per-project thresholds fix).

Conflicts:
- stryker-outputs.ts: main's labeled output list (#320) keeps this branch's
  two extra exclusions, each with its own label for the --ignorePatterns
  help: the incremental file's temp siblings (the killed-run fix) and an fs
  verdict store's directory. An output may now name several files.
- instrument.cell.ts, plan-instrument.cell.ts: main's instrument-phase
  placement (#348), carrying this branch's fileContentDigests.
- plan-request.cell.ts: main's planStageCell; the costs and probe schemas
  this branch retired stay gone, and the plan reads the one incremental file.
- program-digest.integration.test.ts: this branch's config/program digest
  scenarios plus main's importerCheck scenario.
- e2e-harness.fixture.ts, run-artifacts.fixture.ts: main's RunEventWireText
  decode (#339); the store-entry count stays and the checkpoint reader, with
  no caller left, does not come back.

#286 added a local configKeyOf in ts-compiler.handle.ts, so this branch's
schema export of that name becomes checkerConfigKeyOf.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants