Skip to content

ci(mutation): rebalance kernel-errors into eight shards - #3951

Closed
strzhao wants to merge 1 commit into
callstack:mainfrom
strzhao:ci/mutation-kernel-errors-shards
Closed

strzhao wants to merge 1 commit into
callstack:mainfrom
strzhao:ci/mutation-kernel-errors-shards

Conversation

@strzhao

@strzhao strzhao commented Oct 25, 2026

Copy link
Copy Markdown

What happened

The Oct 25 weekly sweep failed two jobs in one chain: Mutants (kernel-errors-4) was killed by the step's 22-minute timeout with Stryker at 41% (20→41% over the last minutes, remaining: ~5m and climbing), and Mutation score report then failed on the incomplete shard set (17/18) — the report lane's designed indictment of a dead shard, not a separate fault.

packages/kernel/src/errors.ts keeps growing (the resource-ownership error reasons #3887 landed alone pushed the module past 350 mutants, and its slow-test region runs ~28 s/mutant), so the shard holding that region can no longer finish inside the budget. The lane's own registry comment names the remedy: "a module big enough to outrun the lane's 30-minute budget declares more".

Fix

kernel-errors shards: 5 → 8. The slow region is diluted to ~10–12 minutes per bin, inside the budget with headroom. Everything the registry drives follows mechanically:

  • shardMatrix() and the weekly matrix gain kernel-errors-6/7/8; --expect-shards 18 → 21.
  • mutation-affected derives its matrix from the same registry and needs no edits; its step budget stays at the tested 22 minutes.
  • The pin tests that assert the matrix and the merged-shard requirement are updated to the new counts.

Verified: node --test scripts/mutation/*.test.ts — 70/70 pass.

The Oct 25 weekly sweep timed out kernel-errors-4: the resource-ownership
error reasons grew errors.ts past 350 mutants, and the shard holding the
slow-test region crawled past the 22-minute step budget before Stryker
wrote a report. The score job then failed on the incomplete shard set
(17/18), which is the designed indictment of a dead shard, not a separate
fault.

Eight slices put every bin inside the budget; the affected lane derives
its matrix from the same registry, so it follows without edits.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/mutation/modules.ts">

<violation number="1" location="scripts/mutation/modules.ts:48">
P3: This comment names the wrong budget: a bin is killed by the 22-minute mutation step, not the 30-minute job. Also, the ~400-mutant figure differs from the PR description's "past 350". Reword the comment to cite the 22-minute step and drop the unverified mutant count, which will go stale as errors.ts grows.</violation>
</file>

Reply to a comment to ask cubic a question or push back. It learns from your replies.

View guided diff | Re-trigger cubic

Comment on lines +48 to +50
// errors.ts keeps growing (resource-ownership reasons landed ~400 mutants,
// and the slow-test region alone outran a 22-minute shard); eight slices
// hold every bin inside the lane's 30-minute job budget.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This comment names the wrong budget: a bin is killed by the 22-minute mutation step, not the 30-minute job. Also, the ~400-mutant figure differs from the PR description's "past 350". Reword the comment to cite the 22-minute step and drop the unverified mutant count, which will go stale as errors.ts grows.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/mutation/modules.ts, line 48:

<comment>This comment names the wrong budget: a bin is killed by the 22-minute mutation step, not the 30-minute job. Also, the ~400-mutant figure differs from the PR description's "past 350". Reword the comment to cite the 22-minute step and drop the unverified mutant count, which will go stale as errors.ts grows.</comment>

<file context>
@@ -45,7 +45,10 @@ export const KERNEL_MODULES: readonly KernelModule[] = [
     mutate: ['packages/kernel/src/errors.ts'],
     owns: ['packages/kernel/src/errors.ts'],
-    shards: 5,
+    // errors.ts keeps growing (resource-ownership reasons landed ~400 mutants,
+    // and the slow-test region alone outran a 22-minute shard); eight slices
+    // hold every bin inside the lane's 30-minute job budget.
</file context>
Suggested change
// errors.ts keeps growing (resource-ownership reasons landed ~400 mutants,
// and the slow-test region alone outran a 22-minute shard); eight slices
// hold every bin inside the lane's 30-minute job budget.
// Eight slices keep each bin inside the 22-minute mutation step timeout; the
// 30-minute job also covers setup and upload, so it is not the binding limit.

@strzhao strzhao closed this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant