Skip to content

chore: update dependencies, fix nix builds, speed up mutation gate - #119

Merged
ryanleecode merged 7 commits into
masterfrom
chore/dependency-updates
Oct 1, 2026
Merged

ryanleecode merged 7 commits into
masterfrom
chore/dependency-updates

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Rust, npm, and Deno dependencies move to their latest mutually compatible versions, and the classifier mutation gate drops from over 17 minutes to under 2. The gate was slow because tree-sitter-language-pack recompiled every grammar on every cargo command. With that fixed, no-op cargo commands take about 50ms instead of about 25s. Two Nix problems are fixed along the way: flake builds on hosts without a Nix sandbox no longer fail on every rebuild, and the Deno tools under scripts/tools/ now run inside the dev shell.

Version decisions

Dependency Change Constraint
tree-sitter 0.26 → 0.27 Required by tree-sitter-language-pack 1.19+
tree-sitter-language-pack 1.14.3 → 1.20.0 Ships memory-safety patches for three grammars we compile (see below)
MSRV (rust-version) 1.85 → 1.90 tree-sitter 0.27 declares 1.90
effect, @effect/platform-node 4.0.0-rc.112 → rc.118 Stays on the v4 RC line; npm's latest tag is 3.22.2
@systemfsoftware/tsconfig ^1.3.3 → ^2.0.1 Major bump; tsc -b passes unchanged
turbo ^2.10.12 → ^2.11.5

@effect/tsgo, tsdown, oxlint, @types/node, the Deno imports in scripts/ and hooks/, and flake.lock (nixpkgs, rust-overlay) move to latest as well. @std/fs and @std/path are dropped from hooks/deno.jsonc: nothing in hooks/run.ts imports them.

Why not stay on tree-sitter-language-pack 1.14.3

Every release from 1.15.0 on has the rebuild loop described below, so staying on 1.14.3 would avoid it without any workaround. It would also keep three known heap-buffer overflows in grammars this binary compiles and runs on every agent write. The python, yaml, and markdown external scanners write past tree-sitter's fixed serialization buffer on deeply nested or indented input, and the only bounds check compiles away in release builds. 1.15.0+ patches all three.

The rebuild loop

With TSLP_LANGUAGES set, the crate's build script looks for its cache at OUT_DIR/_parsers/<lang>/src/parser.c, but the bundle extracts to _parsers/parsers/<lang>/…. The check never matches, so every build-script run re-downloads the 36 MB bundle and re-extracts it. The re-extraction bumps the mtime of _parsers/patches/, which the script watches with rerun-if-changed, so the next cargo command runs the build script again. This is the same class of bug upstream fixed in xberg-io/tree-sitter-language-pack#158.

Pointing PROJECT_ROOT at a pre-extracted bundle sends the build script down its workspace path instead. It applies the grammar patches once (they are idempotent) and then stays fresh.

  • Dev shell: extracts the flake's hash-pinned bundle once into $XDG_CACHE_HOME/comment-checker/tslp-<version> and exports PROJECT_ROOT, but only when Cargo.lock names the pinned version. A stale root would compile another release's grammar sources without any error.
  • CI: the new .github/actions/tslp-project-root action reads the version from Cargo.lock, verifies the bundle against the release's .sha256 sidecar the same way build.rs does, and exports PROJECT_ROOT. rust-gate and mutation use it.
  • Without PROJECT_ROOT (for example, the release builds in platform.yml), builds fall back to the old path: slow, but correct.

Mutation setup

Change Full gate
Before (the rebuild loop on every mutant's build and test phase) 17+ min
Rebuild loop fixed 144s
+ [profile.mutants] with debug = "none" 115s
+ -j 2 under a 4-task jobserver cap, instead of -j 4 102s
  • CPU policy. cargo-mutants runs a jobserver that allows one compile task per CPU across all -j jobs, so the earlier -j nproc/2 never capped CPU at all. --jobserver-tasks now does: half the CPUs locally, all of them in CI (CARGO_MUTANTS_JOBSERVER_TASKS). -j defaults to 2 everywhere, because more parallel mutants measured slower.
  • CI install. cargo-mutants is pinned to 27.1.0, built with --locked, and the binary is cached by version, instead of being compiled from source on every run.
  • turbo. turbo runs tasks in strict env mode, which dropped XDG_CACHE_HOME and DENO_DIR. On any host that sets either, seven hook_commands tests failed in the unmutated baseline, so the gate never tested a mutant. turbo now passes those, the jobserver variable, and PROJECT_ROOT through without hashing them. It hashes .cargo/mutants.toml.

Measured and not adopted: the mold linker (125s against 115s without it), nextest (the suite runs in about 0s), and CI sharding (a full run is now under 2 minutes).

Code the updates forced

  • rc.118 moved effect/unstable/process and effect/unstable/cli to effect/process and effect/cli, and renamed Flag.string to Flag.String. The launcher's imports follow.
  • The MSRV raise turns on clippy's MSRV-gated collapsible_if lint, which fails two nested ifs under -D warnings. They become let-chains in classify.rs and main.rs, with identical control flow.
  • tree-sitter-language-pack 1.20.0 refuses a parser-source bundle with no .sha256 file beside it. flake.nix now writes one from the same pinned digest.

Nix fixes

  • Without a Nix sandbox (the macOS default), cargo inside the flake build treated Nix's placeholder HOME=/homeless-shelter as a real directory. Every later build then failed with "home directory /homeless-shelter exists", which also broke direnv and the repo's PostToolUse hook. The build now sets a private HOME. This fix is visible to consumers, so it carries a patch changeset.
  • The dev shell exports LD_FOR_BUILD, and Deno refuses to spawn a subprocess under a scoped --allow-run while any LD_* variable is set. shellHook now unsets it.

AGENTS.md

Two edits to this locked file, both at the maintainer's direction. turbo 2.11.5 appends its managed agent-guidance block and re-adds it on every agent run, so the block is committed. The mutation gate is now documented as pnpm mutants, and the concurrency rules describe the jobserver policy above.

Release

.changeset/update-deps.md is none, and .changeset/flake-build-home.md is patch. Merging opens a release PR for 0.3.6.

Validation

All of the following ran against the pushed head (95f1bc6d1), inside the dev shell:

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings, and cargo test --all-targets (128 tests) pass.
  • pnpm gate:mutants tested 118 mutants: 113 caught, 5 unviable, 0 missed, in 104s.
  • pnpm lint, pnpm build, and pnpm typecheck pass with turbo's cache bypassed, and pnpm install --frozen-lockfile passes. The built launcher prints --help and names the missing platform package.
  • deno lint, deno check tools/*.ts, check-matrix, lint-workflows, and deno cache --frozen for the hook all pass.
  • nix build .#comment-checker builds and leaves no /homeless-shelter. The built binary flags # increment i above i += 1 as a restatement. nix flake check --no-build passes.
  • The CI action's script was run locally. Against the real release it exported a populated root. With a bundle tampered by one byte it failed on the sha256 check and exported nothing.
  • The dev shell's version guard was run against fake lockfiles. A matching version exports the cached root. For 1.21.0 it warns, exports nothing, and clears an inherited PROJECT_ROOT.
  • cargo install --locked cargo-mutants@27.1.0 succeeds.

Not run: the Nix build on aarch64 or Darwin, and the CI workflows themselves, including the cached-install path.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Rust: tree-sitter 0.26 -> 0.27, tree-sitter-language-pack 1.14 -> 1.20
(the pack's 1.19+ line requires tree-sitter ^0.27). Declared MSRV
1.85 -> 1.90 to match tree-sitter 0.27's floor; the raised MSRV
unlocked clippy's msrv-gated collapsible_if, so two nested ifs became
let-chains (classify.rs, main.rs) — semantics unchanged, mutation score
stays 118/118 (113 caught, 5 unviable, 0 missed). flake.nix parser-sources
pin moved to v1.20.0 with the matching sha256.

npm: effect + @effect/platform-node 4.0.0-rc.112 -> rc.118 (peer-matched
pair), oxlint 1.86.0, turbo 2.10.12 -> 2.11.5, @systemfsoftware/tsconfig
1.3.3 -> 2.0.1, tsdown 0.23.0. rc.118 moved effect/unstable/{process,cli}
to effect/process + effect/cli and renamed Flag.string -> Flag.String, so
the launcher imports were updated. turbo 2.11.5 appends its managed
agent-guidance block to AGENTS.md; kept committed per turbo's guidance.

Deno: scripts/ and hooks/ imports refreshed (effect/platform-deno rc.118,
@std/*, arktype 2.2.5, @libs/diff 4.0.1) with regenerated lockfiles.

Tracked as a none changeset — no observable behaviour change.
…ed nix leak

flake.nix: tree-sitter-language-pack 1.20.0's build.rs hard-requires a
`<bundle>.sha256` sidecar, so the previous commit's bare fetchurl bundle
failed to build. Serve the bundle from a runCommand dir with a sidecar
written from the same pinned digest (tslpVersion/tslpSha256 are now the
single source). Also export HOME inside the build: cargo defaulted
CARGO_HOME to nix's HOME=/homeless-shelter, which without a sandbox is the
real host path, so every rebuild after the first failed nix's purity
check (and with it direnv and the repo's PostToolUse hook).

flake.lock: nixpkgs and rust-overlay updated to latest.

hook_commands tests: pass XDG_CACHE_HOME through to the sandboxed deno.
The warm-cache probe runs with the full environment, so on hosts with
XDG_CACHE_HOME set it checked a different cache than the tests used.
@effect/tsgo 0.47.0 was inside pnpm's minimum release age during the
previous refresh; it is now installable.

hooks/run.ts imports only @std/io and arktype; @std/fs and @std/path were
declared in hooks/deno.jsonc but never used. Removed, and deno.lock
regenerated from run.ts's real import graph (deno cache --frozen passes).
Mutation concurrency lives in the root `mutants` script: -j comes from
CARGO_MUTANTS_JOBS, defaulting to half of nproc (getconf fallback for
macOS). CI sets CARGO_MUTANTS_JOBS=$(nproc). turbo passes the variable
through without hashing it, so CI and local runs share cached verdicts.
AGENTS.md now names `pnpm mutants` as the gate so direct runs get the
policy instead of cargo-mutants' single-job default.

devShell: stdenv exports LD_FOR_BUILD, and Deno refuses to spawn under a
scoped --allow-run while any LD_* var is set, so every scripts/tools/*.ts
that shells out (lint-workflows, plan-release, ...) failed inside the
dev shell. Unset it in shellHook.
turbo runs tasks in strict env mode, so //#mutants dropped XDG_CACHE_HOME
and DENO_DIR. On a host that sets either, deno then looked for hooks/run.ts's
cached deps under $HOME/.cache/deno, missed the warmed cache, and the
unmutated baseline failed seven hook_commands tests before any mutant ran.
Neither variable changes a verdict, so pass them through unhashed.
…command

tree-sitter-language-pack 1.15.0+ never finds its own OUT_DIR parser cache
when TSLP_LANGUAGES is set: the probe checks `_parsers/<lang>/src/parser.c`
while the bundle extracts to `_parsers/parsers/<lang>/...`. Every build
script run therefore re-downloads and re-extracts the 36 MB bundle, and the
extraction bumps the mtime of `_parsers/patches/`, which the script watches
with rerun-if-changed. So every cargo command recompiled all 37 grammars
(~22-26s even for a no-op build) and needed network access, and each
mutant paid it twice. 1.14.3 had the same probe but did not watch
patches/; staying on it would ship the python/yaml/markdown scanner
heap-overflow fixes that 1.15+ patches in.

Setting PROJECT_ROOT at a pre-extracted bundle makes build.rs take its
workspace path instead. It applies the patches once (they are idempotent)
and then stays fresh: no-op cargo commands drop to ~50ms.

- devShell: extract the flake's hash-pinned bundle once into
  $XDG_CACHE_HOME/comment-checker/tslp-<version> and export PROJECT_ROOT,
  only when Cargo.lock names the pinned version; a stale root would compile
  another release's grammar sources silently.
- CI: .github/actions/tslp-project-root derives the version from
  Cargo.lock, verifies the bundle against the release's .sha256 sidecar
  (as build.rs does), and exports PROJECT_ROOT. Used by rust-gate here and
  by mutation in the next commit.
- [profile.mutants] (debug = "none"), selected in .cargo/mutants.toml:
  debug info only slows mutant builds, and a caught mutant never needs a
  backtrace. Full gate 144s -> 115s.
- CPU policy: cargo-mutants' jobserver already allows one compile task per
  CPU across all -j jobs, so `-j nproc/2` never capped CPU.
  --jobserver-tasks now does: half the CPUs locally, all of them in CI
  (CARGO_MUTANTS_JOBSERVER_TASKS). -j defaults to 2 everywhere: under a
  4-token cap, -j2 ran the full gate in 102s against 114s for -j4.
- CI: cargo-mutants is pinned and built with --locked, and the binary is
  cached by version instead of compiled from source on every run.
- mutation.yml provisions PROJECT_ROOT (previous commit); turbo passes it
  and the jobserver variable through unhashed, and hashes .cargo/mutants.toml.
@ryanleecode
ryanleecode merged commit d5bd46f into master Oct 1, 2026
12 checks passed
@ryanleecode
ryanleecode deleted the chore/dependency-updates branch October 1, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants