Repository navigation
fix(bench): trim real_repo sampling; restore sample_size(10) criterion floor - #2695
Merged
Merged
Conversation
…n floor `benches/list.rs::bench_real_repo` inherited the `criterion_group` default `sample_size(30)` / `measurement_time(15s)`. On rust-lang/rust `wt list` runs ~2s warm and several times that for cold/8 (every iteration rebuilds eight 59k-entry git indexes via `git status`), so Criterion can't fit 30 samples in 15s — it just runs 30 single-iteration samples per variant, ~15-20 min for the group. Set `sample_size(10)` (criterion 0.8's hard minimum) + `measurement_time(20s)`, mirroring what #2685 did for `bench_real_repo_many_branches`. Cuts the group's measured time ~3× (the expensive cold variants drop 30→10 iterations); no change to what `wt list` runs or measures — `real_repo/warm/1` stays at ~1.8s median, Criterion reports "No change in performance detected". Also fixes a latent panic introduced by #2685: `bench_real_repo_many_branches` was set to `sample_size(5)`, but criterion 0.8's `sample_size` does `assert!(n >= 10)`, so `cargo bench --bench list` currently panics on `main` during group registration (any filter). Restored to `sample_size(10)`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
worktrunk-bot
approved these changes
May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes to
benches/list.rs, no production code.1. Trim
bench_real_reposampling. It inherited thecriterion_groupdefaultsample_size(30)/measurement_time(15s). On rust-lang/rustwt listruns ~2s warm (dominated by one deepgit for-each-ref %(ahead-behind:main)walk) and several times that forcold/8, where every iteration also rebuilds eight 59k-entry git indexes viagit status. At >1s/iter Criterion can't fit 30 samples in 15s, so it just runs 30 single-iteration samples per variant — the group as a whole runs ~15–20 min, dominated byreal_repo/cold/{4,8}. Setsample_size(10)(criterion 0.8's hard minimum) +measurement_time(20s), mirroring what #2685 did forbench_real_repo_many_branches. The expensive cold variants drop from 30 to 10 iterations → group's measured time roughly thirds (~6–8 min). No change to whatwt listruns or measures.2. Fix a latent panic in
bench_real_repo_many_branches. #2685 set it tosample_size(5), but criterion 0.8'ssample_sizedoesassert!(n >= 10)(benchmark_group.rs:97), socargo bench --bench listcurrently panics onmainduring group registration — for any filter, before any benchmark runs. Restored tosample_size(10). (This was also a prerequisite for runningbench_real_repoto validate change #1.)Why not sparse-checkout the rust clone
A sparse checkout would make the per-variant setup near-instant and
git status/git diff --shortstat HEADon the rust worktrees ~10ms instead of ~1.8s / ~0.8s — but those costs are exactly what thecold(cold 59k-entry index rebuild) andwarm(working-tree scan over 59k files) variants exist to measure. Sparse-checkout would turn "cold index on a huge repo" into "cold index on ~20 files", erasing the documentedcold ~4×insight. The mechanism works; it's the wrong tradeoff. Sharing one rust workspace across the warm/cold pair was also considered and dropped — after the sampling trim, setup is a small slice (~1.5–2 min of ~7 min), and sharing would require up to ~10 GB of simultaneous rust checkouts (CI disk risk) for a marginal gain.Before / after
real_repo/warm/1(rust-lang/rust clone, one worktree): median 1.809s (30 samples) → 1.794s (10 samples). Criterion's regression check on the after run:change: [−23.1% −7.0% +10.8%] (p = 0.59)→ "No change in performance detected." The cold variants — the bulk of the group — drop 3× by arithmetic (30→10 single iterations of the same operation).Test plan
cargo run -- hook pre-merge --yes— 3645 tests pass, clippy + fmt + pre-commit hooks passcargo bench --bench list -- 'real_repo/warm/1'— runs cleanly,sample_size(10)accepted ("Collecting 10 samples"), result abovecargo clippy --benches -- -D warnings,cargo check --benches,cargo fmt --check