Skip to content

fix(bench): trim real_repo sampling; restore sample_size(10) criterion floor - #2695

Merged
max-sixty merged 1 commit into
mainfrom
bench-real-repo-clone
May 11, 2026
Merged

max-sixty merged 1 commit into
mainfrom
bench-real-repo-clone

Conversation

@max-sixty

Copy link
Copy Markdown
Owner

Summary

Two changes to benches/list.rs, no production code.

1. Trim bench_real_repo sampling. It inherited the criterion_group default sample_size(30) / measurement_time(15s). On rust-lang/rust wt list runs ~2s warm (dominated by one deep git for-each-ref %(ahead-behind:main) walk) and several times that for cold/8, where every iteration also rebuilds eight 59k-entry git indexes via git status. At >1s/iter Criterion can't fit 30 samples in 15s, so it just runs 30 single-iteration samples per variant — the group as a whole runs ~15–20 min, dominated by real_repo/cold/{4,8}. Set sample_size(10) (criterion 0.8's hard minimum) + measurement_time(20s), mirroring what #2685 did for bench_real_repo_many_branches. The expensive cold variants drop from 30 to 10 iterations → group's measured time roughly thirds (~6–8 min). No change to what wt list runs or measures.

2. Fix a latent panic in bench_real_repo_many_branches. #2685 set it to sample_size(5), but criterion 0.8's sample_size does assert!(n >= 10) (benchmark_group.rs:97), so cargo bench --bench list currently panics on main during group registration — for any filter, before any benchmark runs. Restored to sample_size(10). (This was also a prerequisite for running bench_real_repo to validate change #1.)

Why not sparse-checkout the rust clone

A sparse checkout would make the per-variant setup near-instant and git status / git diff --shortstat HEAD on the rust worktrees ~10ms instead of ~1.8s / ~0.8s — but those costs are exactly what the cold (cold 59k-entry index rebuild) and warm (working-tree scan over 59k files) variants exist to measure. Sparse-checkout would turn "cold index on a huge repo" into "cold index on ~20 files", erasing the documented cold ~4× insight. The mechanism works; it's the wrong tradeoff. Sharing one rust workspace across the warm/cold pair was also considered and dropped — after the sampling trim, setup is a small slice (~1.5–2 min of ~7 min), and sharing would require up to ~10 GB of simultaneous rust checkouts (CI disk risk) for a marginal gain.

Before / after

real_repo/warm/1 (rust-lang/rust clone, one worktree): median 1.809s (30 samples) → 1.794s (10 samples). Criterion's regression check on the after run: change: [−23.1% −7.0% +10.8%] (p = 0.59) → "No change in performance detected." The cold variants — the bulk of the group — drop 3× by arithmetic (30→10 single iterations of the same operation).

Test plan

  • cargo run -- hook pre-merge --yes — 3645 tests pass, clippy + fmt + pre-commit hooks pass
  • cargo bench --bench list -- 'real_repo/warm/1' — runs cleanly, sample_size(10) accepted ("Collecting 10 samples"), result above
  • cargo clippy --benches -- -D warnings, cargo check --benches, cargo fmt --check

…n floor

`benches/list.rs::bench_real_repo` inherited the `criterion_group` default
`sample_size(30)` / `measurement_time(15s)`. On rust-lang/rust `wt list` runs
~2s warm and several times that for cold/8 (every iteration rebuilds eight
59k-entry git indexes via `git status`), so Criterion can't fit 30 samples in
15s — it just runs 30 single-iteration samples per variant, ~15-20 min for the
group. Set `sample_size(10)` (criterion 0.8's hard minimum) + `measurement_time(20s)`,
mirroring what #2685 did for `bench_real_repo_many_branches`. Cuts the group's
measured time ~3× (the expensive cold variants drop 30→10 iterations); no change
to what `wt list` runs or measures — `real_repo/warm/1` stays at ~1.8s median,
Criterion reports "No change in performance detected".

Also fixes a latent panic introduced by #2685: `bench_real_repo_many_branches`
was set to `sample_size(5)`, but criterion 0.8's `sample_size` does
`assert!(n >= 10)`, so `cargo bench --bench list` currently panics on `main`
during group registration (any filter). Restored to `sample_size(10)`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@max-sixty
max-sixty merged commit 706fb5a into main May 11, 2026
34 checks passed
@max-sixty
max-sixty deleted the bench-real-repo-clone branch May 11, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants