bench: make the benchmarks measure what they claim by lesnik512 · Pull Request #380 · modern-python/modern-di

lesnik512 · 2026-07-27T18:21:25Z

Fixes the nine measurement defects recorded in
planning/audits/2026-07-27-benchmark-fidelity-audit-report.md,
per planning/changes/2026-07-27.02-benchmark-measurement-fidelity.md,
plus three more found by review during the work. Nothing was fabricated — the old ratios
reproduced — the defects were in what the numbers meant.

The headline defect

C4 and G7 called loop.run_until_complete(...) once per timed iteration. That call costs
~27 µs on any body, while the real request cycle costs ~2 µs, so ~93% of every published
C4 cell was a shared asyncio floor that compressed real differences toward 1.0. Both now
time a batch of 100 cycles per loop entry, with test_g7c_event_loop_floor_control measuring
the residual in every run.

G7 was also useless as a guard: a 2x teardown regression moved it ~7%, far under the 150%
alert threshold.

What that changed, published as measured

A verdict flipped against us. C4 vs dishka went from "level with dishka (1.00)" to
1.58, dishka faster. The 1.00 was the floor, not a tie.

Levelling the timing shape moved 8 of 16 ratio cells worse, 7 better, 1 unchanged. The
page states this with every cell enumerated.

The other fixes

Per-framework fairness: dependency-injector was charged for a per-request container
deepcopy no other framework paid; modern-di was timed by reference against dishka/wireup
timed by type (C1–C3 now publish both variants against matching rivals); modern-di's
comparative subjects were dataclass(slots=True) against plain-class rivals; the
comparative tier never adopted the 3.x open() lifecycle.
iterations levelled (found in review): pytest-benchmark auto-calibrated per cell, so
some published cells sat on the ~42 ns timer grid paying full per-round overhead while
others didn't. Every cell is now pinned to an identical rounds × iterations, verified from
the run JSON.
The table is generated, not hand-assembled: just bench-report emits the exact
published markdown. Ratios are paired within each run and carry their own across-run IQR,
so a near-1.00 cell shows its spread instead of asserting "variation small".
Coverage: G16/G17 guard by-type resolve(SomeType) — the path integrations and
@inject take — at small and 200-provider scale.
Disclosure: dishka's default per-get() lock, the C5 registration asymmetry, and the
residual C4 basis asymmetry (which cuts against modern-di) are now stated.

Also found

Container stores itself in its own _scope_map, so every container is a reference cycle —
8 objects per container that only the GC can reclaim. Disabling the GC leaves the C4 median
unchanged but collapses within-run IQR from 51.4% to 3.5%. Audited and specced separately in
planning/changes/2026-07-27.03-container-reference-cycles.md;
no library code changed here.

Verification

just test-ci 449 passed at 100% coverage · just lint-ci clean · just docs-build clean ·
23 guard benchmarks passing, G7 floor share 15.7%.

🤖 Generated with Claude Code

The comparative and guard suites reproduce their published ratios, but C4/G7 time one run_until_complete entry per iteration, so ~93% of both numbers is the event-loop floor rather than DI work. Records that plus eight further defects (per-call container build for dependency-injector, by-reference vs by-type lookup, missing 3.x open(), undisclosed dishka lock default, non-reproducible table, by-type and scale coverage gaps, subject-class asymmetry, incomplete C5 caveat) and specs the fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A single table with four rival columns fills half of every by-reference or by-type row with n/a, which the spec ruled out. Publish one table per rival set instead.

One run_until_complete entry costs ~27us regardless of the body, so the old per-iteration shape reported ~29us for ~2us of request cycle -- a 2x teardown regression moved it ~7%, under any alert threshold. G7 now times 100 cycles per loop entry (floor share 92% -> 15%) and G7c measures the residual floor.

The docstring still claimed a single reused event loop keeps loop overhead out of the signal, which contradicts the batched G7 implementation and the corrected README. Rewritten to describe the K=100 batch shape, the ~27us loop entry cost isolation, and the G7c control benchmark.

@Inject

…scale Every existing scenario resolves by provider reference, skipping the find_provider lookup that resolve(SomeType) pays -- the path integrations and @Inject actually take, previously unmeasured at any registry size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three asymmetries that all ran in modern-di's favour: subject classes were dataclass(slots=True) against plain-class rivals (~10% cheaper to construct, inside the timed call), C1-C3 timed by reference against dishka/wireup timed by type (+11% when levelled), and the file never adopted the 3.x open() lifecycle the guard tier and docs use.

…or setup C4 timed one run_until_complete entry per request, so ~93% of every cell was a shared ~27us asyncio floor that compressed real differences toward 1.0. It now times 100 cycles per entry. dependency-injector additionally built a fresh container inside the timed call -- a per-request deepcopy no other framework paid, and no real app pays.

just bench-compare printed five rows all named test_c1_transient with no framework column, and nothing in the repo turned benchmark output into the published table -- so 'reproduce with just bench-compare' only worked for someone who already knew which row was which. Tests now carry framework suffixes and just bench-report emits the exact published markdown.

ruff formats python code blocks inside markdown, and planning/ is not in extend-exclude, so the unformatted block failed lint-ci's format gate.

dishka acquires a lock per get() at its default while modern-di's cached read is lock-free, and C5 charges dishka for provider registration modern-di does at import time (13% of its cell). Both were invisible to a reader of the table.

The C4 column was ~93% event-loop floor, dependency-injector was charged for a per-request container build, and modern-di was timed by reference against by-type rivals. Table regenerated with just bench-report on the fixed suite; C1-C3 now publish both lookup variants.

The tables and C4 flip were correct; the surrounding prose had drifted from them. Fixes: wireup's C1-C3 range excluded its own C3 minimum; the "gap largest on C2" and by-type-cost causal claims were asserted for both dishka and wireup when the arithmetic only supports each for one of them; the that-depends C2 explanation borrowed dependency-injector's C-level-slot mechanism, which cannot apply to a pure-Python framework; dependency-injector's C4 "dominated by lifecycle calls" claim wasn't backed by a decomposition; the reproduce block still pointed at bench-compare after the page switched to bench-report; the published 12.6% C1 by-type figure didn't match the table (12.3%); and the "each table matches API" claim silently over-scoped to C4, whose modern-di body resolves by reference against by-type-only rivals too.

The generated table asserted "run-to-run variation small" with nothing backing it. Each modern-di cell now carries its across-run IQR as a percentage of the median, and each table gets a footnote stating the worst spread seen among its rendered cells (modern-di and rivals separately) -- so dispersion is visible, not asserted. Ratio columns stay bare since a ratio of two medians has no clean IQR of its own.

Regenerated the three tables from just bench-report on the IQR-annotated generator (53bbf9f) so each modern-di cell states its own spread instead of the page asserting "run-to-run variation small" with nothing behind it. Recomputed every ratio and implied-cost figure in the prose against the fresh run; no verdict changed direction.

The overhead of resolve() vs resolve_provider() is a fixed ~40 ns dict-lookup cost; the percentage it represents varies by baseline, not a framework constant. This aligns the README's methodology documentation with the published performance results.

Each run measures both sides under the same machine state, so the correct statistic is the median of the per-run ratios, not a ratio of two independently-reduced medians. Pairing also gives every ratio a well-defined across-run IQR, now published as the same ±X.X% annotation the modern-di column already carries -- the reason those columns were bare is gone.

report.BATCH turns the C4 batch median into the published per-request figure. Changing K in the five comparative files without changing report.py publishes a number wrong by that factor with nothing failing; this reads the sources as text (the comparative venv is separate) and asserts the literals agree.

pytest-benchmark auto-calibrates `iterations` per benchmark per run, which left C1/C3 cells for modern-di, dishka and dependency-injector at iterations=1 while that-depends and wireup ran at 20-26. An iterations=1 median carries the whole per-round timer pair and lands on the platform timer's ~42 ns grid; a cell at iterations>1 amortizes both away. A published ratio must not divide a grid-snapped number by an unsnapped one. C1-C4 now use `benchmark.pedantic` at a shape pinned identically in all five files: C1-C3 at rounds=200 iterations=1000, C4 at rounds=100 iterations=3 (its callable is already a batch of K=100 cycles). `warmup_rounds=1` stands in for the warm-up calibration used to provide; all other setup stays outside the timed call. C5/C6 are not published and keep auto-calibration. The new AST-based guard in tests/test_bench_report.py fails if a published scenario drops off `pedantic`, uses the wrong constants, or if the five files' values drift apart.

The invariant that every published cell is timed at the same rounds x iterations is only checkable from the pytest-benchmark JSON, which main() discarded with its temp dir. `--json-dir` keeps the run payloads so the claim can be audited against the same data that produced the table.

… moved States the shape, why it is pinned, and that it moved published cells in both directions -- dishka shed proportionally more per-round overhead than modern-di did, so modern-di's C1/C3 ratios against dishka got worse. Also records the C4 estimator shift: at iterations>1 a round is the mean of several batches, so a right-skewed distribution reports nearer its mean. modern-di's C4 mean did not move (245.9 us -> 248.0 us), so that is a change of estimator, not of work.

…defects Tables regenerated verbatim from `just bench-report 5` on the pinned timing shape. Numbers moved against modern-di where the measurement says so: by-type vs dishka C1 1.28 -> 1.45 and C3 1.75 -> 1.87, C4 vs dishka 1.30 -> 1.58, and modern-di's C4 figure 2.05 -> 2.39 us. A "where this run moved against modern-di" paragraph states that plainly. C1 vs that-depends reads 0.98 +-2.5%, a spread that covers 1.00, so it is called a tie rather than a win. Prose fixes: - the batching amortizes the asyncio floor, it does not remove it: ~0.31 us per request remains in every C4 cell, measured by g7c_event_loop_floor_control - adds the missing thread-safety disclosure (dishka's per-get _thread.lock against modern-di's lock-free cached read, both at their defaults) - scopes the exec-codegen mechanism to dishka, whose ratio grows with node count as a per-node cost should; wireup's shrinks with depth, so nothing is claimed - re-derives the by-type lookup cost off the timer grid: 47/41/45 ns on C1/C2/C3 - gives the C4 basis asymmetry its direction (it cuts against modern-di: levelling would move dishka 1.58 -> 1.61) - labels the C2 decomposition as an inferred cross-basis figure - replaces the "ratio columns are bare" note, now that ratios are paired per run and carry their own across-run IQR The bundle summary is updated to the numbers actually published.

…s' spread Adding a paired-ratio ± to every ratio cell left two different across-run IQRs on one table under one name: the footnote bounds each side's OWN median, the cell's ± bounds the paired per-run ratios. In the by-reference table that put "rivals <=1.5%" directly under cells reading ±2.0%, ±2.5% and ±3.3%, so the footnote read as contradicting the table above it. Both quantities are kept -- the footnote is the only place rival dispersion is disclosed at all -- and are now labelled distinctly. Only the label changed; both bounds the test asserts (13.6% / 7.1%) are unchanged.

…duce The "against wireup and the two by-reference rivals the levelling moved slightly in modern-di's favour" sentence was false for five published cells -- C1 and C3 vs dependency-injector, C3 vs that-depends, and C4 vs that-depends and wireup all moved against modern-di. In a paragraph whose whole purpose is disclosing adverse movement, that overstated in the one direction it must not. Replaced with the counted split: eight of sixteen ratio cells worse, seven better, one unchanged, naming what actually improved (wireup C1/C3, every C2 cell, C1 vs that-depends) and what did not. Figures now reproduce from the published cells: - by-type lookup cost 47/43/45 ns at 16%/23%/6%, rounded in one direction (15.5 had been rounded down while 5.6 was rounded up, and the C2 delta came from the raw median rather than the cells) - that-depends' implied C2 cache hit ~65 ns (187 / 2.86), not ~67 - the C2 cross-basis figures are now the paired per-run ratios that the rest of the page uses (1.10 vs dishka, 2.45 vs wireup) instead of a subtraction whose arithmetic did not compose back to the published cells - the footnotes carry the generator's new label, tables otherwise byte-identical benchmarks/README.md drops "each shed ~30-50 ns", which overstated uniformity: the range is ~10 ns to ~50 ns and one cell rose.

Every Container stores itself in its own _scope_map, so none can be freed by refcounting -- 8 objects per container wait for the cycle collector. Found while investigating why modern-di's C4 benchmark cell was the least stable on the page: disabling the GC leaves the median unchanged but collapses within-run IQR from 51.4% to 3.5%. find_container never reads the self-entry, so seeding the map from the parent removes the cycle without touching the hot path.

github-actions

Benchmark

Details

Benchmark suite	Current: `b1ddc73`	Previous: `1e9772b`	Ratio
`benchmarks/test_guard_by_type.py::test_g16_resolve_by_type`	`1761011.3173065581` iter/sec (`stddev: 2.3780199317185235e-7`)
`benchmarks/test_guard_by_type.py::test_g17_resolve_by_type_large_registry`	`2063944.1702857611` iter/sec (`stddev: 7.108020965872546e-8`)
`benchmarks/test_guard_cold.py::test_g8_cold_first_resolve`	`20692.376618761158` iter/sec (`stddev: 0.00017918945428745234`)	`19326.255630016058` iter/sec (`stddev: 0.00017999729726384398`)	`0.93`
`benchmarks/test_guard_concurrency.py::test_g14_concurrent_cached_hit[1]`	`338.42354837297626` iter/sec (`stddev: 0.00004197968016534363`)	`312.48466872091177` iter/sec (`stddev: 0.000045427428651010436`)	`0.92`
`benchmarks/test_guard_concurrency.py::test_g14_concurrent_cached_hit[2]`	`319.1366450472165` iter/sec (`stddev: 0.0002917546562032856`)	`304.46717600690494` iter/sec (`stddev: 0.000034307600019111886`)	`0.95`
`benchmarks/test_guard_concurrency.py::test_g14_concurrent_cached_hit[4]`	`301.5615684477255` iter/sec (`stddev: 0.00004345059222459768`)	`283.640877868282` iter/sec (`stddev: 0.00004728868633776039`)	`0.94`
`benchmarks/test_guard_concurrency.py::test_g15_concurrent_first_resolve[1]`	`1374.1787349586052` iter/sec (`stddev: 0.0015068423342919912`)	`1554.7710065878698` iter/sec (`stddev: 0.0004861497264035832`)	`1.13`
`benchmarks/test_guard_concurrency.py::test_g15_concurrent_first_resolve[2]`	`1323.1216130368598` iter/sec (`stddev: 0.0005440383856053636`)	`1056.8622914078476` iter/sec (`stddev: 0.0013728631586097696`)	`0.80`
`benchmarks/test_guard_concurrency.py::test_g15_concurrent_first_resolve[4]`	`792.5671991317571` iter/sec (`stddev: 0.0020240946584078943`)	`885.5747925038461` iter/sec (`stddev: 0.0006032502721753777`)	`1.12`
`benchmarks/test_guard_lifecycle.py::test_g6_build_child_container`	`480708.8461526863` iter/sec (`stddev: 0.000009915924252264138`)	`461183.2365367938` iter/sec (`stddev: 0.00004295575077322861`)	`0.96`
`benchmarks/test_guard_lifecycle.py::test_g6b_build_child_container_auto_scope`	`459026.2930802838` iter/sec (`stddev: 0.000010071661918185067`)	`462702.60431401915` iter/sec (`stddev: 0.00000989318951724684`)	`1.01`
`benchmarks/test_guard_lifecycle.py::test_g7_request_lifecycle_batch`	`1831.9246638483362` iter/sec (`stddev: 0.00010142727206084961`)
`benchmarks/test_guard_lifecycle.py::test_g7c_event_loop_floor_control`	`61020.93614640296` iter/sec (`stddev: 0.0000014206936327022418`)
`benchmarks/test_guard_lifecycle.py::test_g13_teardown_at_scale`	`40476.3565179226` iter/sec (`stddev: 0.000023552661422462476`)	`42106.66024063769` iter/sec (`stddev: 0.000021517231783469453`)	`1.04`
`benchmarks/test_guard_resolve.py::test_g1_transient_resolve`	`1188042.2598015838` iter/sec (`stddev: 3.7655073176811475e-7`)	`1215812.0707366678` iter/sec (`stddev: 3.438400399584913e-7`)	`1.02`
`benchmarks/test_guard_resolve.py::test_g2_cached_resolve`	`2345696.675629734` iter/sec (`stddev: 9.115821240454069e-8`)	`2782344.866286612` iter/sec (`stddev: 5.411759845713959e-8`)	`1.19`
`benchmarks/test_guard_resolve.py::test_g3_deep_chain`	`488161.4576590155` iter/sec (`stddev: 6.659434982028826e-7`)	`483179.3397999431` iter/sec (`stddev: 5.464708210173301e-7`)	`0.99`
`benchmarks/test_guard_resolve.py::test_g4_wide_resolve`	`307978.61590606126` iter/sec (`stddev: 6.921498888378652e-7`)	`306457.3317270641` iter/sec (`stddev: 7.313391134673951e-7`)	`1.00`
`benchmarks/test_guard_resolve.py::test_g5_cross_scope`	`1053883.625958277` iter/sec (`stddev: 4.359325720962046e-7`)	`1067323.0513755598` iter/sec (`stddev: 4.366812887869296e-7`)	`1.01`
`benchmarks/test_guard_resolve.py::test_g9_context_resolve`	`561516.2251046948` iter/sec (`stddev: 5.145131399815818e-7`)	`561226.1184838566` iter/sec (`stddev: 5.463566726807361e-7`)	`1.00`
`benchmarks/test_guard_resolve.py::test_g12_override_active_resolve`	`378784.99166710215` iter/sec (`stddev: 4.5848494460226515e-7`)	`370096.5096003888` iter/sec (`stddev: 4.943904911034173e-7`)	`0.98`
`benchmarks/test_guard_validate.py::test_g10_validate_deep_chain`	`27197.599097448732` iter/sec (`stddev: 0.00006553349786272064`)	`25722.24858746826` iter/sec (`stddev: 0.000024894308623793018`)	`0.95`
`benchmarks/test_guard_validate.py::test_g11_validate_wide`	`18201.57958148899` iter/sec (`stddev: 0.000010682488289732963`)	`16738.51469518524` iter/sec (`stddev: 0.000014214761119129636`)	`0.92`

This comment was automatically generated by workflow using github-action-benchmark.

lesnik512 and others added 25 commits July 27, 2026 17:35

docs(planning): three tables, not one, for the split lookup axis

f4de427

A single table with four rival columns fills half of every by-reference or by-type row with n/a, which the spec ruled out. Publish one table per rival set instead.

style(benchmarks): apply ruff format to the by-type guard file

62f4572

style(planning): ruff-format the python block in the change bundle

e40c830

ruff formats python code blocks inside markdown, and planning/ is not in extend-exclude, so the unformatted block failed lint-ci's format gate.

github-actions Bot reviewed Jul 27, 2026

View reviewed changes

lesnik512 merged commit 5c46fd6 into main Jul 27, 2026
9 checks passed

lesnik512 deleted the bench-measurement-fidelity branch July 27, 2026 18:36

lesnik512 mentioned this pull request Jul 27, 2026

bench: publish C6, the per-request context scenario #382

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bench: make the benchmarks measure what they claim - #380

bench: make the benchmarks measure what they claim#380
lesnik512 merged 25 commits into
mainfrom
bench-measurement-fidelity

lesnik512 commented Jul 27, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

lesnik512 commented Jul 27, 2026

The headline defect

What that changed, published as measured

The other fixes

Also found

Verification

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Benchmark

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant