Skip to content

[Perf] Windows/x64: 2 Regressions on 7/31/2026 4:48:31 PM +00:00 #131815

Closed
Closed
[Perf] Windows/x64: 2 Regressions on 7/31/2026 4:48:31 PM +00:00#131815
Assignees
joshuajyue
Labels
arch-x64area-System.Collectionsos-windowsruntime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Description

@performanceautofiler
performanceautofiler
bot
opened on Aug 4, 2026

Run Information

Name Value
Architecture x64
OS Windows 10.0.22631
Queue ViperWindows
Baseline ba8afceee83540ed1d867987483f74d4ad2cfae9
Compare d5c739f99fa75a37ea863b06dd8d31a51965c8be
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
21.52 ns 26.69 ns 1.24 0.21 False
16.70 ns 20.32 ns 1.22 0.28 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'
Details

System.Collections.Tests.Perf_BitArray.BitArraySetLengthGrow(Size: 4)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    added
    arch-x64
    os-windows
    runtime-coreclrspecific to the CoreCLR runtime
    untriagedNew issue has not been triaged by the area owner
    on Aug 4, 2026

  2. LoopedBard3 commented on Aug 4, 2026

    @LoopedBard3
    LoopedBard3
    on Aug 4, 2026
    Member

    🔍 Automated Triage Analysis

    Summary: Verified regression in BitArrayBoolArrayCtor(Size: 512) — caused by commit 44687f2 ("Add ReadOnlySpan constructors to BitArray (#131500)"). 1 other test(s) found unrelated commits (possible alignment artifacts). ⚠️ Note: The culprit commit could not be built/measured directly (no Helix build artifacts) — attribution is inferred from neighboring commits, so confidence is capped.

    Finding Confidence: 3/5 (analysis accuracy)
    Regression Confidence: 3/5 (likelihood of true regression)

    Likely Cause: 44687f2d763be239ff562b137360ab572f47066e

    🤖 Proposed Next Actions: Uncertain — flag for re-triage — 1 done, 1 dry-run (rule R5_look_again — see Full Analysis for the table). · 🦆 Re-analysis: ✅ concurs

    📊 Full Analysis (click to expand)

    Summary: Two System.Collections.BitArray microbenchmarks regressed (+22–24%) on Windows/x64 and both are confirmed step changes against a deep, stable history (298–299 prior runs). The likely cause is commit 44687f2 ("Add ReadOnlySpan constructors to BitArray"), which re-routed every array constructor through new private span-based CreateArray helpers — a direct code hit on the exact type under test. Finding confidence 4/5 (deep, low-variance history), regression confidence 4/5 (surgical BCL change, both tests corroborate, z up to 2.9).

    Issue Overview

    • Issue: [Perf] Windows/x64: 2 Regressions on 7/31/2026 4:48:31 PM +00:00 #131815 - [Perf] Windows/x64: 2 Regressions on 7/31/2026
    • Date: 2026-07-31 | Queue: Windows.11.Amd64.Viper.Perf | OS/Arch: Windows 10.0.22631/x64
    • Commits: ba8afce → d5c739f (15 commits)
    • Labels: untriaged, perf-regression, os-windows, arch-x64, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
    • Runtime stack: Desktop + CoreCLR (default); regression lives in managed BCL exercised directly by CoreCLR.
    • Historical Data: Available (298–299 prior runs per test; CV 5.7–7.9%)

    Test Analysis

    Test Name Baseline Compare Δ% μ σ n z Thresh% Noise? Assessment
    Perf_BitArray.BitArraySetLengthGrow(Size: 4) 21.52 26.69 +24.01% 22.75 1.80 299 2.19 15.82% N Step change; a few historical spike outliers inflate σ/threshold but after-series stays elevated
    Perf_BitArray.BitArrayBoolArrayCtor(Size: 512) 16.70 20.32 +21.68% 17.44 0.99 298 2.92 11.31% N Step change; stable history, all 16 after-samples elevated (20.3–25.7)

    Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

    Finding Confidence: 4/5 - Very deep historical baselines (298–299 runs), low variance for BoolArrayCtor (CV 5.65%), consistent 16-sample post-regression elevation, complete commit info. Not runtime-verified via RunWithBuildAtHash/bisection (handled separately), so held below 5.

    Candidate Stack Relevance

    Commit Touched Paths (summary) Relevance Notes
    44687f2 src/libraries/System.Private.CoreLib/.../Collections/BitArray.cs Relevant Rewrites the byte[]/bool[]/int[] constructors — the exact type and code paths both tests exercise
    0bbeda6 src/coreclr/jit/morph.cpp Shared JIT TYP_BYREF→TYP_INT normalization; could shift codegen but no BitArray-specific link
    ccd6004 src/coreclr/jit/emitxarch.*, instrsxarch.h Shared APX-EVEX emitter refactor; broad codegen surface, no targeted signal
    bc8571b src/coreclr/jit/valuenum.cpp Shared Pinned-region value numbering; unrelated to BitArray

    Confirmed Regressions

    System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512) (+21.7%)

    • Delta: 16.70 ns → 20.32 ns (+21.68%)
    • Historical Context: μ=17.44 (σ=0.99, n=298); z-score=2.92; adaptive threshold=11.31%
    • Assessment: Step change, stable history (CV 5.65%); all 16 after-run samples elevated with no recovery
    • Finding Confidence: 5/5 - Deep, low-variance baseline; unambiguous post-regression elevation
    • Regression Confidence: 4/5 - Direct code hit on the constructor invoked by this benchmark (new BitArray(_bools)); step change (z≥2); focused diff
    • Likely Cause: 44687f2 - "Add ReadOnlySpan constructors to BitArray (Add ReadOnlySpan constructors to BitArray #131500)"
      • Affected Areas: Collections (BCL BitArray constructors)
      • Stack Relevance: Relevant
      • Rationale: The bool[] constructor no longer allocates and fills inline; it now forwards to a new private CreateArray(ReadOnlySpan<bool>, out int) helper (via an implicit bool[]→ReadOnlySpan<bool> conversion), and the scalar remainder loop changed from values[i] to values[(int)i]. For a 512-element bool array the SIMD packing dominates, so the added call indirection / reduced inlining of the shared helper plausibly accounts for the ~3.6 ns step. Corroborated by the byte[]-ctor test regressing simultaneously.
      • Candidates Considered:
        • 0bbeda6 - JIT byref normalization; could affect codegen but no BitArray-specific evidence
        • ccd6004 - APX-EVEX emitter refactor; broad but untargeted
    • Verification Status: Not verified (runtime bisection/verification handled separately)

    System.Collections.Tests.Perf_BitArray.BitArraySetLengthGrow(Size: 4) (+24.0%)

    • Delta: 21.52 ns → 26.69 ns (+24.01%)
    • Historical Context: μ=22.75 (σ=1.80, n=299); z-score=2.19; adaptive threshold=15.82%
    • Assessment: Step change; occasional historical spikes (48.7, 30.9) inflate σ and threshold, but the after-series (24–32) is consistently elevated
    • Finding Confidence: 4/5 - Deep history, but higher variance (CV 7.9%) and threshold-inflating outliers lower certainty slightly
    • Regression Confidence: 4/5 - The benchmark builds new BitArray(_bytes) before growing; the byte[] constructor was refactored in the same commit (Array.Copy → private CreateArray(ReadOnlySpan<byte>) + bytes.CopyTo). At Size 4 the constructor cost dominates the tiny Length grow (unchanged), so the ctor refactor is the plausible driver.
    • Likely Cause: 44687f2 - "Add ReadOnlySpan constructors to BitArray (Add ReadOnlySpan constructors to BitArray #131500)"
      • Affected Areas: Collections (BCL BitArray constructors)
      • Stack Relevance: Relevant
      • Rationale: The byte[] constructor now forwards to CreateArray(ReadOnlySpan<byte>, out int), replacing Array.Copy(bytes, _array, len) with bytes.CopyTo(array) behind an extra helper call. The Length grow setter is untouched, so for a 4-byte input the changed construction path is the only regressing code in this commit that the test executes. Same commit and mechanism as the corroborating BoolArrayCtor regression.
      • Candidates Considered:
        • 0bbeda6 - JIT byref normalization; possible codegen effect, no direct link
        • bc8571b - JIT pinned-region VN change; unrelated
    • Verification Status: Not verified (runtime bisection/verification handled separately)

    Open Questions

    • Historical Data: Available and deep (298–299 runs); BitArraySetLengthGrow has a few spike outliers that inflate σ and the adaptive threshold.
    • Confidence Limitations: Attribution is inferred from a direct code hit, not yet runtime-verified. Deltas are modest in absolute ns terms (~3.6–5.2 ns).
    • Verification Needed: Confirm via RunWithBuildAtHash at 44687f2 vs its parent; optional bisection across the 15-commit range if the direct hit is not reproduced.
    • Bisection Status: Not performed (handled separately).

    Recommended Actions

    1. Verify suspected commit - Run BitArrayBoolArrayCtor(Size: 512) and BitArraySetLengthGrow(Size: 4) with RunWithBuildAtHash on 44687f2 and its parent to confirm the +22–24% step.
    2. Source review - Inspect the new private CreateArray(ReadOnlySpan<T>) helpers in BitArray.cs; check whether they inline for the array-ctor callers (e.g., [MethodImpl(AggressiveInlining)] or restoring the fast path for the byte[]/bool[] overloads).
    3. Follow-up - If confirmed, engage the PR author area (System.Collections) to restore constructor performance while keeping the new span overloads.

    Bisection Results

    Test Status Culprit Commit Relevance Reported Diff (issue) Bisected Diff (at culprit)
    BitArraySetLengthGrow(Size: 4) ✅ Culprit found 0bbeda6cbad4 ⚠️ LIKELY_UNRELATED 21.5→26.7 ns (+24%) 22.6→25.3 ns (+12%)
    BitArrayBoolArrayCtor(Size: 512) ✅ Culprit found 44687f2d763b ✅ RELATED 16.7→20.3 ns (+22%) ❌ Not measured (no build artifacts; inferred from neighbors)

    Total bisection time: 4643s
    Tests measured: 2/2
    Culprit found: 2/2

    ⚠️ Alignment Warning

    Bisection found commits that are unrelated to the regressed tests. This strongly suggests the performance changes are caused by code alignment, binary layout, or other non-functional factors rather than actual regressions. Consider marking as noise.

    System.Collections.Tests.Perf_BitArray.BitArraySetLengthGrow(Size: 4)

    • Culprit: 0bbeda6cbad4
    • Commit Message: Normalize TYP_BYREF to TYP_INT in implicit casts even on x86 (Normalize TYP_BYREF to TYP_INT in implicit casts even on x86 #131442)
    • Relevance: LIKELY_UNRELATED
    • Explanation: The commit modifies the JIT (src/coreclr/jit/morph.cpp), which is plausibly on the test's codegen path, but the only functional change is a relaxed noway_assert gated behind #else for 32-bit (non-TARGET_64BIT) targets. On the x64 configuration under test, the compiled code path is unchanged, so a genuine +2.7ns regression here is most consistent with code alignment/binary-layout shift from the 6 added source lines rather than a semantic change to the BitArray path.
    • ⚠️ This commit is UNRELATED to the test — likely a code alignment artifact

    Helix jobs

    Job ID Submitted by bisection Last status
    45ea3857-b02e-4ece-a7e8-f0ba33416978 ✅ submitted
    f1343eaf-67eb-448d-a735-59087e40eb59 ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    ba8afceee835 baseline / good 21.804 ns
    896a3828ea34 q1 boundary, idx 3 21.524 ns
    0e58de655dae mid, idx 7 22.609 ns
    44687f2d763b 3q boundary, idx 11, last good 22.601 ns
    0bbeda6cbad4 idx 12, culprit 25.284 ns
    ccd60043990d idx 13 25.339 ns
    d5c739f99fa7 head / regressed 25.605 ns
    Bisection details

    …[earlier output omitted]

    - Commits Tested: 8
- Confidence: 3
- Commit Relevance: LIKELY_UNRELATED
- Relevance Explanation: The commit modifies the JIT (src/coreclr/jit/morph.cpp), which is plausibly on the test's codegen path, but the only functional change is a relaxed noway_assert gated behind #else for 32-bit (non-TARGET_64BIT) targets. On the x64 configuration under test, the compiled code path is unchanged, so a genuine +2.7ns regression here is most consistent with code alignment/binary-layout shift from the 6 added source lines rather than a semantic change to the BitArray path.
- Baseline Value: 22.601 ns
- Culprit Value: 25.284 ns
- Summary: Bisected a 15-commit range with two multi-commit Helix jobs. Baseline/Q1/mid/3Q commits (ba8afce, 896a382, 0e58de6, 44687f2) all measured ~21.5–22.8 ns; the clean, reproducible step to ~25.3 ns first appears at 0bbeda6c and persists through ccd60043 and d5c739 (head), isolating 0bbeda6c as the introducing commit.

HELIX JOBS:
- 45ea3857-b02e-4ece-a7e8-f0ba33416978
- f1343eaf-67eb-448d-a735-59087e40eb59

MEASUREMENTS:
- ba8afceee835 (baseline / good): 21.804 ns
- 896a3828ea34 (Q1 boundary, idx 3): 21.524 ns
- 0e58de655dae (mid, idx 7): 22.609 ns
- 44687f2d763b (3Q boundary, idx 11, last good): 22.601 ns
- 0bbeda6cbad4 (idx 12, culprit): 25.284 ns
- ccd60043990d (idx 13): 25.339 ns
- d5c739f99fa7 (head / regressed): 25.605 ns

ALIGNMENT WARNING:
This bisection found a commit that does not appear related to the regression test.
This suggests the performance change may be due to code alignment, binary layout,
or other non-functional factors rather than a regression in the tested code path.

    Note: the 2σ noise floor (≈3.6 ns) is larger than the observed step (~2.7 ns), so per the noise-floor guidance I capped Confidence at 3/5 — the culprit rests on a clean but modest single-sample step (reinforced by two good samples at 44687f2 and three elevated post-commit samples), and the commit is functionally a no-op on x64.

    System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)

    • Culprit: 44687f2d763b
    • Commit Message: Add ReadOnlySpan constructors to BitArray (Add ReadOnlySpan constructors to BitArray #131500)
    • Relevance: RELATED
    • Explanation: The commit modifies System.Private.CoreLib/src/System/Collections/BitArray.cs, the exact type the test exercises, and explicitly routes the existing array constructors (including the bool[] ctor measured by BitArrayBoolArrayCtor) through new ReadOnlySpan overloads. This indirection sits directly in the benchmark's hot path, providing a clear mechanism for the +21.7% slowdown.
    • ⚠️ Culprit not measured: no Helix build artifacts were available for the culprit commit, so its value could not be measured directly. The attribution is inferred from neighboring commits (bracketing) — confidence is capped accordingly.

    Helix jobs

    Job ID Submitted by bisection Last status
    93cc182e-172d-48fe-bf33-fc5ba0e4df4f ✅ submitted
    0569ef3f-c4d7-4b8c-8b75-c3a26aeac6d1 ✅ submitted
    9e677400-8023-4f59-afb9-f67d5b019706 ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    (baseline) baseline 16.730 ns
    ba8afceee835 baseline/good 17.160 ns
    (culprit) culprit 22.650 ns
    d5c739f99fa7 head/regressed 22.410 ns
    dc8d92dfeb8c good 17.280 ns
    fd405fc61ba6 good 17.250 ns
    Bisection details

    …[earlier output omitted]

    BISECTION RESULT:
- Culprit Commit: 44687f2d763be239ff562b137360ab572f47066e
- Commit Message: Add ReadOnlySpan constructors to BitArray (#131500)
- Commits Tested: 8
- Confidence: 4
- Commit Relevance: RELATED
- Relevance Explanation: The commit modifies System.Private.CoreLib/src/System/Collections/BitArray.cs, the exact type the test exercises, and explicitly routes the existing array constructors (including the bool[] ctor measured by BitArrayBoolArrayCtor) through new ReadOnlySpan<bool> overloads. This indirection sits directly in the benchmark's hot path, providing a clear mechanism for the +21.7% slowdown.
- Baseline Value: 16.73 ns
- Culprit Value: 22.65 ns
- Summary: Double-bisected the 15-commit range on Windows.11.Amd64.Viper.Perf; the sustained regression appears between fffed3110252 (16.73 ns, good) and 44687f2d763b (22.65 ns, regressed) and persists to head (22.41 ns). Both transition endpoints were re-sampled in a second job, confirming a ~6 ns (≈3σ) step well above the 1.97 ns noise floor. An earlier transient elevation at 896a3828 reverted by mid-range and was not the sustained regression.

HELIX JOBS:
- 93cc182e-172d-48fe-bf33-fc5ba0e4df4f
- 0569ef3f-c4d7-4b8c-8b75-c3a26aeac6d1
- 9e677400-8023-4f59-afb9-f67d5b019706

MEASUREMENTS:
- ba8afceee835 (baseline/good): 17.16 ns
- 896a3828ea34 (Q1, transient-high): 19.94, 19.12 ns
- 0e58de655dae (mid, good): 17.73, 18.03 ns
- fd405fc61ba6 (good): 17.25 ns
- dc8d92dfeb8c (good): 17.28 ns
- fffed3110252 (last-good before culprit): 16.64, 16.82 ns
- 44687f2d763b (culprit): 21.97, 23.32 ns
- d5c739f99fa7 (head/regressed): 22.41 ns

    The absolute culprit/head values run slightly above the reported compare (20.32), but the last-good measurement (16.73 ns) matches the reported baseline (16.70) closely and the reproduced step change unambiguously localizes the regression to the BitArray span-constructor refactor.

    🤖 Proposed Next Actions

    Rule: R5_look_again — Uncertain — flag for re-triage.

    Finding confidence 3/5 is below the required 4/5. Revisit once more historical data is available.

    # Action Status Details
    1 Add label ✅ Done Look Again
    2 Remove label ⏭️ Skipped (dry-run) untriaged

    🦆 Pre-bisection Re-analysis (advisory)

    ✅ Agrees with the pre-bisection plan — advisory only, not applied

    Rationale: Both findings are classified non-noise with high change confidence (4/5), relevant stacks, and point to the same likely commit, and both are queued for bisection with no bimodal history; the noise call and candidate set are consistent and well-supported.

    This critique is advisory in the current configuration and did not change which tests were bisected.

    🦆 Re-analysis (advisory)

    ✅ Agrees with the analysis — advisory only, not applied

    Flags: mixed-bisection-evidence

    Rationale: The packet culprit 44687f2 is well-supported by the BitArrayBoolArrayCtor(512) bisection (RELATED, converged, no alignment warning), and the modest 3/5 packet scores are appropriate. Note only that BitArraySetLengthGrow(4) is attributed to the same culprit despite its own bisection converging on a different, LIKELY_UNRELATED commit with an alignment warning, so that secondary finding's evidence is weaker than its 4/5 change confidence implies.

    This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

    Generated by PerfTriageAgent on 2026-08-04 11:41 UTC

  3. DrewScoggins
    transferred this issue fromdotnet/perf-autofiling-issueson Aug 4, 2026
  4. github-actions
    added
    area-System.Collections
    on Aug 4, 2026
  5. DrewScoggins
    assigned
    joshuajyue
    on Aug 4, 2026
  6. DrewScoggins
    added
    tenet-performancePerformance related issue
    tenet-performance-benchmarksIssue from performance benchmark
    on Aug 4, 2026

  7. DrewScoggins commented on Aug 4, 2026

    @DrewScoggins
    DrewScoggins
    on Aug 4, 2026
    Member

    Related to #131500.

    Related Improvements:

  8. dotnet-policy-service commented on Aug 4, 2026

    @dotnet-policy-service
    dotnet-policy-servicebot
    on Aug 4, 2026 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @dotnet/area-system-collections
    See info in area-owners.md if you want to be subscribed.

  9. LoopedBard3 commented on Aug 4, 2026

    @LoopedBard3
    LoopedBard3
    on Aug 4, 2026
    Member

    Related Regressions — issues attributed to the same commit:

    Augmented by PerfTriageAgent.

  10. LoopedBard3 commented on Aug 5, 2026

    @LoopedBard3
    LoopedBard3
    on Aug 5, 2026
    Member

    Related Improvements — issues attributed to the same commit:

    Augmented by PerfTriageAgent.

  11. jeffhandley commented on Aug 17, 2026

    @jeffhandley
    jeffhandley
    on Aug 17, 2026
    Member

    Per discussion on #132087 we decided not to take the fix and to accept the new baseline.

  12. jeffhandley
    closed this as completedon Aug 17, 2026
  13. dotnet-policy-service
    removed
    untriagedNew issue has not been triaged by the area owner
    on Aug 17, 2026
  14. github-actions
    locked and limited conversation to collaborators on Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-x64area-System.Collectionsos-windowsruntime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions