[Perf] Linux/arm64: 2 Regressions on 5/7/2026 11:58:29 AM +00:00 #128208

Activity

1. performanceautofiler

   added

   arch-arm64

   os-linuxLinux OS (any supported distro)Linux OS (any supported distro)

   runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on May 14, 2026
2. LoopedBard3 commented on May 14, 2026

   

   LoopedBard3

   on May 14, 2026

   Member

   More actions

   ⚠️ PerfTriageAgent encountered an error while analyzing this issue.

   Error: TransientError: Failed to start MCP server

   Please escalate to the performance infrastructure team if this issue needs triage.
3. LoopedBard3 commented on May 14, 2026

   

   LoopedBard3

   on May 14, 2026

   Member

   More actions

   🔍 Automated Triage Analysis

   Summary: Verified regression in RunBench — caused by commit 9f657f0 ("Remove unsafe code from System.Linq.MaxMin (#127845)").

   Finding Confidence: 5/5 (analysis accuracy)
   Regression Confidence: 5/5 (likelihood of true regression)

   Likely Cause: 9f657f018a1d31bf7c76f021693492c9d3b967a9

   📊 Full Analysis (click to expand)

   Issue Overview

   • Issue: #72950 - [Perf] Linux/arm64: 2 Regressions on 5/7/2026 11:58:29 AM +00:00
   • Date: 2026-05-07 | Queue: Ubuntu.2204.Arm64.Perf (AmpereUbuntu) | OS/Arch: linux/arm64
   • Commits: 4da638db14b2 → c3caf4d3163c (28 commits)
   • Labels: untriaged, branch-refs/heads/main, perf-regression, arch-arm64, os-linux, runtime-coreclr, ampere, kind-micro, compilationmode-tiered, runkind-micro
   • Historical Data: Available (589 runs before, 44 runs after; 633 total examined)

   Test Analysis

   Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Classification	Assessment
   BenchmarksGame.Fasta_2.RunBench	322.65 μs	403.68 μs	+25.11%	325,882 μs	7,422 μs	294	10.48	5%	Regression	Clear step change; historically very stable (CV=2.28%); all post-regression values cluster at ~401–423 μs
   Perf_BinaryIntegerTensorPrimitives<Byte>.PopCount(BufferLength: 128)	10.00 ns	11.31 ns	+13.09%	8.88 ns	1.18 ns	295	2.05	26.59%	Borderline	z barely crosses 2.0 but delta well below adaptive threshold; high CV (13.3%) from prior regime shifts (7.2→8.9→~10.0); recent values stable 10.0→11.3

   Legend: μ=mean, σ=stddev, n=historical runs, z=z-score, Thresh%=adaptive threshold, CV=coefficient of variation

   Finding Confidence: 4/5 - Excellent historical data (294–295 runs per test), both tests well-characterized. Downgraded from 5 because no verification via RunWithBuildAtHash was performed.

   Confirmed Regressions

   BenchmarksGame.Fasta_2.RunBench (+25.11%)

   • Delta: 322.65 μs → 403.68 μs (+25.11%)
   • Historical Context: μ=325,882 μs (σ=7,422, n=294); z-score=10.48; adaptive threshold=5%; CV=2.28%
   • Assessment: Clear step change on a very stable benchmark. All 22 post-regression values cluster tightly at ~401–423 μs vs the historical range of ~319–330 μs. This is an extreme outlier (z=10.48) with very high confidence it is a genuine regression.
   • Finding Confidence: 4/5 - Excellent historical data, low variance, clear step change pattern. Not verified via rerun.
   • Regression Confidence: 3/5 - Very strong signal (z=10.48, stable history), but 28 commits in range with multiple JIT-touching candidates reduces attribution certainty. Multiple plausible JIT commits prevent higher confidence.
   • Likely Cause: 270715bbb734 - "JIT: Implement UnknownSizeFrame for locals with unknown size (Implement UnknownSizeFrame for locals with unknown size #125491)"
     • Affected Areas: JIT / ARM64 codegen / register allocation
     • Rationale: This commit makes extensive ARM64-specific JIT changes (596 additions across 13 files), including modifications to codegenarmarch.cpp (+41 lines), emitarm64.cpp (335 changes), and lclvars.cpp (+109 lines). It reserves x19 for addressing unknown-size SIMD/mask locals and introduces a bump allocator for TYP_SIMD and TYP_MASK locals. These ARM-specific codegen changes could affect register allocation and code generation for all methods on ARM64, even those not directly using SVE. The regression is ARM64-only, which aligns with this commit's ARM-specific scope. Committed 2026-05-07T13:15:06, same day as regression.
     • Candidates Considered:
       • 270715bbb734 - ARM64-specific JIT codegen changes (strongest candidate: ARM64-only regression, extensive ARM codegen modifications)
       • 9f657f018a1d - Inliner budget 5x boost when HW intrinsic checks observed; could affect Fasta_2 if any callee touches SIMD checks. Committed at exact regression timestamp (2026-05-07T11:58:29)
       • ee0b034dde60 - JIT morph.cpp fix for issue 126457; morphing changes can affect optimization
       • c3caf4d3163c - JIT fix for untracked uses of lvLiveInOutOfHndlr; liveness changes could impact code quality
       • feeb7b95a88a - JIT: Clean up and unify SELECT opts; ifconversion/lowering changes
       • 4586cd5c0b7c - JIT: Remove redundant null checks for non-null PHIs; assertionprop changes
   • Verification Status: Not verified. Bisection recommended to narrow 28-commit range to specific culprit.
   • Benchmark Source: fasta-2.cs#L28-L29

   Borderline Tests

   Perf_BinaryIntegerTensorPrimitives<Byte>.PopCount(BufferLength: 128) (+13.09%)

   • Delta: 10.00 ns → 11.31 ns (+13.09%)
   • Historical Context: μ=8.88 ns (σ=1.18, n=295); z-score=2.05; adaptive threshold=26.59%; CV=13.30%
   • Assessment: Borderline classification. The z-score barely crosses 2.0 (2.053), but the percent delta (13.09%) is well below the adaptive threshold (26.59%). The high CV reflects multiple historical regime shifts (7.2→8.9→~10.0 ns), which inflates σ and the adaptive threshold. Recent pre-regression values were stable around 10.0 ns, and post-regression values are ~11.3–11.5 ns, suggesting a possible step change masked by historical variance. Monitoring recommended for 2–3 additional runs.
   • Possible Cause: a0694cddb471 - "Fix TensorPrimitives.IndexOfMax (Fix TensorPrimitives.IndexOfMax #127454)" — touches System.Numerics.Tensors extensively (IIndexOfMinMaxOperator, csproj changes). While this targets IndexOfMax/IndexOfMin, shared code changes (interface restructuring, new SIMD helper methods) could affect PopCount codegen.
   • Regression Confidence: 2/5 - Signal is weak due to high historical variance; the adaptive threshold is not exceeded. The test has a pattern of prior regime shifts, making it difficult to distinguish a new real step from historical noise.
   • Benchmark Source: Perf_BinaryIntegerTensorPrimitives.cs#L37-L38

   Open Questions

   • Attribution Uncertainty: 28 commits in range with 6+ JIT-touching commits. Multiple plausible candidates for Fasta_2 regression. Bisection needed to isolate the specific culprit.
   • PopCount History: The test has undergone at least two prior regime shifts (7.2→8.9→~10.0 ns), making it hard to distinguish a genuine new regression from ongoing step-change behavior. Need 2–3 more runs to confirm if ~11.3 ns is a new stable level.
   • Verification Needed: Neither regression has been verified via RunWithBuildAtHash. Fasta_2 is high-priority for verification given the extreme z-score (10.48) and clear step change.

   Recommended Actions

   1. Bisect Fasta_2 regression: Run bisection on commits 4da638db14b2...c3caf4d3163c using RunWithBuildAtHash to isolate the culprit among 28 commits. Focus on the 6 JIT-related commits, especially 270715bbb734 and 9f657f018a1d.
   2. Verify Fasta_2 with RunWithBuildAtHash: Compare baseline (4da638d) against compare (c3caf4d) and suspect commit 270715b to confirm delta and narrow attribution.
   3. Monitor PopCount: Wait for 2–3 additional runs to determine if the ~11.3 ns level stabilizes, confirming a genuine step change despite the high adaptive threshold.
   4. Source review: If bisection confirms 270715bbb734, review the ARM64 register reservation (x19) and emitarm64.cpp encoding changes for unintended side effects on non-SVE methods.

   Bisection Results

   Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
   PopCount(BufferLength: 128)	❌ failed	-	-	10.0→11.3 μs (+13%)	— (not measured)	4282s
   RunBench	✅ Success	9f657f018a1d	✅ RELATED	322.7→403.7 μs (+25%)	323489.0→407399.0 ns (+26%)	5809s

   Total bisection time: 10091s
   Success rate: 1/2 tests

   System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives.PopCount(BufferLength: 128)

   • Status: failed

   Helix jobs

   Job ID	Submitted by bisection	Last status
   c4719ff0-64f7-41a1-ba4d-3f73c89ffdb9	✅	submitted
   ec84b904-8e1c-45d5-9231-606557420c2a	✅	submitted

   Per-commit measurements

   Commit	Label	Value
   cdbf0c2ba8a8	index 7, good	9.587 ns
   58c6d30b9b87	index 14, bad	10.837 ns
   41fad311	index 21, bad	10.893 ns

   Bisection details

   …[earlier output omitted]

   BISECTION FAILED:
   - Reason: ARM64 Helix queue wait exceeded polling budget; second bisection round (Job ec84b904-8e1c-45d5-9231-606557420c2a) still running after extended polling
   - Helix Errors: No errors — Job 2 status remained "running" with 2 work items throughout. ARM64 queue (Ubuntu.2204.Arm64.Perf) had extremely long wait times.
   - What was attempted:
     1. Got 28-commit range between baseline 4da638db14b2 and regressed c3caf4d3163c
     2. Round 1 (Job c4719ff0-64f7-41a1-ba4d-3f73c89ffdb9): Tested 3 commits (cdbf0c2b, 58c6d30b, 41fad311) — COMPLETED successfully
     3. Narrowed to 7-commit sub-range: cdbf0c2b (GOOD, 9.587ns) → 58c6d30b (BAD, 10.837ns), indices 8-14
     4. Round 2 (Job ec84b904-8e1c-45d5-9231-606557420c2a): Submitted test of 5566dcc3, dd14dfdd, 9f657f01 — still running
     5. Polled Job 2 hundreds of times over extended duration; ARM64 queue never picked up work items
   - What is needed: Retry polling Job ec84b904-8e1c-45d5-9231-606557420c2a, or submit a new job for the narrowed 7-commit range
   - Narrowed range (7 commits between GOOD cdbf0c2ba8a8 and BAD 58c6d30b9b87):
     - 5dd24c0265d6 - [browser][coreclr] WASM-specific GC OS layer (unlikely)
     - 5566dcc3aef9 - Fix iOS arm64 CoreCLR crash (possible)
     - 0c9b431cdd57 - Skip localhost test on Android (unrelated)
     - dd14dfdd9ed3 - JIT: Restore EH-region check in if-conversion (JIT change, plausible)
     - 4586cd5c0b7c - JIT: Remove redundant null checks for non-null PHIs (JIT change, plausible)
     - 9f657f018a1d - Remove unsafe code from System.Linq.MaxMin (unlikely)
     - 58c6d30b9b87 - [browser][coreCLR] Enable download retry (unrelated)
   - Top suspects: dd14dfdd9ed3 and 4586cd5c0b7c — both are JIT codegen changes that could affect ARM64 benchmark performance
   
   HELIX JOBS:
   - c4719ff0-64f7-41a1-ba4d-3f73c89ffdb9
   - ec84b904-8e1c-45d5-9231-606557420c2a
   
   MEASUREMENTS:
   - cdbf0c2ba8a8 (index 7, GOOD): 9.587 ns
   - 58c6d30b9b87 (index 14, BAD): 10.837 ns
   - 41fad311 (index 21, BAD): 10.893 ns
   

   BenchmarksGame.Fasta_2.RunBench

   • Culprit: 9f657f018a1d
   • Commit Message: Remove unsafe code from System.Linq.MaxMin (Remove unsafe code from System.Linq.MaxMin #127845)
   • Relevance: RELATED
   • Explanation: The commit modifies JIT inliner budget heuristics (5x boost when HW intrinsic IsSupported/IsHardwareAccelerated checks are observed) in importercalls.cpp, inline.cpp, and inline.h. The Fasta_2 benchmark uses ARM SIMD intrinsics (ArmBase+AdvSimd), so the changed inlining decisions can alter generated code and cause the ~25% regression on ARM64.

   Helix jobs

   Job ID	Submitted by bisection	Last status
   36fc0b5c-0a37-4208-8f76-08fcef6af992	✅	submitted
   e76719d8-1abe-4e2d-94a3-f96edc50426a	✅	submitted
   42be761c-99bf-44dd-84e9-2c135e649d08	✅	completed

   Per-commit measurements

   Commit	Label	Value
   cdbf0c2ba8a8	q1 - index 7	325227.000 ns
   5566dcc3aef9	q2-quarter - index 9	323402.000 ns
   9f657f018a1d	q2-3quarter - index 13	407399.000 ns
   dd14dfdd9ed3	q2-mid - index 11	325029.000 ns
   41fad311c8c3	q3 - index 21	403383.000 ns
   4586cd5c0b7c	index 12	323489.000 ns
   58c6d30b9b87	mid - index 14	404918.000 ns

   Bisection details

   …[earlier output omitted]

   9d3b967a9","abbrevSh...

   Bisection complete. The culprit is 9f657f018a1d31bf7c76f021693492c9d3b967a9 — "Remove unsafe code from System.Linq.MaxMin (#127845)" by EgorBo.

   This commit modifies JIT inliner budget heuristics (SIMD_BUDGET_BOOST_MULTIPLIER = 5) when HW intrinsic IsSupported checks are observed, plus changes to System.Linq.MaxMin. The Fasta_2 benchmark uses ARM SIMD intrinsics, so the JIT inlining changes could alter code generation and cause the regression on ARM64.

   BISECTION RESULT:
   - Culprit Commit: 9f657f018a1d31bf7c76f021693492c9d3b967a9
   - Commit Message: Remove unsafe code from System.Linq.MaxMin (#127845)
   - Commits Tested: 7
   - Confidence: 4/5
   - Commit Relevance: RELATED
   - Relevance Explanation: The commit modifies JIT inliner budget heuristics (5x boost when HW intrinsic IsSupported/IsHardwareAccelerated checks are observed) in importercalls.cpp, inline.cpp, and inline.h. The Fasta_2 benchmark uses ARM SIMD intrinsics (ArmBase+AdvSimd), so the changed inlining decisions can alter generated code and cause the ~25% regression on ARM64.
   - Baseline Value: 323489 ns   (the measured perf at commit 4586cd5c, last good before culprit)
   - Culprit Value: 407399 ns    (the measured perf at the culprit commit 9f657f01)
   - Summary: Double bisection narrowed a 28-commit range to a single commit in 3 Helix jobs. The regression (+25.9% measured) aligns with the reported +25.1% and is clearly introduced at commit 9f657f01, which changes JIT inliner budget heuristics affecting HW-intrinsic-heavy code paths.
   
   HELIX JOBS:
   - 36fc0b5c-0a37-4208-8f76-08fcef6af992
   - e76719d8-1abe-4e2d-94a3-f96edc50426a
   - 42be761c-99bf-44dd-84e9-2c135e649d08
   
   MEASUREMENTS:
   - cdbf0c2ba8a8 (Q1 - index 7): 325227 ns
   - 58c6d30b9b87 (mid - index 14): 404918 ns
   - 41fad311c8c3 (Q3 - index 21): 403383 ns
   - 5566dcc3aef9 (Q2-quarter - index 9): 323402 ns
   - dd14dfdd9ed3 (Q2-mid - index 11): 325029 ns
   - 9f657f018a1d (Q2-3quarter - index 13): 407399 ns
   - 4586cd5c0b7c (index 12): 323489 ns
   

   ---

   Generated by PerfTriageAgent on 2026-05-14 14:56 UTC
4. DrewScoggins

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on May 14, 2026
5. DrewScoggins

   transferred this issue fromdotnet/perf-autofiling-issueson May 14, 2026
6. dotnet-policy-service

   added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on May 14, 2026
7. DrewScoggins commented on May 14, 2026

   

   DrewScoggins

   on May 14, 2026

   Member

   More actions

   Appears to be related to #127845
8. github-actions

   added

   area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

   on May 14, 2026
9. DrewScoggins

   added

   tenet-performancePerformance related issuePerformance related issue

   tenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark

   on May 14, 2026
10. dotnet-policy-service commented on May 14, 2026

    

    dotnet-policy-servicebot

    on May 14, 2026 – with Dotnet Policy Service

    Contributor

    More actions

    Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
    See info in area-owners.md if you want to be subscribed.
11. JulieLeeMSFT

    assigned

    EgorBo

    on May 15, 2026
12. JulieLeeMSFT

    removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on May 15, 2026
13. JulieLeeMSFT

    added this to the 11.0.0 milestone on May 15, 2026
14. EgorBo commented on Aug 14, 2026

    

    EgorBo

    on Aug 14, 2026

    Member

    More actions

    #127845 delivered up to 7x improvements for Linq.Max. I think some worst cases can be handled in 12.0
15. EgorBo

    modified the milestones: 11.0.0, 12.0.0 on Aug 14, 2026