Repository navigation
[Perf] Linux/arm64: 6 Regressions on 8/25/2026 10:44:15 PM +00:00 #132840
Description
Activity
🔍 Automated Triage Analysis
Summary: Verified regression in FormatterUInt64(value: 18446744073709551615) — caused by commit cd25434. 2 tests bisected to related culprit commits. 1 other test(s) found unrelated commits (possible alignment artifacts).
Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)Likely Cause: cd254340926a9de135f596224d5d4daff12c2395
🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 3 dry-run (rule
R4_known_regression_new— see Full Analysis for the table).📊 Full Analysis (click to expand)
Summary: 6 System.* formatting microbenchmarks (integer/date
TryFormat+Utf8Formatter) stepped up +12% to +37% on Linux/arm64, all coinciding exactly with commit cd254340 — "Remove unsafe code from number parsing and formatting" (#131913), whose authored timestamp is identical to the regression date and which rewrites every formatting path under test. Finding confidence 4/5 (292–296 prior runs, clean step changes); regression confidence 5/5 (exact timing match, direct code path, highest test at z=5.08, and a co-occurring sibling issue #78533).Issue Overview
- Issue: #78542 - [Perf] Linux/arm64: 6 Regressions on 8/25/2026 10:44:15 PM +00:00
- Date: 2026-08-25 | Queue: CobaltAzureLinux (AzureLinux 3.0) | OS/Arch: linux/arm64
- Commits: 8baacb00 → 9f18a550 (11 commits)
- Labels: untriaged, perf-regression, os-linux, branch-refs/heads/main, arch-arm64, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
- Runtime stack: Desktop + CoreCLR (tiered, arm64). All regressing tests exercise shared BCL formatting code.
- Historical Data: Available (292–296 prior runs per test; 1,177 before-runs total, high statistical power)
Test Analysis
Test Name Baseline Compare Δ% μ σ n z Thresh% Noise? Assessment Utf8Formatter.FormatterDateTimeOffsetNow 23.14 31.71 +37.1% 22.61 1.79 292 5.08 15.87% N Step change, non-overlapping distributions Utf8Formatter.FormatterUInt32 11.05 12.94 +17.1% 10.85 0.95 294 2.20 17.45% N Step change (borderline: Δ 0.3pp under thresh) Utf8Formatter.FormatterUInt64 25.84 28.95 +12.0% 25.05 2.51 295 1.55 20.02% N Step change; thresh inflated by low outliers Perf_Version.TryFormatL 46.89 55.50 +18.4% 45.61 4.71 296 2.10 20.66% N Step change; thresh inflated by low outliers Perf_Int32.TryFormat(2147483647) 11.07 12.92 +16.7% 10.74 0.97 296 2.24 18.10% N Step change, after-cluster above before-max Perf_UInt32.TryFormat(4294967295) 10.51 12.53 +19.2% 10.21 1.03 296 2.25 20.21% N Step change (borderline: CV at boundary) Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold
Statistical classifier reported 2 firm Regressions (FormatterDateTimeOffsetNow, Perf_Int32.TryFormat) and 4 Borderline. The 4 Borderline cases each show clean, tightly-clustered step changes in their after-values that sit entirely above the recent before-cluster; their inflated adaptive thresholds are artifacts of sporadic low-outlier historical runs (~16 ns for UInt64, ~29 ns for TryFormatL), not genuine variance. Given all 6 move together on the same build and all route through the same rewritten formatting code, they are treated as a single real regression cluster.
Finding Confidence: 4/5 - Strong historical data (292–296 runs), clean step-change patterns, complete commit info, and an exact timestamp match to the culprit. Not raised to 5 because no independent rerun/bisection was performed and after-sample is only 6 runs per test.
Candidate Stack Relevance
Commit Touched Paths (summary) Relevance Notes cd254340 Number.Formatting*.cs,DateTimeFormat.cs,TimeSpanFormat.cs,Number.NumberBuffer.cs,IUtfChar.csShared (directly under test) BCL number/date formatting rewrite; the exact code path for every failing test. Authored 2026-08-25T22:44:15Z = regression date. 7c33e9b3 src/coreclr/vm/**(JIT static-base asm helpers, arm64)LessLikely CoreCLR VM helper deletion; unrelated to formatting; no plausible mechanism for these tests. 8c610774 System.Net.Security/**LessLikely SslStream fix; unrelated code path. 6bebb650 / 5ec6efc1 src/native/managed/cdac/**LessLikely Diagnostics cDAC; not on the measured path. Others (#132703, #131969, #132749, #132748, #132247, #132755) tests / eng / docs / illink LessLikely Test-only, pipeline, docs, or linker changes; cannot affect these benchmarks. Confirmed Regressions
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDateTimeOffsetNow (+37.1%)
- Delta: 23.14 ns → 31.71 ns (+37.1%)
- Historical Context: μ=22.61 (σ=1.79, n=292); z-score=5.08; adaptive threshold=15.87%
- Assessment: Step change; before/after distributions completely non-overlapping (highest-confidence signal in the set)
- Finding Confidence: 5/5 - 292 runs, tight history, unambiguous step
- Regression Confidence: 5/5 - Extreme outlier (z≥3), exact timing match, direct code path
- Likely Cause: cd254340 - "Remove unsafe code from number parsing and formatting (#131913)"
- Affected Areas: BCL number/date formatting (
Number.Formatting,DateTimeFormat,TimeSpanFormat) - Stack Relevance: Shared (BCL code directly exercised by the test)
- Rationale: The commit replaces pointer-based (
unsafe) formatting with span-based implementations acrossNumberBuffer, integer types,DateTime, andTimeSpan.FormatterDateTimeOffsetNowroutes throughDateTimeFormat.cs(186+/226- lines rewritten), the single largest behavioral change in the window. Its authored timestamp (2026-08-25T22:44:15Z) is identical to the regression date, and no other commit in the range touches formatting. The rewrite added span bounds handling that regresses on arm64 despite the author's x64-oriented harness showing neutral/faster results. - Candidates Considered:
- Affected Areas: BCL number/date formatting (
- Verification Status: Not verified (bisection/rerun deferred to downstream step per instructions). Attribution is single-commit by construction — only one formatting commit exists in the range.
System.Tests.Perf_Int32.TryFormat(value: 2147483647) (+16.7%)
- Delta: 11.07 ns → 12.92 ns (+16.7%)
- Historical Context: μ=10.74 (σ=0.97, n=296); z-score=2.24; adaptive threshold=18.10%
- Assessment: Step change; all 6 after-values (12.92–13.06) lie above the entire recent before-cluster
- Finding Confidence: 4/5 - 296 runs, clean step; small after-sample
- Regression Confidence: 5/5 - Direct integer
TryFormatcode path, corroborated by UInt32 variant - Likely Cause: cd254340 - "Remove unsafe code from number parsing and formatting (#131913)"
- Affected Areas: Integer decimal formatting (
Number.Formatting.cs/Number.Formatting.Common.cs) - Stack Relevance: Shared (BCL integer formatting directly under test)
- Rationale:
Int32.TryFormatcalls the rewritten integer-formatting span path. The simultaneous, near-identical regression of bothInt32.TryFormatandUInt32.TryFormatpoints to the shared rewritten code path rather than test noise. - Candidates Considered:
- cd254340 - Only formatting change in range
- Affected Areas: Integer decimal formatting (
- Verification Status: Not verified (deferred).
Corroborating step changes (same cause)
The remaining four move as one cluster with the two firm regressions and share the identical culprit (cd254340); classified Borderline only because sporadic historical low-outliers inflated their adaptive thresholds:
-
Utf8Formatter.FormatterUInt32 (+17.1%, z=2.20) — integer formatting path
-
Utf8Formatter.FormatterUInt64 (+12.0%, z=1.55) — integer formatting path
-
Perf_Version.TryFormatL (+18.4%, z=2.10) — integer formatting via
Version.TryFormat -
Perf_UInt32.TryFormat (+19.2%, z=2.25) — integer formatting path
-
Regression Confidence (cluster): 4/5 - Individually below adaptive thresholds but consistent step changes on the same build through the same rewritten code; sibling issue #78533 co-occurs, lowering the odds of pure noise.
Open Questions
- Historical Data: Available and strong (292–296 before-runs). Recurring low-outlier spikes in the before-series for FormatterUInt64, TryFormatL, and the UInt32 tests inflate σ and adaptive thresholds, pushing 4 tests to Borderline despite clean step patterns.
- Confidence Limitations: After-sample is only 6 runs per test. Four tests sit just under their (inflated) adaptive thresholds; z-scores confirm significance for three of them.
- Verification Needed: A confirmatory rerun of
cd254340vs its parent0ea1e369via RunWithBuildAtHash on the arm64 Cobalt queue would raise both confidences to 5/5, especially for the borderline UInt64/UInt32 cases. - Bisection Status: Not performed. Not required — only one formatting-relevant commit exists in the 11-commit range, so attribution is unambiguous.
Recommended Actions
- Verification - Rerun the 6 tests with RunWithBuildAtHash on
cd254340926avs parent0ea1e3690e47(arm64, AzureLinux Cobalt) to confirm the step and quantify per-test deltas. - Source review - Manual review of cd254340 focusing on the arm64 codegen of the new span-based integer/date formatting paths in
Number.Formatting.cs,Number.Formatting.Common.cs, andDateTimeFormat.cs(the pre-existing pointer paths may have avoided bounds checks that the span rewrite reintroduced on arm64). - Follow-up - Cross-reference sibling issue #78533 and any x64/other-arch autofiles for the same commit to determine whether the regression is arm64-specific.
Bisection Results
Test Status Culprit Commit Relevance Reported Diff (issue) Bisected Diff (at culprit) FormatterDateTimeOffsetNow(value: 12/... ❌ Not measured - - 23.1→31.7 ns (+37%) — (not measured) FormatterUInt32(value: 4294967295) ✅ Culprit found cd254340926a UNCERTAIN 11.0→12.9 ns (+17%) 11.0→12.9 ns (+17%) FormatterUInt64(value: 18446744073709... ✅ Culprit found cd254340926a ✅ RELATED 25.8→28.9 ns (+12%) 25.8→28.9 ns (+12%) TryFormatL ✅ Culprit found 9f18a550aaa9 UNRELATED 46.9→55.5 ns (+18%) 34.3→55.6 ns (+62%) TryFormat(value: 2147483647) 🔁 Did not reproduce - - 11.1→12.9 ns (+17%) ≈ (baseline ≈ head) TryFormat(value: 4294967295) ✅ Culprit found cd254340926a ✅ RELATED 10.5→12.5 ns (+19%) 10.5→12.4 ns (+18%) Total bisection time: 120m 51s
Tests measured: 5/6
Culprit found: 4/6⚠️ Alignment WarningBisection found commits that are unrelated to the regressed tests. This strongly suggests the performance changes are caused by code alignment, binary layout, or other non-functional factors rather than actual regressions. Consider marking as noise.
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDateTimeOffsetNow(value: 12/30/2017 3:45:22 AM -08:00)
- Status: failed
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)
- Culprit: cd254340926a
- Relevance: UNCERTAIN
- Explanation: The good→bad transition isolates to cd25434 (0ea1e369 good at 11.04 ns, cd25434 bad at 12.89 ns). Commit contents were not inspected here, so functional relevance to UInt32 UTF8 formatting cannot be confirmed.
Bisection details
- Culprit:
cd254340926a - Relevance: UNCERTAIN
- Baseline: 11.048 ns
- Culprit value: 12.889 ns
Per-commit measurements (oldest → newest) (ns)
# Commit Result Measurement 0 8baacb006f56baseline/good 11.038 ns 4 7c33e9b3fdb2good 11.042 ns 7 bebb6503c200good 11.040 ns 8 5ec6efc171b1good 11.047 ns 9 0ea1e3690e47good 11.042 ns 10 cd254340926aculprit/bad 12.889 ns 11 9f18a550aaa9head/bad 13.179 ns Helix jobs
Job ID 135b1f4f-3b5d-4421-b94f-413867cab4a6868d996d-4302-4f81-8fa7-cc23ebf33ef098917c27-a4c8-4a70-91cb-e3995b657df7System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 18446744073709551615)
- Culprit: cd254340926a
- Relevance: RELATED
- Explanation: Commit cd25434 (#131913) rewrites integer/number formatting paths in Number.Formatting.cs and Number.Formatting.Common.cs, which are exactly what FormatterUInt64 exercises. The regression from 25.84 ns to 28.93 ns aligns with 0ea1e369 (good) -> cd25434 (bad).
Bisection details
- Culprit:
cd254340926a - Relevance: RELATED
- Baseline: 25.839 ns
- Culprit value: 28.933 ns
Per-commit measurements (oldest → newest) (ns)
# Commit Result Measurement 0 8baacb006f56baseline/good 25.847 ns 4 7c33e9b3fdb2good 25.846 ns 7 bebb6503c200good 25.850 ns 8 5ec6efc171b1good 25.846 ns 9 0ea1e3690e47good 25.841 ns 10 cd254340926aculprit/bad 28.933 ns 11 9f18a550aaa9head/bad 28.951 ns Helix jobs
Job ID 213c6025-ab59-4103-a873-fb7ca0559cefc04296be-d653-4e91-a14a-1a92abb4ad028bdb1e2f-cb45-42ef-a231-e45ca98d00d3System.Tests.Perf_Version.TryFormatL
- Culprit: 9f18a550aaa9
- Relevance: UNRELATED
- Explanation: Commit 9f18a55 only modifies ILLink test infrastructure files (Mono.Linker.Tests TestDatabase.cs and TestSuites.cs) to improve test parallelism. It touches no runtime/library code related to System.Version or its TryFormat implementation, so it cannot plausibly cause the Perf_Version.TryFormatL regression. The measured good→bad transition is likely benchmarking noise/measurement environment rather than a true code-caused regression.
Bisection details
- Culprit:
9f18a550aaa9 - Relevance: UNRELATED
- Baseline: 46.887 ns
- Culprit value: 55.589 ns
Per-commit measurements (oldest → newest) (ns)
# Commit Result Measurement 0 8baacb006f56baseline/good 46.867 ns 4 7c33e9b3fdb2good 46.967 ns 7 bebb6503c200good 46.871 ns 8 5ec6efc171b1good 46.860 ns 10 cd254340926agood 34.306 ns 11 9f18a550aaa9head/culprit/bad 55.589 ns Helix jobs
Job ID f1dfe87b-0f1c-4c52-a4c8-ed934f0472e739efa690-2409-4ea1-98bd-3ce3d0afea39System.Tests.Perf_Int32.TryFormat(value: 2147483647)
- Status: inconclusive
Bisection details
- Baseline: 11.069 ns
Per-commit measurements (oldest → newest) (ns)
# Commit Result Measurement 0 8baacb006f56baseline 11.006 ns 4 7c33e9b3fdb2- 11.004 ns 7 bebb6503c200- 9.122 ns 11 9f18a550aaa9head 9.390 ns Helix jobs
Job ID 5da731ed-5f17-4724-88ce-688e41bff2d5System.Tests.Perf_UInt32.TryFormat(value: 4294967295)
- Culprit: cd254340926a
- Relevance: RELATED
- Explanation: Commit cd25434 (#131913) rewrites integer/number parsing and formatting in Number.Formatting.cs and related files, directly touching the UInt32.TryFormat code path exercised by this benchmark. The window is ordered 5ec6efc(good,10.51)->0ea1e369(good,10.52)->cd254340(bad,12.37), so the good->bad transition is this single adjacent commit.
Bisection details
- Culprit:
cd254340926a - Relevance: RELATED
- Baseline: 10.510 ns
- Culprit value: 12.375 ns
Per-commit measurements (oldest → newest) (ns)
# Commit Result Measurement 0 8baacb006f56baseline/good 10.510 ns 4 7c33e9b3fdb2good 10.512 ns 7 bebb6503c200good 10.513 ns 8 5ec6efc171b1good 10.512 ns 9 0ea1e3690e47good 10.520 ns 10 cd254340926aculprit/bad 12.375 ns 11 9f18a550aaa9head/bad 12.550 ns Helix jobs
Job ID ea5282f0-bf1d-4f48-9ee8-938a388581d7387d5060-7a03-426b-967e-63a346d12eb0741c2e3f-4c29-45f0-b6b6-4774ed7deffc🤖 Proposed Next Actions
Rule:
R4_known_regression_new— Known-cause regression — transfer & notify.Verified regression attributed to commit cd25434 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.
# Action Status Details 1 Back-reference cause PR ⏭️ Skipped (dry-run) on culprit PR for cd254340926a2 Transfer issue ⏭️ Skipped (dry-run) → dotnet/runtime 3 Ping owners ⏭️ Skipped (dry-run) ping tannergooding 🦆 Pre-bisection Re-analysis (advisory)
✅ Agrees with the pre-bisection plan — advisory only, not applied
Rationale: All six findings are non-noise with strong change confidence (4-5/5) and converge on the same likely commit, so the actionable classification and the candidate set are consistent and well-supported.
This critique is advisory in the current configuration and did not change which tests were bisected.
🦆 Re-analysis (advisory)
⚠️ Disputes the analysis — advisory only, not appliedFlags:
overconfident-change,weak-bisection-evidenceRationale: The aggregate culprit cd25434 is fairly supported since three successful bisections (FormatterUInt32, FormatterUInt64, Perf_UInt32.TryFormat, two marked RELATED) converge on it, so I do not dispute it, but a top-level change confidence of 5/5 is too high because two of the findings carrying 5/5 change confidence (FormatterDateTimeOffsetNow and Perf_Int32.TryFormat) had bisections that failed or were inconclusive and no z-score/variance evidence is provided.
Suggested confidence: Change Confidence → 4/5 (advisory — not applied)
This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.
Generated by PerfTriageAgent on 2026-08-27 11:17 UTC
Bisected to #131913
Related regressions:
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.Most of these, except
TryFormatLforSystem.Versionare within the general noise and should likely rather be look again instead (and same with the "related regressions")I'm going to take a look at the Version one locally, but closing this as everything else (even if it is noise) is within 1ns and unlikely to be a hot path bottleneck. It's an acceptable tradeoff for making the code safer and more robust.
Related Improvements — issues attributed to the same commit:
- [Perf] Linux/x64: 15 Improvements on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78714 (bot added)
- [Perf] Linux/x64: 8 Improvements on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78758 (bot added)
- [Perf] Windows/x64: 27 Improvements on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78701 (DrewScoggins added)
- [Perf] Windows/x64: 27 Improvements on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78701 (bot added)
- [Perf] Linux/arm64: 7 Improvements on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78871 (bot added)
- [Perf] Linux/x64: 11 Improvements on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78899 (bot added)
Augmented by PerfTriageAgent.
Related Regressions — issues attributed to the same commit:
- [Perf] Linux/x64: 5 Regressions on 8/26/2026 4:02:35 AM +00:00 perf-autofiling-issues#78709 (bot added)
- [Perf] Linux/arm64: 13 Regressions on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78878 (bot added)
- [Perf] Linux/arm64: 21 Regressions on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78867 (bot added)
- [Perf] Linux/x64: 2 Regressions on 8/25/2026 5:23:38 PM +00:00 perf-autofiling-issues#78895 (bot added)
- [Perf] Linux/x64: 24 Regressions on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78896 (bot added)
- [Perf] Linux/x64: 112 Regressions on 8/25/2026 10:44:15 PM +00:00 perf-autofiling-issues#78905 (bot added)
Augmented by PerfTriageAgent.
- locked and limited conversation to collaborators
on Oct 4, 2026
Run Information
Regressions in System.Buffers.Text.Tests.Utf8FormatterTests
Test Report
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Details
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDateTimeOffsetNow(value: 12/30/2017 3:45:22 AM -08:00)
ETL Files
Histogram
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)
ETL Files
Histogram
JIT Disasms
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 18446744073709551615)
ETL Files
Histogram
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Regressions in System.Tests.Perf_Version
Test Report
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Details
System.Tests.Perf_Version.TryFormatL
ETL Files
Histogram
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Regressions in System.Tests.Perf_Int32
Test Report
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Details
System.Tests.Perf_Int32.TryFormat(value: 2147483647)
ETL Files
Histogram
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Regressions in System.Tests.Perf_UInt32
Test Report
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Details
System.Tests.Perf_UInt32.TryFormat(value: 4294967295)
ETL Files
Histogram
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository