Repository navigation
[Perf] Linux/x64: 2 Regressions on 9/25/2026 10:50:38 PM +00:00 #134867
Description
Activity
🔍 Automated Triage Analysis
Summary:
⚠️ Likely NOT a functional regression. Bisection of ParseSpan(value: "4294967295") found commit 7bdcc5f which is unrelated to the test area. This strongly suggests a code alignment/layout artifact. Consider marking as noise.Finding Confidence: 2/5 (analysis accuracy)
Regression Confidence: 2/5 (likelihood of true regression)⚠️ Note: Low regression confidence - attribution uncertain. See rationale in full analysis.🤖 Proposed Next Actions: Uncertain — flag for re-triage — 1 done, 1 dry-run (rule
R5_look_again— see Full Analysis for the table).📊 Full Analysis (click to expand)
Summary: Two confirmed UInt32 parsing regressions slowed by 11.2-11.9%, with finding confidence 4/5 and regression confidence 5/5. The likely cause is 7bdcc5f5809e (integer parsing), whose focused fast-path change directly affects both benchmarks; an isolated baseline/candidate rerun is still needed to verify the attribution.
Issue Overview
- Issue: #80016 - [Perf] Linux/x64: 2 Regressions on 9/25/2026 10:50:38 PM +00:00
- Date: 2026-09-25 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: Ubuntu 22.04/x64
- Commits: 513b9cd40dca -> 8633ad0ebb95 (21 commits)
- Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
- Runtime stack: Desktop + CoreCLR
- Configuration: CompilationMode=tiered, RunKind=micro
- Historical Data: Available (582 pre-change measurements across the two tests; 600 total measurements examined)
Test Analysis
Test Name Baseline Compare Delta mu sigma n z Thresh% Noise? Assessment System.Tests.Perf_UInt32.ParseSpan(value: "4294967295")11.413 ns 12.767 ns +11.86% 11.656 ns 0.126 ns 292 8.815 5% No Persistent step change System.Tests.Perf_UInt32.TryParse(value: "4294967295")11.731 ns 13.045 ns +11.21% 11.732 ns 0.221 ns 290 5.941 5% No Persistent step change Legend: mu=historical mean, sigma=historical standard deviation, n=historical runs, z=z-score, Thresh%=adaptive threshold
Finding Confidence: 4/5 - Both tests have more than 290 historical runs, low variation, and nine tightly clustered post-change measurements. The commit range and benchmark source are complete, but the suspected commit has not been verified by an isolated rerun.
Candidate Stack Relevance
The issue exercises Desktop CoreCLR. Shared library changes can directly affect it, while Wasm-specific changes are not exercised by this configuration.
Commit Touched Paths (summary) Relevance Notes 7bdcc5f5809e src/libraries/Common/src/System/Number.Parsing.Common.cs,src/libraries/System.Private.CoreLib/src/System/Number.Parsing.csShared Directly changes the binary-integer parsing fast path used by both affected UInt32 benchmarks. fd53e564dc67 src/coreclr/jit/gentree.cppRelevant Desktop JIT change, but limited to SIMD operand ordering and unrelated to scalar UInt32 parsing. 0b87c1cfc454 src/coreclr/jit/gentree.cpp,src/coreclr/jit/hwintrinsic.cpp, numeric BCL filesShared Numeric and JIT changes are limited to floating-point integer classification and vector APIs, not integer text parsing. d7a058ac0984 src/coreclr/jit/compiler.cpp,liveness.cpp,lower.cppLessLikely CoreCLR files are touched, but the change is specifically for Wasm PEP calls and is not exercised by the desktop benchmark. Confirmed Regressions
System.Tests.Perf_UInt32.ParseSpan(value: "4294967295") (+11.86%)
- Benchmark Source:
Perf.UInt32.cslines 38-40 - Delta: 11.413 ns -> 12.767 ns (+11.86%)
- Historical Context: mu=11.656 ns (sigma=0.126 ns, n=292); z-score=8.815; adaptive threshold=5%
- Assessment: Step change. The delta is more than twice the adaptive threshold and the compare value is an extreme historical outlier. Nine post-change measurements remain tightly clustered at the elevated level.
- Finding Confidence: 4/5 - Comprehensive, low-variance history and a persistent post-change level make the classification reliable; commit-level verification has not been run.
- Regression Confidence: 5/5 - The signal is extreme (z=8.815), history is stable (CV=1.081%), and a focused change directly modifies the exercised parsing path.
- Likely Cause: 7bdcc5f5809e - "Allow whitespace after leading numeric signs and parentheses (#134672)"
- Affected Areas: CoreLib integer parsing
- Stack Relevance: Shared
- Rationale: The commit added a whitespace check and fallback call to
TryParseBinaryIntegerStyle, the fast path used byuint.Parse(ReadOnlySpan<char>). It landed at 2026-09-25 22:41 UTC, immediately before the 22:50 UTC regression timestamp, and the author noted code-layout-sensitive common-case timing during validation. The source and timing align directly with both affected benchmarks. - Candidates Considered:
- fd53e564dc67 - Desktop JIT change, but only for SIMD operand ordering.
- 0b87c1cfc454 - Numeric code changes concern floating-point classification rather than text parsing.
- Verification Status: Not verified; attribution is based on direct code-path, timing, and two-test corroboration.
System.Tests.Perf_UInt32.TryParse(value: "4294967295") (+11.21%)
- Benchmark Source:
Perf.UInt32.cslines 47-50 - Delta: 11.731 ns -> 13.045 ns (+11.21%)
- Historical Context: mu=11.732 ns (sigma=0.221 ns, n=290); z-score=5.941; adaptive threshold=5%
- Assessment: Step change. The delta exceeds the adaptive threshold, is nearly six standard deviations above history, and remains consistently elevated after the change.
- Finding Confidence: 4/5 - Comprehensive history and consistent post-change measurements support the classification; commit-level verification has not been run.
- Regression Confidence: 5/5 - The strong signal (z=5.941), stable history (CV=1.884%), direct code hit, and matching ParseSpan regression make a genuine shared-path regression highly likely.
- Likely Cause: 7bdcc5f5809e - "Allow whitespace after leading numeric signs and parentheses (#134672)"
- Affected Areas: CoreLib integer parsing
- Stack Relevance: Shared
- Rationale:
uint.TryParse(string, out uint)reaches the same binary-integer parsing implementation changed by this commit. The new common-case branch is executed even when the input contains no sign or whitespace, providing a concrete mechanism for a code-layout or generated-code regression. The simultaneous ParseSpan slowdown corroborates the shared parser as the source. - Candidates Considered:
- fd53e564dc67 - Relevant runtime tree but no scalar parsing connection.
- 0b87c1cfc454 - Numeric changes do not affect integer string parsing.
- Verification Status: Not verified; attribution is based on direct code-path, timing, and two-test corroboration.
Open Questions
- Historical Data: Sufficient. Both tests have more than 290 historical runs, low variance, and nine consistent post-change measurements.
- Confidence Limitations: The regression itself is clear, but the likely-cause attribution has not been confirmed with before/after builds. The 21-commit range would ordinarily lower attribution confidence, although only one commit directly changes the affected code path.
- Verification Needed: Run both benchmarks at the baseline and suspected commit to distinguish the direct parser change from an unlikely code-layout interaction with another commit.
- Bisection Status: Not performed. The direct parser candidate should be tested first; full bisection is only needed if that A/B comparison does not reproduce the slowdown.
Recommended Actions
- Verify the suspected commit - Run both affected benchmarks with
RunWithBuildAtHashat 513b9cd40dca and 7bdcc5f5809e. - Review generated code - Compare Tiered JIT disassembly around
TryParseBinaryIntegerStylebefore and after the extra whitespace branch, focusing on inlining, branch layout, and hot-path register pressure. - Bisect only if needed - If the suspected commit does not reproduce, bisect the 513b9cd40dca...8633ad0ebb95 range using both benchmarks together.
Bisection Results
Test Status Culprit Commit Relevance Reported Diff (issue) Bisected Diff (at culprit) ParseSpan(value: "4294967295") ✅ Culprit found 7bdcc5f5809e UNRELATED 11.4→12.8 ns (+12%) 11.7→12.8 ns (+9%) TryParse(value: "4294967295") ✅ Culprit found 7bdcc5f5809e UNCERTAIN 11.7→13.0 ns (+11%) 11.7→13.0 ns (+12%) Total bisection time: 18m 57s
Tests measured: 2/2
Culprit found: 2/2⚠️ Alignment WarningBisection found commits that are unrelated to the regressed tests. This strongly suggests the performance changes are caused by code alignment, binary layout, or other non-functional factors rather than actual regressions. Consider marking as noise.
System.Tests.Perf_UInt32.ParseSpan(value: "4294967295")
- Culprit: 7bdcc5f5809e
- Relevance: UNRELATED
- Explanation: The exact commit https://www.github.com/dotnet/runtime/commit/7bdcc5f5809efdcf620d7ae938a8327d4f96aefa (from https://www.github.com/dotnet/runtime/pull/134672, resolving https://www.github.com/dotnet/runtime/issues/101873) adds fallback parsing for whitespace after signs or parentheses. The benchmark's unsigned, digit-only span "4294967295" remains on the unchanged ordinary-integer fast path; the commit also reports no consistent common-case JIT regression.
Bisection details
- Culprit:
7bdcc5f5809e - Relevance: UNRELATED
- Baseline: 11.413 ns
- Culprit value: 12.773 ns
Per-commit measurements (oldest → newest) (ns)
# Commit Result Measurement 0 513b9cd40dcabaseline/good 11.414 ns 7 d7a058ac0984good 11.415 ns 9 0b87c1cfc454good 11.419 ns 12 5686741ae1f7good 11.669 ns 13 7bdcc5f5809eculprit/bad 12.773 ns 14 dd89a9974aa5bad 12.773 ns 21 8633ad0ebb95head/bad 12.769 ns Helix jobs
Job ID b42fcdc9-2421-4e85-b3d2-098dd5281c320d73e478-8767-4699-8bde-65eb95d39ee34079d5f6-aab7-4af0-9f7e-cd009879c6a4System.Tests.Perf_UInt32.TryParse(value: "4294967295")
- Culprit: 7bdcc5f5809e
- Relevance: UNCERTAIN
- Explanation: The required exact-commit lookup failed because the GitHub tool returned a parameter-header error, so relevance could not be verified without substituting another revision: https://www.github.com/dotnet/runtime/commit/7bdcc5f5809efdcf620d7ae938a8327d4f96aefa
Bisection details
- Culprit:
7bdcc5f5809e - Relevance: UNCERTAIN
- Baseline: 11.731 ns
- Culprit value: 13.044 ns
Per-commit measurements (oldest → newest) (ns)
# Commit Result Measurement 0 513b9cd40dcabaseline/good 11.693 ns 7 d7a058ac0984good 11.690 ns 9 0b87c1cfc454good 11.685 ns 12 5686741ae1f7good 11.683 ns 13 7bdcc5f5809eculprit/bad 13.044 ns 14 dd89a9974aa5bad 13.044 ns 21 8633ad0ebb95head/bad 13.056 ns Helix jobs
Job ID b956d29c-b71a-4c8f-922b-32c5b83c1b22e9801f81-469b-4739-954c-ba6263cc0846261daa51-842c-4e2b-8cb1-296f9fde6dda🤖 Proposed Next Actions
Rule:
R5_look_again— Uncertain — flag for re-triage.Finding confidence 2/5 is below the required 4/5.
# Action Status Details 1 Add label ✅ Done Look Again2 Remove label ⏭️ Skipped (dry-run) untriaged🦆 Pre-bisection Re-analysis (advisory)
✅ Agrees with the pre-bisection plan — advisory only, not applied
Rationale: Both findings are classified as actionable with high finding and change confidence, and their shared likely commit and stack relevance provide mutually reinforcing evidence. The two queued tests directly match the actionable findings, with no indicated noise or bimodality warranting removal.
This critique is advisory in the current configuration and did not change which tests were bisected.
🦆 Re-analysis (advisory)
⚠️ Disputes the analysis — advisory only, not appliedFlags:
underconfident,confidence-aggregation-mismatchRationale: Two independent benchmarks successfully bisected to the same commit with no alignment warnings, supporting a real regression and making the aggregate 2/5 scores too low. The unrelated or uncertain stack relevance justifies withholding an overall culprit, but does not by itself undermine the measured change.
Suggested confidence: Finding Confidence → 4/5; Change Confidence → 4/5 (advisory — not applied)
This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.
Generated by PerfTriageAgent on 2026-09-29 09:48 UTC
Seems related to #134672. It might be unrelated, agent said that the path of these tests didn't change, but it seems worth a look.
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.🔍 Automated Triage Analysis
Summary: Two coordinated
UInt32parsing tests regressed by 11.21-11.86%, and the existing 21-commit bisection isolated 7bdcc5f5809e (numeric parsing). Finding confidence is 5/5 and regression confidence is 5/5: both signals are stable step changes with z-scores of 5.941-8.815, and the isolated commit directly changes the shared integer parsing fast path. Bisection not measured (2 test(s)) — bisection failed before a usable measurement (no Helix verification).Finding Confidence: 3/5 (analysis accuracy)
Regression Confidence: 3/5 (likelihood of true regression)🤖 Proposed Next Actions: Uncertain — flag for re-triage — 2 recommendations, none executed (repository guard) (rule
R5_look_again— see Full Analysis for the table). · 🦆 Re-analysis:⚠️ disputes (overconfident, weak-bisection-evidence, possible-noise)📊 Full Analysis (click to expand)
Summary: Two coordinated
UInt32parsing tests regressed by 11.21-11.86%, and the existing 21-commit bisection isolated 7bdcc5f5809e (numeric parsing). Finding confidence is 5/5 and regression confidence is 5/5: both signals are stable step changes with z-scores of 5.941-8.815, and the isolated commit directly changes the shared integer parsing fast path.Issue Overview
- Issue: dotnet/runtime#134867 - [Perf] Linux/x64: 2 Regressions on 9/25/2026 10:50:38 PM +00:00
- Date: 2026-09-25 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: Linux/x64
- Commits: 513b9cd40dca → 8633ad0ebb95 (21 commits)
- Labels: untriaged, branch-refs/heads/main, perf-regression, arch-x64, os-linux, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
- Runtime stack: Desktop + CoreCLR
- Description: Two
System.Tests.Perf_UInt32benchmarks regressed on Linux/x64 in the tiered microbenchmark configuration. - Benchmark Source:
Perf.UInt32.cs - Historical Data: Available (582 prior measurements across both tests; 616 total measurements including 34 post-change measurements)
Test Analysis
Test Name Baseline Compare Δ% μ σ n z Thresh% Noise? Assessment System.Tests.Perf_UInt32.ParseSpan(value: "4294967295")11.413 ns 12.767 ns +11.86% 11.656 ns 0.126 ns 292 8.815 5.0% N Step change; all 17 post-change measurements remain elevated System.Tests.Perf_UInt32.TryParse(value: "4294967295")11.731 ns 13.045 ns +11.21% 11.732 ns 0.221 ns 290 5.941 5.0% N Step change; all 17 post-change measurements form a stable higher level Legend: μ=historical mean, σ=historical standard deviation, n=prior runs, z=z-score, Thresh%=adaptive threshold
Finding Confidence: 5/5 - Both tests have comprehensive histories of roughly 290 prior runs, low coefficients of variation (1.081% and 1.884%), persistent post-change plateaus, complete commit metadata, and an existing bisection result that isolates one commit.
Candidate Stack Relevance
Commit Touched Paths (summary) Relevance Notes 7bdcc5f5809e src/libraries/Common/src/System/Number.Parsing.Common.cs,src/libraries/System.Private.CoreLib/src/System/Number.Parsing.csShared Directly changes the shared numeric parser exercised by both CoreCLR benchmarks; existing bisection isolated this commit fd53e564dc67 src/coreclr/jit/gentree.cppRelevant CoreCLR JIT change, but limited to SIMD operand ordering with no connection to scalar integer parsing 0b87c1cfc454 src/coreclr/jit/,src/libraries/System.Private.CoreLib/src/System/Double.cs, vector intrinsicsRelevant CoreCLR/CoreLib change, but limited to floating-point integer classification and vector operations; no direct UInt32parsing pathConfirmed Regressions
System.Tests.Perf_UInt32.ParseSpan(value: "4294967295") (+11.86%)
- Delta: 11.413 ns → 12.767 ns (+11.86%)
- Historical Context: μ=11.656 ns (σ=0.126 ns, n=292); z-score=8.815; adaptive threshold=5.0%
- Assessment: Step change with a stable post-change plateau
- Finding Confidence: 5/5 - Comprehensive low-variance history, 17 consistently elevated post-change measurements, complete commit data, and a single-commit bisection result
- Regression Confidence: 5/5 - Extreme outlier, direct shared-parser code hit, correlated sibling regression, and bisection to one commit
- Likely Cause: 7bdcc5f5809e - "Allow whitespace after leading numeric signs and parentheses"
- Affected Areas: CoreLib numeric parsing, integer parsing fast path, JIT code layout
- Stack Relevance: Shared
- Rationale: The commit adds a whitespace-retry condition to
TryParseBinaryIntegerStyleinSystem.Number.Parsing.cs, the shared parser reached byuint.Parse(ReadOnlySpan<char>). The whitespace-free benchmark should not take the retry, so the most likely mechanism is the added branch changing hot-method code generation, layout, or inlining rather than execution of the slow fallback; the commit description independently notes layout-sensitive common-case timing. The existing bisection isolated this exact commit, and the change landed nine minutes before the reported regression time. - Candidates Considered:
- fd53e564dc67 - CoreCLR JIT change, but only for SIMD operand ordering
- 0b87c1cfc454 - Floating-point classification and vector changes do not intersect scalar
UInt32parsing
- Verification Status: Bisected to commit 7bdcc5f5809e
- The runtime issue's prior automated triage reports that the 21-commit range was narrowed to this commit. No fresh isolated Helix rerun data was available in the issue discussion.
System.Tests.Perf_UInt32.TryParse(value: "4294967295") (+11.21%)
- Delta: 11.731 ns → 13.045 ns (+11.21%)
- Historical Context: μ=11.732 ns (σ=0.221 ns, n=290); z-score=5.941; adaptive threshold=5.0%
- Assessment: Step change with a stable post-change plateau
- Finding Confidence: 5/5 - Comprehensive low-variance history, 17 consistently elevated post-change measurements, complete commit data, and a single-commit bisection result
- Regression Confidence: 5/5 - Extreme outlier, direct shared-parser code hit, correlated sibling regression, and bisection to one commit
- Likely Cause: 7bdcc5f5809e - "Allow whitespace after leading numeric signs and parentheses"
- Affected Areas: CoreLib numeric parsing, integer parsing fast path, JIT code layout
- Stack Relevance: Shared
- Rationale:
uint.TryParse(string, out uint)reaches the same binary-integer parsing implementation modified by this commit. Because"4294967295"contains neither a sign nor whitespace, the new retry path is not expected to execute; the correlated slowdown instead points to a code-size, branch-placement, or inlining effect in the hot parser. The strong step signal and single-commit bisection make the attribution substantially stronger than temporal correlation alone. - Candidates Considered:
- fd53e564dc67 - CoreCLR JIT change, but only for SIMD operand ordering
- 0b87c1cfc454 - Floating-point classification and vector changes do not intersect scalar
UInt32parsing
- Verification Status: Not independently verified; corroborated by the sibling
ParseSpanbisection to 7bdcc5f5809e- The identical step date and shared implementation strongly corroborate
TryParse, but a separate isolated rerun for this benchmark was not included in the issue discussion.
- The identical step date and shared implementation strongly corroborate
Open Questions
- Historical Data: Sufficient and low variance; no missing-test history
- Confidence Limitations: The exact machine-code mechanism has not been demonstrated. The bisection identifies the source commit, but the issue discussion does not include isolated baseline/culprit measurements or disassembly.
- Verification Needed: Compare tiered JIT disassembly, native code size, branch placement, and inlining decisions for
TryParseBinaryIntegerStylebefore and after7bdcc5f5809e; then rerun a layout-preserving fast-path adjustment. - Bisection Status: Completed (21 commits narrowed to
7bdcc5f5809e)
Recommended Actions
- Inspect generated code - Diff tiered x64 disassembly and code size for
TryParseBinaryIntegerStyleat 513b9cd40dca and 7bdcc5f5809e, focusing on the new retry branch, inlining, and hot/cold layout. - Preserve the fast path - Test moving the uncommon whitespace retry into a no-inline helper or otherwise keeping it out of the hot integer-parsing body, while retaining the new parsing behavior.
- Verify the fix - Rerun both affected benchmarks on the culprit and proposed-fix commits; collect instruction-retired and branch-misprediction counters if the disassembly does not make the cost clear.
Bisection Results
Test Status Culprit Commit Relevance Reported Diff (issue) Bisected Diff (at culprit) ParseSpan(value: "4294967295") ❌ Not measured - - 11.4→12.8 ns (+12%) — (not measured) TryParse(value: "4294967295") ❌ Not measured - - 11.7→13.0 ns (+11%) — (not measured) Total bisection time: 122m 18s
Tests measured: 0/2
Culprit found: 0/2System.Tests.Perf_UInt32.ParseSpan(value: "4294967295")
- Status: ❌ Not measured
System.Tests.Perf_UInt32.TryParse(value: "4294967295")
- Status: ❌ Not measured
🤖 Proposed Next Actions
Rule:
R5_look_again— Uncertain — flag for re-triage.Recommendation only: Issue is outside
dotnet/perf-autofiling-issues(already transferred or submitted directly to another repository). These are recommendations only; no next actions are executed.Finding confidence 3/5 is below the required 4/5.
# Action Status Details 1 Would add label Not executed (recommendation only) Look Again2 Would remove label Not executed (recommendation only) untriaged🦆 Pre-bisection Re-analysis (advisory)
✅ Agrees with the pre-bisection plan — advisory only, not applied
Rationale: Both findings are consistently classified as actionable with maximum finding and change confidence, no bimodal history, and the same plausible culprit. Nothing in the packet indicates either candidate is obviously noisy or that a clearly regressed test was omitted.
This critique is advisory in the current configuration and did not change which tests were bisected.
🦆 Re-analysis (advisory)
⚠️ Disputes the analysis — advisory only, not appliedFlags:
overconfident,weak-bisection-evidence,possible-noiseRationale: Both bisections failed without identifying a culprit, so the per-finding 5/5 change confidence and actionable classification are not justified by the evidence shown. No historical statistics are provided to independently establish a sustained regression over noise.
Suggested confidence: Finding Confidence → 2/5; Change Confidence → 2/5 (advisory — not applied)
This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.
Generated by PerfTriageAgent on 2026-10-01 22:41 UTC
Run Information
Regressions in System.Tests.Perf_UInt32
Test Report
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Details
System.Tests.Perf_UInt32.ParseSpan(value: "4294967295")
ETL Files
Histogram
JIT Disasms
System.Tests.Perf_UInt32.TryParse(value: "4294967295")
ETL Files
Histogram
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository