Repository navigation
[release/11.0] Fix negative tmp-buffer size in vxsort align_vectorized - #135229
Merged
Merged
Conversation
## Issue Server/Workstation GC background threads can crash with an access violation (read) inside coreclr!memcpy, called from vxsort<T,M,Unroll,Shift>::vectorized_partition, itself called from do_vxsort during mark_phase/plan_phase sorting of the GC mark list. This reproduces the crash originally reported in #86548 ("The size calculated to be negative and cause Access Violation when doing vxsort vectorized_partition"), which was closed without a fix in 2023 due to lack of a reliable repro. Analysis of a production Windows crash dump (Second Chance Exception c0000005, AMD EPYC/Zen4 AVX-512) root-caused the same failure independently and pinpointed the exact algorithmic defect below. ## Root cause In vxsort::align_vectorized(), when the left read pointer isn't vector-aligned, the code pre-aligns by reading a full vector starting before `left`. The low `L = -leftAlign` lanes of that vector are "junk" elements that lie before the current [left, right] partition range (already settled by an ancestor partition step), while the remaining lanes are real elements of the current range. Both junk and real lanes are compared against the current pivot and compacted together (via compress-store or an order-preserving permute), so junk elements can land on either side of the pivot - they are not guaranteed to all be <= pivot. The code advanced `tmpLeft` by the true, data-dependent count of <=pivot lanes (`ltPopCountLeftPart`), but advanced `tmpStartLeft` - the base pointer later used to size the copy-back region (`leftTmpSize = tmpLeft - tmpStartLeft`) - by the *constant* `L`, implicitly assuming all L junk lanes are <=pivot. When fewer than L junk lanes are actually <=pivot, tmpStartLeft overshoots tmpLeft, producing a negative leftTmpSize. That negative value, reinterpreted as an unsigned byte count in the subsequent memcpy, causes a massive out-of-bounds read and the access violation. The formula was validated against both the original #86548 report (L=7, ltMask=0x3f -> ltPopCountLeftPart=2 -> deficit of -5 elements = -20 bytes, exactly matching the reported value) and against the investigated crash dump (1-element/4-byte deficit reproduced from register/stack state captured in the dump). The right-side counterpart (`tmpStartRight -= rightAlign & rai`) has the same class of defect for junk elements read beyond `right`. It was partially papered over by `rtPopCountRightPart = max(mask_popcount( rtMask), rightAlign)`, which avoids the crash but can instead leave an uninitialized "hole" in the tmp buffer that gets silently copied back into the array as if it were valid sorted data. ## Fix Replace the constant junk-lane skip on both sides with the exact, data-dependent count of junk lanes that landed on the "keep" side of the pivot, computed via a popcount masked to just the junk-lane bits (low L bits for the left side, high `rightAlign` bits for the right side). This removes the incorrect assumption entirely instead of papering over its symptoms, and makes the previous `max()` band-aid on the right side unnecessary (removed). Validated by a full CoreCLR Debug x64 native build (build-runtime.cmd), confirming the change compiles cleanly across the WKS/SVR GC, gcsample, and NativeAOT runtime consumers, for both AVX2 and AVX-512 machine traits and both int32/int64 element types. Fixes #86548 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b029aa4e-2a19-4582-b9e5-4918cb4c3095
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @anicka-net, @dotnet/gc |
JulieLeeMSFT
added
Servicing-approved
JulieLeeMSFT
approved these changes
Oct 6, 2026
Member
|
/ba-g known issues. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #134881 to release/11.0
/cc @janvorli
Customer Impact
Server/Workstation GC background threads can crash with an access violation inside coreclr!memcpy, called from vxsort code during mark_phase/plan_phase sorting of the GC mark list.
The issue was first reported in May 2023, but we were never able to reproduce it.
Now a customer started to hit this issue on a regular basis in Azure App Service powering a multi-tenant SaaS solution. It resulted in SLA violations impacting the customer.
The culprit is a bug in the vxsort that has been in vxsort code ever since it was implemented in .NET Core 6
Regression
Testing
coreclr tests executed with server GC with GC stress C
Risk
Low, the change is local to boundary-lane classification in vxsort and preserves the existing sorting and buffer-management logic. It forces out-of-range lanes onto the side from which they are subsequently trimmed, preventing invalid copy sizes and contamination of the sorted output