JIT: (bug) rorx reg, [stack], imm doesn't kill the destination register's GC liveness (GC hole) #135390

When rorx with a stack operand overwrites a register that held a GC reference, the register stays reported as a live gcref until the end of the instruction group.

Minimal Repro

using System;
using System.Numerics;
using System.Runtime.CompilerServices;
using System.Threading;

public class Program
{
    static object s_o = new object();

    [MethodImpl(MethodImplOptions.NoInlining)]
    static object Get(ref int v) => s_o;

    [MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.AggressiveOptimization)]
    static int Test(int[] arr, int v)
    {
        object o = Get(ref v);
        GC.KeepAlive(o);
        int r = (int)BitOperations.RotateRight((uint)v, 5);
        int sum = 0;
        for (int i = 0; i < arr.Length; i++)
            sum += arr[i];
        return sum + r;
    }

    public static void Main()
    {
        new Thread(() => { while (true) GC.Collect(); }) { IsBackground = true }.Start();
        int[] a = new int[1];
        int res = 0;
        for (int i = 0; i < 50_000_000; i++)
            res += Test(a, 0x12345678);
        Console.WriteLine(res);
    }
}

Requires BMI2. The failure below is on a Checked runtime (object validation).

Expected

438774144

Actual

After a few seconds:

Assert failure(PID ..., Thread: ...): !CREATE_CHECK_STRING(!"Detected use of a corrupted OBJECTREF. Possible GC hole.")
CORECLR! Object::ValidateInner + 0x103
CORECLR! Object::Validate + 0xB0
CORECLR! TGcInfoDecoder<AMD64GcInfoEncoding>::ReportRegisterToGC + 0x1AD
CORECLR! TGcInfoDecoder<AMD64GcInfoEncoding>::EnumerateLiveSlots + 0x1582
    File: src\coreclr\vm\object.cpp:618

JitDisasmWithGC output (the method is fully interruptible):

G_M7200_IG02:        ; bbWeight=1, gcrefRegs=0008 {rbx}, byrefRegs=0000 {}, byref, isz
       lea      rcx, [rsp+0x38]
       call     [Program:Get(byref):System.Object]
                            ; gcrRegs +[rax]
                            ; gcr arg pop 0
       rorx     eax, dword ptr [rsp+0x38], 5        ; eax now holds an int, but rax is still reported as a gcref
       xor      ecx, ecx
       mov      edx, dword ptr [rbx+0x08]
       test     edx, edx
       jle      SHORT G_M7200_IG05
G_M7200_IG03:        ; bbWeight=0.90, gcrefRegs=0008 {rbx}, byrefRegs=0000 {}, byref, isz
                            ; gcrRegs -[rax]

Regression?

This codegen first shows up in .NET 10 (.NET 8.0/9.0 use ror on a register). .NET 10.0 and 11.0-rc2 emit the same rorx but don't crash with this repro: release runtimes don't validate object references, and the 32-bit value is outside the GC heap.

Notes

• emitOutputSV (emitxarch.cpp) calls emitGCregDeadUpd only for IF_RWR_SRD, IF_RRW_SRD, IF_RWR_RRD_SRD, IF_RRW_RRD_SRD and IF_RWR_RWR_SRD; rorx uses IF_RWR_SRD_CNS, which hits default: break;.
• emitOutputCV has the same omission for IF_RWR_MRD_CNS (rorx reg, [static], imm).
• With a 64-bit value that points into the GC heap (e.g. RotateRight(ulong, n)), a release runtime would mark/relocate a bogus object.