.NET 10.0.11 crashes while trying to JIT a method

[filipnavara]

We started observing this crash in our application:

[0x0]   KERNELBASE!RaiseFailFastException+0x188   0xc0ec7b880   0x7ff80268bcb6   
[0x1]   coreclr!WatsonLastChance+0x1b6   0xc0ec7be60   0x7ff8026c7fd4   
[0x2]   coreclr!EEPolicy::LogFatalError+0x65c   0xc0ec7bec0   0x7ff8026c735d   
[0x3]   coreclr!EEPolicy::HandleFatalError+0x129   0xc0ec7cb80   0x7ff8025d10fc   
[0x4]   coreclr!CLRVectoredExceptionHandlerPhase3+0xdb6ec   (Inline Function)   (Inline Function)   
[0x5]   coreclr!CLRVectoredExceptionHandlerPhase2+0xdb710   0xc0ec7d180   0x7ff8024f59c1   
[0x6]   coreclr!CLRVectoredExceptionHandler+0x181   0xc0ec7d1e0   0x7ff8024f57fd   
[0x7]   coreclr!CLRVectoredExceptionHandlerShim+0xdd   0xc0ec7d250   0x7ff87f106896   
[0x8]   ntdll!RtlpCallVectoredHandlers+0xd6   0xc0ec7d2a0   0x7ff87f105c66   
[0x9]   ntdll!RtlDispatchException+0x206   0xc0ec7d330   0x7ff87f2243fe   
[0xa]   ntdll!KiUserExceptionDispatch+0x2e   0xc0ec7d580   0x7ff80242db69   
[0xb]   coreclr!StringObject::GetBuffer+0x4   (Inline Function)   (Inline Function)   
[0xc]   coreclr!StringObject::RefInterpretGetStringValuesDangerousForGC+0x4   (Inline Function)   (Inline Function)   
[0xd]   coreclr!StringLiteralEntry::GetStringData+0x11   (Inline Function)   (Inline Function)   
[0xe]   coreclr!EEUnicodeStringLiteralHashTableHelper::CompareKeys+0x75   0xc0ec7dd00   0x7ff80242dc74   
[0xf]   coreclr!EEHashTableBase<EEStringData *,EEUnicodeStringLiteralHashTableHelper,1>::FindItemSpeculative+0x39   (Inline Function)   (Inline Function)   
[0x10]   coreclr!EEHashTableBase<EEStringData *,EEUnicodeStringLiteralHashTableHelper,1>::GetValueSpeculative+0x39   (Inline Function)   (Inline Function)   
[0x11]   coreclr!GlobalStringLiteralMap::GetStringLiteral+0x39   (Inline Function)   (Inline Function)   
[0x12]   coreclr!StringLiteralMap::GetStringLiteral+0xc4   0xc0ec7dd40   0x7ff80242df2a   
[0x13]   coreclr!LoaderAllocator::GetStringObjRefPtrFromUnicodeString+0x43   (Inline Function)   (Inline Function)   
[0x14]   coreclr!ModuleBase::ResolveStringRef+0x122   (Inline Function)   (Inline Function)   
[0x15]   coreclr!CEECodeGenInfo::constructStringLiteral+0x18a   0xc0ec7dda0   0x7ff819247571   
[0x16]   clrjit!Compiler::fgMorphConst+0x12c   (Inline Function)   (Inline Function)   
[0x17]   clrjit!Compiler::fgMorphTree+0xfd1   0xc0ec7de30   0x7ff8192449c0   
[0x18]   clrjit!Compiler::fgMorphArgs+0xa0   0xc0ec7df20   0x7ff819243bcd   
[0x19]   clrjit!Compiler::fgMorphCall+0x27d   0xc0ec7df70   0x7ff819247a2e   
[0x1a]   clrjit!Compiler::fgMorphTree+0x148e   0xc0ec7e080   0x7ff819248f86   
[0x1b]   clrjit!Compiler::fgMorphSmpOp+0xdc6   0xc0ec7e170   0x7ff819247863   
[0x1c]   clrjit!Compiler::fgMorphTree+0x12c3   0xc0ec7e2f0   0x7ff819248f86   
[0x1d]   clrjit!Compiler::fgMorphSmpOp+0xdc6   0xc0ec7e3e0   0x7ff819247863   
[0x1e]   clrjit!Compiler::fgMorphTree+0x12c3   0xc0ec7e560   0x7ff819245258   
[0x1f]   clrjit!Compiler::fgMorphStmts+0xe8   0xc0ec7e650   0x7ff81927e77c   
[0x20]   clrjit!Compiler::fgMorphBlock+0x48c   0xc0ec7e690   0x7ff81927e12c   
[0x21]   clrjit!Compiler::fgMorphBlocks+0xdc   0xc0ec7e6e0   0x7ff8192dd27e   
[0x22]   clrjit!Phase::Run+0x24   (Inline Function)   (Inline Function)   
[0x23]   clrjit!DoPhase+0x5f   (Inline Function)   (Inline Function)   
[0x24]   clrjit!Compiler::compCompile+0xeee   0xc0ec7e730   0x7ff8192baf84   
[0x25]   clrjit!Compiler::compCompileHelper+0x9a4   0xc0ec7ed00   0x7ff81931ea91   
[0x26]   clrjit!Compiler::compCompile+0x3f1   0xc0ec7edc0   0x7ff81929739a   
[0x27]   clrjit!jitNativeCode+0x25a   0xc0ec7eeb0   0x7ff81929699a   
[0x28]   clrjit!CILJit::compileMethod+0xba   0xc0ec7f060   0x7ff802412ac7   
[0x29]   coreclr!invokeCompileMethod+0xba   (Inline Function)   (Inline Function)   
[0x2a]   coreclr!UnsafeJitFunctionWorker+0x3bb   0xc0ec7f100   0x7ff802411aa7   
[0x2b]   coreclr!UnsafeJitFunction+0x51f   0xc0ec7f1d0   0x7ff80241144a   
[0x2c]   coreclr!MethodDesc::JitCompileCodeLocked+0xce   0xc0ec7f510   0x7ff8024110cd   
[0x2d]   coreclr!MethodDesc::JitCompileCodeLockedEventWrapper+0x521   0xc0ec7f5e0   0x7ff8024109f8   
[0x2e]   coreclr!MethodDesc::JitCompileCode+0x538   0xc0ec7f710   0x7ff80240f7a9   
[0x2f]   coreclr!MethodDesc::PrepareILBasedCode+0x241   0xc0ec7f7b0   0x7ff80240ae93   
[0x30]   coreclr!MethodDesc::PrepareCode+0x10   (Inline Function)   (Inline Function)   
[0x31]   coreclr!TieredCompilationManager::CompileCodeVersion+0xb3   0xc0ec7f830   0x7ff80240a7a3   
[0x32]   coreclr!TieredCompilationManager::OptimizeMethod+0x1c   (Inline Function)   (Inline Function)   
[0x33]   coreclr!TieredCompilationManager::DoBackgroundWork+0x2d3   0xc0ec7f950   0x7ff80254e956   
[0x34]   coreclr!TieredCompilationManager::BackgroundWorkerStart+0xd2   0xc0ec7faa0   0x7ff80254e83d   
[0x35]   coreclr!TieredCompilationManager::BackgroundWorkerBootstrapper1+0x5d   0xc0ec7faf0   0x7ff8024d212e

Notably, the issue had the exact same stack trace on 5 different machines. While it suspiciously looks like a heap corruption inside CoreCLR the fault address is always identical as are the symptoms.

I shared the memory dump on https://developercommunity.visualstudio.com/t/NET-CLR-JIT-crashes-with-0xC0000005-in/11142515. Both the dump and the app are available on request.

We also tried to see if the unreleased GC hole fix (#131708) made any difference and we internally built a version with latest .NET 10 daily build. Unfortunately it didn't resolve the issue.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[EgorBo]

Doesn't look to be a JIT issue. Someone corrupted the global string-literal map, JIT just happens to run into it deterministcally; every subsequent JIT of any ldstr hashing to that bucket re-walks it and dies identically. It's interesting that it looks like someone modifies the lower part of the string-literal slot with 0xFF2D2D2D (looks like a dark-theme color in that app - many instances of that in the dump).

I don't have ideas for futher investigation. Perhaps @jkotas @dotnet/jit-contrib might have?

From Filip's words, the GC heap verifies just fine.

[EgorBo]

I've looked into the public sources of the CEF wrapper this app uses with AI:

cc @filipnavara could this cause the issue in theory?

[filipnavara]

Thanks, that sounds like a plausible theory. It's not immediately obvious how would we access the CefBrowserSettings memory after it was already freed. Then again, it was not supposed to be freed like this in the first place so it's only prudent to fix that and re-test.

[filipnavara]

We had no reports of the crash for couple of hours, so let's hope for the best and close this issue 🤞 Thanks a lot for help!

[JulieLeeMSFT]

@filipnavara, is it still stable? Then, please close this ticket.

[filipnavara]

D'oh. I thought I pressed the "Close with comment" button. Sorry.

[JulieLeeMSFT]

No problem. Thanks for confirming that it's now resolved.