crossgen2 fails with 139 exit code on Arm64

This is causing a failure in the VMR for .NET 9 Preview 1 when building the installer repo. It fails in the Ubuntu2204Arm64_Offline_MsftSdk_arm64 job (example build [internal link]).

Error:

/vmr/src/installer/src/redist/targets/Crossgen.targets(163,5): error MSB6006: "crossgen2" exited with code 139. [/vmr/src/installer/src/redist/redist.csproj]

Here is a binlog:
sourcebuild.installer.zip

We need this resolved for .NET 9 Preview 1.

@jkoritzinsky

@jkoritzinsky - Can you help investigate? This was working recently and regressed.

I can try but I don't have a Linux ARM64 machine. Do you know the first VMR build that this broke in? I'll see if I can narrow down where the failure from.

@EgorBo

It looks like the most recent passing build was https://dev.azure.com/dnceng/internal/_build/results?buildId=2359219&view=results and the first failing build was https://dev.azure.com/dnceng/internal/_build/results?buildId=2359821&view=results.

The runtime range of commits between these two is dotnet/runtime@8accd80...e99836a

The commits that pop out to me as possibly causing crossgen2 crashes are:

I think the second is more likely than the first.

cc: @EgorBo

Checking now

It's getting a different error now and failing with error code 134:

/vmr/src/installer/src/redist/targets/Crossgen.targets(172,5): *** stack smashing detected ***: terminated [/vmr/src/installer/src/redist/redist.csproj]

binlog

Failing build (internal link)

It's consistently failing but not always failing with the same error. The two errors mentioned in this issue above are all I've seen so far.

Interestingly, the build succeeded in this recent run (internal link). But I'm not sure I trust that the issue is resolved. Is it perhaps an intermittent issue? The source changes that exist between the two builds (failing and passing) don't include any runtime changes at all.

Yes, it is possible that the failure is intermittent given your sample with the 134 exit code. It's likely a native corruption somewhere in crossgen2. If this is using a live crossgen2 (which I think it is), then it could be anywhere in the JIT, GC, or CoreCLR runtime. This also means it could have appeared before the first instance during the time frame where the build was failing before getting to installer, which makes this harder to diagnose...

There's been 3 other builds since that one that passed and they've all failed.

I don't think this has anything to do with R2R itself. The failure in the linked issue is due to an unhandled exception. The error message is:

      Unhandled exception. System.CommandLine.CommandLineException: Target OS 'centos' is not supported
         at System.CommandLine.Helpers.GetTargetOS(String) in /_/src/runtime/src/coreclr/tools/Common/CommandLineHelpers.cs:line 84
         at System.CommandLine.CliArgument`1.<>c__DisplayClass8_0.<set_CustomParser>b__0(ArgumentResult argumentResult, Object& parsedValue)
         at System.CommandLine.Parsing.ArgumentResult.ValidateAndConvert(Boolean)
         at System.CommandLine.Parsing.CommandResult.ValidateOptions(Boolean)
         at System.CommandLine.Parsing.CommandResult.Validate(Boolean)
         at System.CommandLine.Parsing.ParseOperation.Validate()
         at System.CommandLine.Parsing.ParseOperation.Parse()
         at System.CommandLine.Parsing.CliParser.Parse(CliCommand, IReadOnlyList`1, String , CliConfiguration )
         at System.CommandLine.CliConfiguration.Invoke(String[])
         at ILCompiler.Program.Main(String[]) in /_/src/runtime/src/coreclr/tools/aot/ILCompiler/Program.cs:line 753

That was a separate issue (identified in #4111 (comment)), that has since been fixed.

@mthalman

@mthalman mystery solved. It is not a new issue. The thing is that crossgen doesn't use the current build of runtime, but rather an older version (it uses the dotnet sdk in the .dotnet folder). In this case, it used version 9.0.0-preview.2.24080.1 built from commit d40c654c274fe4f4afe66328f0599130f3eb2ea6 that predates the fix in main by two days.

Ok, I'll update the VMR to use a more recent SDK and check the results.

This is continuing to be an issue. The most recent related error that I've seen is in runtime:

/vmr/src/runtime/artifacts/bin/Crossgen2Tasks/Release/net9.0/Microsoft.NET.CrossGen.targets(357,5): error NETSDK1096: Optimizing assemblies for performance failed. You can either exclude the failing assemblies from being optimized, or set the PublishReadyToRun property to false. [/vmr/src/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.sfxproj]

Link to build (internal link)

The bootstrap work to resolve this is still pending.

This should be fixed by dotnet/installer#18763

Hi,
I also started to hit this issue, just under FreeBSD, could I ask for some info what was the issue and/or which commit fixed that?

Reopening this because it's not yet resolved for source build in the VMR. We're still waiting for a bootstrap of the VMR onto an SDK build which contains the fix for this issue. But that's currently blocked on #4206.

@sec

Hi, I also started to hit this issue, just under FreeBSD, could I ask for some info what was the issue and/or which commit fixed that?

@sec - This is the fix: dotnet/runtime#97817. But it's not enough to have the fix in the source that you're currently building. You need to build with an SDK that has this fix. That's what we're currently working towards for the VMR, to get it bootstrapped with an SDK that has this fix in it. That will allow builds to work using that SDK.

@mthalman

@mthalman Thanks for the info. If I get it correctly, I should wait for your work to be done, then I can crosscompile SDK for FreeBSD and use it as bootstrap for preview-2, which should be fine. BUT, as looking at this, it's a bug in libunwind - which under FreeBSD is taken from ports - looking in there I've already found libunwind/libunwind#715 - this should also be fixed otherwise I will still hit this in future :)

The fix was flowed into the VMR with dotnet/installer#19145

Navigation Menu

Repository navigation

crossgen2 fails with 139 exit code on Arm64 #4007

35 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

crossgen2 fails with 139 exit code on Arm64 #4007

Description

Activity

mthalman commented on Jan 23, 2024

jkoritzinsky commented on Jan 23, 2024

jkoritzinsky commented on Jan 23, 2024

EgorBo commented on Jan 23, 2024

mthalman commented on Jan 24, 2024

mthalman commented on Jan 24, 2024

mthalman commented on Jan 24, 2024

jkoritzinsky commented on Jan 24, 2024

mthalman commented on Jan 25, 2024

35 remaining items

mthalman commented on Feb 16, 2024

janvorli commented on Feb 17, 2024

mthalman commented on Feb 20, 2024

ellahathaway commented on Mar 4, 2024

mthalman commented on Mar 4, 2024

mthalman commented on Mar 8, 2024

sec commented on Mar 14, 2024

mthalman commented on Mar 22, 2024

mthalman commented on Mar 22, 2024

sec commented on Mar 22, 2024

mthalman commented on Apr 1, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions