Skip to content

.NET 7 - Segmentation Fault on BeagleBone Black (arm32) #78657

Open
Open
.NET 7 - Segmentation Fault on BeagleBone Black (arm32)#78657
Labels
area-VM-coreclr
Milestone
Future

Description

@Limb
Limb
opened on Nov 9, 2022

I'm getting a Segmentation Fault after installing .NET 7.0 SDK (v7.0.100) - Linux Arm32 on a BeagleBone Black running Debian 10.4. I tried both an existing upgrade from 6.0, as well as a fresh SD card image.

Running just the dotnet command works fine, but any other command (even dotnet --version) results in a segmentation fault.

debian@beaglebone:~$ dotnet

Usage: dotnet [options]
Usage: dotnet [path-to-application]

Options:
  -h|--help         Display help.
  --info            Display .NET information.
  --list-sdks       Display the installed SDKs.
  --list-runtimes   Display the installed runtimes.

path-to-application:
  The path to an application .dll file to execute.
debian@beaglebone:~$ dotnet --version
Segmentation fault

Running strace dotnet --version the command seems to output properly (however I am not a strace expert by any means but I am seeing a proper "write" command in the output):

clock_gettime(CLOCK_MONOTONIC, {tv_sec=1948, tv_nsec=283537198}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=1948, tv_nsec=285314775}) = 0
cacheflush(0xb55e7948, 0xb55e7960, 0)   = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=1948, tv_nsec=290090158}) = 0
cacheflush(0xab7cab64, 0xab7cabd0, 0)   = 0
ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0
write(69, "7.0.100", 77.0.100)                 = 7
write(69, "\n", 1
)                      = 1
unlink("/tmp/dotnet-diagnostic-13773-193303-socket") = 0
futex(0x216f248, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x216f208, FUTEX_WAKE_PRIVATE, 1) = 1
unlink("/tmp/clr-debug-pipe-13773-193303-in") = 0
unlink("/tmp/clr-debug-pipe-13773-193303-out") = 0
write(4, "\3", 1)                       = 1
clock_gettime(CLOCK_MONOTONIC, {tv_sec=1948, tv_nsec=318814829}) = 0
munmap(0xb6ca3000, 225874)              = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Activity

  2. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Nov 9, 2022
  3. marcpopMSFT
    self-assigned this
    on Nov 15, 2022
  4. marcpopMSFT
    transferred this issue fromdotnet/sdkon Nov 21, 2022
  5. ghost added
    area-Host
    on Nov 21, 2022

  7. marcpopMSFT commented on Nov 21, 2022

    @marcpopMSFT
    marcpopMSFT
    on Nov 21, 2022
    Member

    Routing to runtime as a likely issue in the runtime or host itself.

  8. elinor-fung commented on Nov 22, 2022

    @elinor-fung
    elinor-fung
    on Nov 22, 2022
    Member

    @Limb Would it be possible for you to get/share a core dump of the crash?

  9. Limb commented on Nov 22, 2022

    @Limb
    Limb
    on Nov 22, 2022
    Author

    @elinor-fung Hopefully I created the core dump successfully, please see core

  10. elinor-fung commented on Nov 23, 2022

    @elinor-fung
    elinor-fung
    on Nov 23, 2022
    Member

    Seems to be in Microsoft.TemplateEngine.Cli.dll:

    [0x0]   0xb556c001!+   0xbebcc000   0xabb0b04b   
[0x1]   Microsoft_TemplateEngine_Cli + 0x5c04b!Microsoft_TemplateEngine_Cli+0x5c04b   0xbebcc000   0xabb09ca1   
[0x2]   Microsoft_TemplateEngine_Cli + 0x5aca1!Microsoft_TemplateEngine_Cli+0x5aca1   0xbebcc050   0xabaec809   
[0x3]   Microsoft_TemplateEngine_Cli + 0x3d809!Microsoft_TemplateEngine_Cli+0x3d809   0xbebcc068   0xae8333a9   
[0x4]   dotnet_ae760000 + 0xd33a9!dotnet_ae760000+0xd33a9   0xbebcc080   0xae82e42b   
[0x5]   dotnet_ae760000 + 0xce42b!dotnet_ae760000+0xce42b   0xbebcc0b8   0x0

    @dotnet/dotnet-diag I'm failing at getting managed stacks from the dump. Can someone tell me what I'm missing?
    I'm running a docker container with linux arm32 and getting this:

    (lldb) sosstatus
Target OS: LINUX Architecture: Arm ProcessId: 6323 (0x18B3)
#0 .NET Core runtime at 00000000B677F000 size 0046E2B9 index ad7c731feb7243bba7f5fa9b0f931444aaad01e0
    Runtime module path: /home/issues/libcoreclr.so
    Runtime module directory: /home/issues
    DAC: /home/issues/libmscordaccore.so


Current symbol store settings:
-> Cache: /root/.dotnet/symbolcache
-> Server: https://msdl.microsoft.com/download/symbols/ Timeout: 4 RetryCount: 0
GC memory usage for managed SOS components: 628,552 bytes
(lldb) clrstack
Failed to load data access module, 0x80004002
Can not load or initialize libmscordaccore.so. The target runtime may not be initialized.

For more information see https://go.microsoft.com/fwlink/?linkid=2135652
ClrStack  failed

  11. michaldobrodenka commented on Nov 23, 2022

    @michaldobrodenka
    michaldobrodenka
    on Nov 23, 2022

    I think that dump might not be complete or doesn't contain managed info. I'm not expert on this, but I had problems with core dumps some time ago (also on arm32 ).

    image

    It usually means:

    • disk space issue - did you have enough space on partition where you made dump?
    • limit set to dump - maybe something like ulimit -S -c unlimited will help - not to limit dumps
    • correct flags - to include all info including managed - I use echo 0xff > /proc/<PID>/coredump_filter in systemd for my process

    but i may be wrong :)

  12. tommcdon commented on Nov 23, 2022

    @tommcdon
    tommcdon
    on Nov 23, 2022
    Member

    https://github.com/orgs/dotnet/teams/dotnet-diag I'm failing at getting managed stacks from the dump. Can someone tell me what I'm missing?
    I think that dump might not be complete or doesn't contain managed info. I'm not expert on this, but I had problems with core dumps some time ago (also on arm32 ).

    Hi @elinor-fung and @michaldobrodenka! It does indeed look like the dump is incomplete. For more details on crash dump generation see here. I suggest collecting a full dump by setting DOTNET_DbgMiniDumpType=4. Hope this helps!

  13. elinor-fung commented on Nov 23, 2022

    @elinor-fung
    elinor-fung
    on Nov 23, 2022
    Member

    Thanks, @michaldobrodenka and @tommcdon.

    @Limb could you try collecting a full dump per #78657 (comment)?

  14. Limb commented on Nov 28, 2022

    @Limb
    Limb
    on Nov 28, 2022
    Author

    @elinor-fung here is the core dump using DOTNET_DbgMiniDumpType=4

  15. Limb commented on Nov 28, 2022

    @Limb
    Limb
    on Nov 28, 2022
    Author

    I will add that by chance I happened to run the dotnet --version command again after generating the dump and it worked, but executing the command again resulted in a segmentation fault. So this issue does seem to be inconsistent in generating a fault all the time.

    debian@beaglebone:~$ dotnet --version
7.0.100
debian@beaglebone:~$ dotnet --version
Segmentation fault (core dumped)
debian@beaglebone:~$ dotnet --version
Segmentation fault (core dumped)
debian@beaglebone:~$ dotnet --version
Segmentation fault (core dumped)
debian@beaglebone:~$ dotnet --version
7.0.100

  16. 4 remaining items

  17. hoyosjs commented on Dec 17, 2022

    @hoyosjs
    hoyosjs
    on Dec 17, 2022
    Member

    Using IL Spy I see the last two frames have md tokens that are Microsoft.DotNet.Cli.Program.Main -> Microsoft.DotNet.Cli.Program.ProcessArgs. They are both r2r, but the disassembly is also odd for ARM. (The oddity was LLDB failint to detect it was thumbv2 assembly). Maybe running with DOTNET_ReadyToRun=0 might help?

    The assembly right before the call is:

    d152e: 49 f6 6a 74     movw    r4, #40810
d1532: c0 f2 11 04     movt    r4, #17
d1536: 7c 44           add     r4, pc
d1538: 23 68           ldr     r3, [r4]
d153a: 98 47           blx     r3

    with fixups it's

       0xae83100e   movw   r4, #0xc3b6
   0xae831012   movt   r4, #0x11
   0xae831016   add    r4, pc
   0xae831018   ldr    r1, [r4]
   0xae83101a   blx    r1

    The assembly doesn't explain the next frames in the stack though, so not sure I am interpreting this quite right.

  18. JustSuperHuman commented on Mar 10, 2023

    @JustSuperHuman
    JustSuperHuman
    on Mar 10, 2023

    I'm wondering if anyone has an update on this. Does it happen w/ the .NET 8 beta?

  19. hoyosjs commented on Mar 10, 2023

    @hoyosjs
    hoyosjs
    on Mar 10, 2023
    Member

    I don't think anyone is actively working on this. I am not sure anything has changed here.

  20. Limb commented on Mar 11, 2023

    @Limb
    Limb
    on Mar 11, 2023 via email
    Author

  21. Limb commented on Mar 13, 2023

    @Limb
    Limb
    on Mar 13, 2023
    Author

    Some good and bad news to report. The issue still happens with SDK 8.0.100-preview.1.

    However, running commands with DOTNET_ReadyToRun=0 as recommended by @hoyosjs results in the dotnet command running properly. This is true for both .NET 7 and 8.

    I was able to run dotnet --version dotnet new console and dotnet run without issues.

  22. JustSuperHuman commented on Mar 13, 2023

    @JustSuperHuman
    JustSuperHuman
    on Mar 13, 2023

    Ty for testing @Limb - Weird enough I'm testing now w/ 7.0.200 and it seems to be working without adding the ReadyToRun flag 🤔

    image

  23. JustSuperHuman commented on Mar 13, 2023

    @JustSuperHuman
    JustSuperHuman
    on Mar 13, 2023

    Ran into new issues sadly, Am able to run projects normally by launching w/ dotnet program.dll (.NET 7 and 8) but get a segmentation fault when running from a service. systemctl start program (works fine w/ .NET 6)

    Just putting it out there in case anyone finds a way around it.

    Tried putting
    DOTNET_ReadyToRun=0 in /etc/environment but no go.

  24. mangod9
    removed
    untriagedNew issue has not been triaged by the area owner
    on Jul 24, 2023
  25. mangod9
    added this to the Future milestone on Jul 24, 2023

  26. outlookhazy commented on Jul 26, 2023

    @outlookhazy
    outlookhazy
    on Jul 26, 2023

    I'm not positive this is related, but I've been running into an issue with a .net7 published application running on BBB with behavior that sounds extremely similar (inconsistent segfaults). For what it's worth, the application appears to run fine if I turn off address randomization for the process (setarch -R ./program).
    I recompiled under .net6 with the only change being some datetime microseconds references (that aren't supported) and the application runs correctly every time. Unfortunately that native high-precision timing support is necessary for my application.

    A couple more datapoints:

    • Running with strace (but without setarch -R), it does not seem to segfault at all.
    • I have observed instances where the application does partially run, segfaulting after the first 1-2 log entries, and on a couple rare occasions it runs fine indefinitely.

    I can't prove it, but it almost seems like it's more likely to reach execution of application code as the system gets loaded down. This smells like some kind of use-after-free race condition with maybe garbage collection(?)

  27. bklop commented on Jan 10, 2024

    @bklop
    bklop
    on Jan 10, 2024
    Contributor

    Any update to this? Will this be fixed in a future release?

    We ran into this after upgrading our application from .net 6 to 8, took a while to figure out that the problem was not in our code. At least the workaround mentioned by @outlookhazy seems to work for now (disabling ASLR)...

  28. Blackclaws commented on Sep 30, 2024

    @Blackclaws
    Blackclaws
    on Sep 30, 2024

    So we're still seeing this issue on the current net 8.0 the workaround with DOTNET_ReadyToRun=0 seems to work but this should get fixed somehow.

  29. michaldobrodenka commented on Oct 1, 2024

    @michaldobrodenka
    michaldobrodenka
    on Oct 1, 2024

    This issue fit into pattern mentioned here #102396 (comment)

    Still a small sample, but maybe there is a problem with older ARMv7 cores. Beaglebone black has ARM A8. Maybe it's connected to VFPv3 vs VFPv4

  30. agocke
    added this to AppModelon Sep 23, 2025
  31. jkotas
    unassigned
    marcpopMSFT
    on Feb 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-VM-coreclr

    Type

    No type

    Projects

    • AppModel
      Status
      No status

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions