Stackoverflow on macOS x64 hangs

[jkotas]

Stackoverflow on macOS x64 can cause infinite memory allocation loop. See #97049 for repro and other context.

[janvorli]

The infinite loop is caused by an insufficient stack size allocated for the stack overflow handling. We allocate 32kB out of which 4kB is a guard page (on x64). The frames on the stack contain large data structures and that end up resulting in stack overflow again in unw_step when the code is unwinding to the first managed frame. Maybe some recent update in the apple libunwind has caused that it tripped the line. There also seems to be a bug in the ___chkstk_darwin that probes too deep (effectively twice the requested amount - it first correctly probes page by page the requested amount, but at the end it probes the requested size below the lowest address is has probed before). That can prevent us from detecting it as a stack overflow, as the probe then goes below the guard page we create.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.