NativeAOT linux-x64: GC return-address hijack corrupts the caller's frame when a thread is suspended in the epilogue of a self-recursive fast tail call

Description

AI-assisted: the investigation, the repro and this write-up were done with Claude (Anthropic) at my direction. The repro runs and the core dumps analysed below are from my machine.

On linux-x64 Native AOT, a thread that a GC suspends while it runs the epilogue of a self-recursive fast tail
call (add rsp, …; pop …; jmp <own entry>) gets its return-address hijack written to the wrong stack slot.
The slot is the saved RBP of the caller, or another slot of the caller's frame. When the callee returns, the caller
keeps running with RBP = RhpGcProbeHijack (or with an overwritten local) and reads or writes its locals through that
pointer. The results are garbage object references (NullReferenceException far away from the cause), a
SIGSEGV/"Access Violation" FailFast, or silent memory corruption.

Root cause (details under "Other information"): UnixNativeCodeManager::TrailingEpilogueInstructionsCount recognises
a jmp as the end of an epilogue only when its target is outside [start_ip, end_ip) of the current method. A jump
to the method's own start_ip is excluded, although the comment directly above the condition says "A branch to the
start of self implies a recursive call, so is treated as an epilogue". With the epilogue unrecognised, the IP is
reported as unwindable, the body CFI (CFA = rsp+48) is applied to an rsp that the epilogue has already raised, and
GetReturnAddressHijackInfo returns a slot above the real return address.

A common BCL method hits it: ImmutableSortedDictionary<TKey, TValue>.Node.Search (used by ContainsKey,
TryGetValue, the indexer, …). ILC compiles its shared canonical instantiation with two tail calls to its own
entry. Any thread that does lookups while other threads allocate can be affected. CoreCLR (JIT) does not reproduce.
Native AOT compiled without optimization (Debug) does not reproduce either.

Reproduction Steps

Self-contained console project, no package references. Program.cs starts 4 threads that only allocate short-lived
arrays and share no state with the main thread. The main thread fills an
ImmutableSortedDictionary<string, object>.Builder with 356,000 keys and calls ContainsKey before each Add,
15 times over. Optional arguments: iterations count allocatingThreads.

dotnet publish -c Release -r linux-x64 -o out
for i in 1 2 3 4 5; do ./out/TailCallHijackRepro; echo "exit $?"; done

TailCallHijackRepro.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net11.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <PublishAot>true</PublishAot>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>

  <!-- Workaround check: dotnet publish -c Release -r linux-x64 -p:NoFastTailCalls=true -->
  <ItemGroup Condition="'$(NoFastTailCalls)' == 'true'">
    <IlcArg Include="--codegenopt:FastTailCalls=0" />
  </ItemGroup>

</Project>

Program.cs

// SPDX-License-Identifier: MIT
// Native AOT (linux-x64): a GC suspension inside the epilogue of a self-recursive fast tail call
// (ImmutableSortedDictionary<,>.Node.Search) hijacks the wrong stack slot and corrupts the caller's frame.
using System.Collections.Immutable;
using System.Globalization;

int iterations = args.Length > 0 ? int.Parse(args[0], CultureInfo.InvariantCulture) : 15;
int count = args.Length > 1 ? int.Parse(args[1], CultureInfo.InvariantCulture) : 356_000;
int allocatingThreads = args.Length > 2 ? int.Parse(args[2], CultureInfo.InvariantCulture) : 4;

// Other threads only allocate, so GCs start while the main thread runs Search (no shared state).
bool stop = false;
var threads = new List<Thread>();
for (int n = 0; n < allocatingThreads; n++)
{
    var thread = new Thread(() =>
    {
        var keep = new List<object>();
        long k = 0;
        while (!Volatile.Read(ref stop))
        {
            keep.Add(new byte[(int)(k++ % 200) + 16]);
            if (keep.Count > 50_000)
            {
                keep.Clear();
            }
        }
    }) { IsBackground = true };
    thread.Start();
    threads.Add(thread);
}

string[] keys = new string[count];
for (int j = 0; j < count; j++)
{
    keys[j] = "k" + j.ToString("D6", CultureInfo.InvariantCulture);
}

for (int i = 0; i < iterations; i++)
{
    Fill(keys);
}

Volatile.Write(ref stop, true);
foreach (Thread thread in threads)
{
    thread.Join();
}

Console.WriteLine("completed without corruption");
return 0;

// ContainsKey -> Node.Search (compiled as "pop ...; jmp Search"); the caller uses RBP afterwards (out arguments of Add).
static ImmutableSortedDictionary<string, object> Fill(string[] keys)
{
    var map = ImmutableSortedDictionary.CreateBuilder<string, object>(StringComparer.Ordinal);
    foreach (string key in keys)
    {
        if (map.ContainsKey(key))
        {
            throw new InvalidOperationException("duplicate key");
        }

        map.Add(key, new object());
    }

    return map.ToImmutable();
}

Expected behavior

Every run prints completed without corruption and exits with 0.

Actual behavior

A large share of runs crash, at a varying point. In one measurement 2 of 5 runs ended with exit code 134 (SIGABRT after
FailFast); an independent re-run on the same machine gave the same ratio, and a further run produced three crashes within
16 seconds (core dumps analysed below):

Process terminated. Access Violation: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. The application will be terminated since this platform does not support throwing an AccessViolationException.
   at System.RuntimeExceptionHelpers.FailFast(String, Exception, String, RhFailFastReason, IntPtr, IntPtr)
   at System.RuntimeExceptionHelpers.GetRuntimeException(ExceptionIDs)
   at System.Runtime.EH.GetClasslibException(ExceptionIDs, IntPtr)
   at System.Collections.Immutable.ImmutableSortedDictionary`2.Node.SetOrAdd(TKey, TValue, IComparer`1, IEqualityComparer`1, Boolean, Boolean&, Boolean&)
   at Program.<<Main>$>g__Fill|0_0(String[])
   at Program.<Main>$(String[] args)

The symptom we first saw in a real application (Native AOT test executable) was a NullReferenceException in the
caller of ContainsKey, which continued with a corrupted RBP and read garbage references; in another run a corrupted
frame yielded an object field holding 0x7f0f6660244c7f0f, i.e. x86 machine code bytes read through the bogus RBP.

Variants tried with the same setup:

Variant	Result
`ContainsKey` + `Add` interleaved (as in the repro)	crashes intermittently (2 of 5 runs above)
same, calling method marked `MethodImplOptions.NoOptimization`	crashes
only `Add`, or all `ContainsKey` calls after all `Add` calls	no crash seen (the caller does not use RBP after the call, so the corruption stays invisible)
no allocating threads (single thread)	no crash in 80 iterations
JIT (`dotnet run`, any tiering setting)	no crash
Native AOT, Debug configuration	no crash
Native AOT, Release, `--codegenopt:FastTailCalls=0`	no crash (see "Known Workarounds")

It also reproduces with DOTNET_gcServer=1, DOTNET_gcConcurrent=0 and a larger DOTNET_GCgen0size.

Regression?

Not tested on earlier releases. The condition in TrailingEpilogueInstructionsCount is identical in release/10.0,
release/11.0 and main, so the runtime side is probably not new; whether ILC emitted the same self-recursive fast
tail call for Node.Search in .NET 10 was not checked.

Known Workarounds

Disable fast tail calls in ILC's code generator:

<ItemGroup Condition="'$(PublishAot)' == 'true'">
  <IlcArg Include="--codegenopt:FastTailCalls=0" />
</ItemGroup>

With it, Node.Search uses ordinary recursive calls (no jumps to its own entry remain in the binary), and the
repro and our test suites run without failures. The cost is more stack use in deep tail recursion (each recursive
tail call becomes a real call with its own frame).

Configuration

SDK 11.0.100-rc.2.26504.105, runtime / ILCompiler 11.0.0-rc.2.26504.105
PublishAot=true, Release, linux-x64, default GC settings (also reproduces with server GC, non-concurrent GC and
a larger gen0 size)
Linux x64 (CachyOS, kernel 7.2), 16 logical CPUs
Specific to Native AOT on Unix x64 as far as we can tell: the faulty check is in
src/coreclr/nativeaot/Runtime/unix/UnixNativeCodeManager.cpp; CoreCLR does not reproduce. Other Unix
architectures were not tested (the same condition is shared, the epilogue decoding is per architecture).

Other information

Analysis

Code shape. Node<__Canon,__Canon>.Search keeps this as the generic context (mov %rdi,-0x20(%rbp)), so the
recursive call is not turned into a loop. It is emitted as a fast tail call that restores the frame and jumps to
the method's own first instruction (d1cce: add $0x8,%rsp; pop %rbx; pop %r14; pop %r15; pop %rbp; jmp d1c80).
Full disassembly below.
Unwind info. The DWARF CFI describes only the prologue. From the end of the prologue to the end of the method,
CFA = rsp+48, including the two tail-call epilogues. Inside such an epilogue the real CFA is rsp+48-8k, where k
is the number of 8-byte slots already released (the add $0x8,%rsp counts as one).
Epilogue detection. UnixNativeCodeManager::TrailingEpilogueInstructionsCount is supposed to stop the runtime
from unwinding or hijacking at an IP inside an epilogue. It accepts a jmp rel8/rel32 as the end of an epilogue
only when the target lies outside [start_ip, end_ip) of the current method. For a self-recursive tail call
the target is exactly start_ip, so it counts as a jump inside the method, the epilogue is not recognised, and the
IP is reported as unwindable.

The source states the intended rule in its comments: "An unconditional branch to a target that is equal to the
start of or outside of this routine is logically a call to another function" and "A branch to the start of self
implies a recursive call, so is treated as an epilogue". The condition below them,
if (branchTarget < procInfo.start_ip || branchTarget >= procInfo.end_ip), excludes start_ip itself, so the
code does not match its comment (unchanged on main, release/11.0 and release/10.0).
Effect. For example, the thread is interrupted at d1cd2 (pop %rbx): the real rsp = entry_rsp - 0x20, but
unwinding with the body CFI gives CFA = entry_rsp + 0x10. GetReturnAddressHijackInfo then uses
[entry_rsp + 8] as the "return address" slot and writes RhpGcProbeHijack there (entry_rsp = RSP at the
method's first instruction, pointing at the real return address). That slot is the saved RBP pushed by the caller
ContainsKey (push %rbp; mov %rsp,%rbp; call Search). Search returns normally, ContainsKey pops
RBP = RhpGcProbeHijack and returns to its caller, which addresses its locals and out arguments relative to that
value. Interruptions at the other epilogue instructions overwrite slots further up in the caller's frame.
Core dumps (three crashed runs, all identical): the fault is SetOrAdd+50, movb $0x0,(%r8) (the
out bool replacedExistingValue store), with r8 loaded from the first stack argument. Both stack arguments are
RhpGcProbeHijack - 0x30 and RhpGcProbeHijack - 0x28, exactly what the caller Fill computes as
lea -0x30(%rbp) / lea -0x28(%rbp) right before call SetOrAdd, so Fill's RBP was RhpGcProbeHijack. The slot
in which SetOrAdd pushed that corrupted RBP holds Fill's correct RBP again (intact frame chain up to Main):
it is the same slot ContainsKey had pushed into, i.e. [entry_rsp + 8] of Search, and the only writer of the
correct value after SetOrAdd's push is Thread::Unhijack (called by the stack-frame iterator during exception
dispatch), which restores the recorded original value at m_ppvHijackedReturnAddressLocation. That confirms the
hijack location.

Suggested fixes (any one of them)

In TrailingEpilogueInstructionsCount (and the corresponding non-amd64 checks, if they have the same rule), treat a
direct jump to the start of the current method as a tail call, i.e. an epilogue, as its comment already says:
branchTarget <= procInfo.start_ip || branchTarget >= procInfo.end_ip. The Windows x64 unwinder's epilogue
recognition may need the same rule for consistency.
Have the JIT/ILC emit CFI for mid-method epilogues (remember_state/restore_state around each epilogue), so
unwinding inside them is correct.
Avoid the shape in the JIT: turn the recursive tail call into a loop even when the generic context is this, or do
not use a fast tail call to the own entry in methods that keep the generic context alive.

Related issues

Assertion 'codeManager->IsUnwindable(pvAddress) || runtime->IsConservativeStackReportingEnabled()' in System.Runtime.Tests on ARM64 #134912 (closed by Arm64:[PAC-RET]Fix ARM64 epilog detection and PAC signing SP recovery #135088): the same function on ARM64, where a stackalloc SP adjustment was misclassified as an
epilogue after the PAC-RET change. Different defect; the fix did not touch the x64 start_ip condition.
[NativeAOT] Consider encoding epilogs/prologs in GC or Debug info. #101932: encoding prologues/epilogues in GC info would remove the instruction-pattern matching this bug lives in.
ExecutionEngineException fail-fast in GC thread-suspension / return-address hijack while root-scanning a Win32 message-pump P/Invoke frame (.NET 10.0.8, Windows, Avalonia desktop) #129071 (comment of 2026-10-14): the same failure class on CoreCLR/Windows x64, a hijack written into the caller's
frame because the unwinder does not recognise an epilogue ending in jmp qword ptr [abs32], same workaround.
Different trigger and code manager, same mechanism.

Evidence

Node.Search disassembly (objdump, repro binary)

# Node<__Canon,__Canon>.Search as compiled by ILC (repro/, Release, linux-x64, no workaround)
# objdump -d --no-show-raw-insn; symbol prefix System_Collections_Immutable_System_Collections_Immutable_ImmutableSortedDictionary_2_Node<System___Canon__System___Canon>__ shortened to Node::
# The two tail-call epilogues (add rsp / pop x4 / jmp Node::Search) jump to the method's own entry.

00000000000d1c80 <Node::Search>:
   d1c80:	push   %rbp
   d1c81:	push   %r15
   d1c83:	push   %r14
   d1c85:	push   %rbx
   d1c86:	push   %rax
   d1c87:	lea    0x20(%rsp),%rbp
   d1c8c:	mov    %rdi,-0x20(%rbp)
   d1c90:	mov    %rdi,%rbx
   d1c93:	mov    %rsi,%r15
   d1c96:	mov    %rdx,%r14
   d1c99:	cmpq   $0x0,0x18(%rbx)
   d1c9e:	je     d1cf8 <Node::Search+0x78>
   d1ca0:	mov    (%rbx),%rdx
   d1ca3:	mov    0x30(%rdx),%rdx
   d1ca7:	mov    0x30(%rdx),%r11
   d1cab:	mov    0x8(%rbx),%rdx
   d1caf:	mov    %r14,%rdi
   d1cb2:	mov    %r15,%rsi
   d1cb5:	call   1c750 <RhpInterfaceDispatch>
   d1cba:	test   %eax,%eax
   d1cbc:	je     d1cf8 <Node::Search+0x78>
   d1cbe:	test   %eax,%eax
   d1cc0:	jg     d1cdd <Node::Search+0x5d>
   d1cc2:	mov    0x18(%rbx),%rdi
   d1cc6:	mov    %r15,%rsi
   d1cc9:	mov    %r14,%rdx
   d1ccc:	cmp    %edi,(%rdi)
   d1cce:	add    $0x8,%rsp
   d1cd2:	pop    %rbx
   d1cd3:	pop    %r14
   d1cd5:	pop    %r15
   d1cd7:	pop    %rbp
   d1cd8:	jmp    d1c80 <Node::Search>
   d1cdd:	mov    0x20(%rbx),%rdi
   d1ce1:	mov    %r15,%rsi
   d1ce4:	mov    %r14,%rdx
   d1ce7:	cmp    %edi,(%rdi)
   d1ce9:	add    $0x8,%rsp
   d1ced:	pop    %rbx
   d1cee:	pop    %r14
   d1cf0:	pop    %r15
   d1cf2:	pop    %rbp
   d1cf3:	jmp    d1c80 <Node::Search>
   d1cf8:	mov    %rbx,%rax
   d1cfb:	add    $0x8,%rsp
   d1cff:	pop    %rbx
   d1d00:	pop    %r14
   d1d02:	pop    %r15
   d1d04:	pop    %rbp
   d1d05:	ret

Node.Search CFI (readelf --debug-dump=frames-interp)

# readelf --debug-dump=frames-interp, FDE of Node<__Canon,__Canon>.Search (same binary as search-disassembly.txt)
# The CFI describes only the prologue. From d1c87 to the end of the method the CFA is rsp+48, including the
# tail-call epilogues at d1cce..d1cd8 and d1ce9..d1cf3, where rsp is already 8..40 bytes higher.

0001a5dc 0000000000000028 0001a570 FDE cie=00000070 pc=00000000000d1c80..00000000000d1d06
   LOC           CFA      rbx   rbp   r14   r15   ra
00000000000d1c80 rsp+8    u     u     u     u     c-8
00000000000d1c81 rsp+16   u     c-16  u     u     c-8
00000000000d1c83 rsp+24   u     c-16  u     c-24  c-8
00000000000d1c85 rsp+32   u     c-16  c-32  c-24  c-8
00000000000d1c86 rsp+40   c-40  c-16  c-32  c-24  c-8
00000000000d1c87 rsp+48   c-40  c-16  c-32  c-24  c-8

# Example: interrupted at d1cd2 (pop %rbx, after add $0x8,%rsp): actual rsp = entry_rsp - 0x20.
# CFA by CFI = rsp + 48 = entry_rsp + 0x10, so "return address" = [CFA - 8] = [entry_rsp + 8], one slot above the
# real return address. For the caller ContainsKey (push %rbp; mov %rsp,%rbp; call Search) that slot is its saved RBP.

TrailingEpilogueInstructionsCount, the jmp branch as linked into the binary (objdump)

# UnixNativeCodeManager::TrailingEpilogueInstructionsCount (runtime 11.0.0-rc.2.26504.105, linked into the AOT binary)
# objdump excerpt of the "jmp rel8/rel32" branch. r14 = jump target; GetUnwindProcInfo fills unw_proc_info_t at
# (%rbx) = start_ip, -0x60(%rbp) = end_ip of the method containing the interrupted IP.

   call   <UnwindHelpers::GetUnwindProcInfo(...)>
   cmp    (%rbx),%r14          ; target <  start_ip ?
   jb     <+0x176>             ;   yes -> return instruction count (epilogue)
   cmp    -0x60(%rbp),%r14     ; target <  end_ip ?
   mov    %r15d,%eax
   jb     <+0x15a>             ;   yes -> return 0 (treated as a jump inside the method, NOT an epilogue)
   jmp    <+0x15c>             ;   no  -> return instruction count (epilogue)

# Source: src/coreclr/nativeaot/Runtime/unix/UnixNativeCodeManager.cpp,
#   if (branchTarget < procInfo.start_ip || branchTarget >= procInfo.end_ip) return trailingEpilogueInstructions;
# (its comment says a branch to the start of self is to be treated as an epilogue).
# For a self-recursive fast tail call the target is exactly start_ip, so start_ip <= target < end_ip and the
# epilogue is not recognised. The IP is then considered unwindable with the body CFI (search-cfi.txt).

Core-dump analysis (gdb, three dumps, with the script used)

# Core dumps of three crashed repro runs (2026-10-16, repro/ published without the workaround, same binary as
# repro/out/TailCallHijackRepro; SDK/runtime 11.0.0-rc.2.26504.105, linux-x64, ASLR on, so addresses differ per run).
# Dumps collected by systemd-coredump (coredumpctl PIDs 322433, 322722, 324376; SIGABRT from FailFast after the
# access violation). Symbol prefix of ImmutableSortedDictionary`2.Node<__Canon,__Canon> shortened to Node::,
# TailCallHijackRepro_Program___ to Program::.
#
# How the fault context was recovered: the crashing thread is in abort() below FailFast, so gdb's backtrace ends at
# RhpThrowHwEx. The signal handler enters RhpThrowHwEx with RSI = faulting RIP and RAX = faulting RSP, and its
# prologue pushes r15, r14, r13, r12, 0, rbx, 0, rbp, rax, rsi and then subtracts 0x190 (PAL_LIMITED_CONTEXT, see
# the disassembly below). The context therefore starts at RSP+0x190 of the RhpThrowHwEx frame:
#   [+0x00] faulting RIP, [+0x08] faulting RSP, [+0x10] faulting RBP, [+0x20] RBX, [+0x30..+0x48] R12..R15.
#
# Findings, identical in all three dumps:
# 1. Faulting instruction: Node::SetOrAdd+50 "movb $0x0,(%r8)" (= replacedExistingValue = false), with
#    r8 = [rbp+0x10] = the first stack argument, i.e. the out-argument pointer passed by the caller.
# 2. SetOrAdd's frame (rbp = faulting RSP + 0x60): [rbp+0x10] and [rbp+0x18] (the two stack arguments) are
#    RhpGcProbeHijack - 0x30 and RhpGcProbeHijack - 0x28 in every dump. They point into read-only code, hence SIGSEGV.
# 3. The caller Program::Fill computes exactly these arguments as "lea -0x30(%rbp)" and "lea -0x28(%rbp)" directly
#    before "call Node::SetOrAdd", so Fill's RBP register was RhpGcProbeHijack at that moment.
# 4. Fill's previous call is Node::ContainsKey = "push %rbp; mov %rsp,%rbp; call Node::Search; ...; pop %rbp; ret".
#    Its saved-RBP slot is [Search entry RSP + 8], the slot the analysis (step 4 above) names as the hijack target.
# 5. Node::Search in this binary has two epilogues ending in "pop %rbp; jmp Node::Search" (jump to its own entry).
# 6. [rbp] of SetOrAdd's frame (the slot where SetOrAdd pushed Fill's RBP) holds Fill's *correct* RBP (a stack
#    address with an intact frame chain to Program::Main), not RhpGcProbeHijack. This slot is the same stack slot
#    that ContainsKey's "push %rbp" used (both calls are made with the same RSP in Fill), i.e. the hijacked slot.
#    Thread::Unhijack(), called by the stack-frame iterator during exception dispatch
#    (StackFrameIterator.cpp, "we need to unhijack every time we callback into C++"), writes the recorded original
#    value back to m_ppvHijackedReturnAddressLocation. Fill's correct RBP reappearing in exactly this slot after
#    SetOrAdd had overwritten it with the corrupted value is only explained by that restore, which confirms that the
#    hijack location was this slot. (Inferred from dump + runtime source; the hijack fields themselves are cleared
#    by Unhijack and carry no information in the dump.)
# 7. The four allocating threads sit in gc_heap::allocate_soh -> GCToOSInterface::Sleep, the background GC thread in
#    gc_heap::background_sweep, i.e. the GC was active around the crash.
#
# Commands: coredumpctl dump <pid> -o core.<pid>; gdb -batch -x ctx.gdb <exe> core.<pid> with the script below.

# ---- ctx.gdb ----
set pagination off
set debuginfod enabled off
frame function RhpThrowHwEx
set $ctx = $rsp + 0x190
set $fip = *(long*)($ctx)
set $frsp = *(long*)($ctx+8)
set $frbp = *(long*)($ctx+0x10)
set $hij = (long)&RhpGcProbeHijack
printf "RhpGcProbeHijack = 0x%lx\n", $hij
printf "fault IP  = 0x%lx\n", $fip
info symbol $fip
printf "fault RSP = 0x%lx\n", $frsp
printf "fault RBP = 0x%lx\n", $frbp
info symbol $frbp
printf "context (ip, rsp, rbp, 0, rbx, 0, r12, r13, r14, r15):\n"
x/10gx $ctx
printf "stack at fault RSP:\n"
x/24gx $frsp
printf "search RhpGcProbeHijack on the fault stack:\n"
find /g $frsp, +0x400, $hij

# ======================================================================
# core.322433
# ======================================================================
RhpGcProbeHijack = 0x556fb73f5c20
fault IP  = 0x556fb74aa982
Node::SetOrAdd + 50
fault RSP = 0x7ffc91264ca0
fault RBP = 0x7ffc91264d00
No symbol matches $frbp.
context (ip, rsp, rbp, 0, rbx, 0, r12, r13, r14, r15):
0x7ffc91264c50:	0x0000556fb74aa982	0x00007ffc91264ca0
0x7ffc91264c60:	0x00007ffc91264d00	0x0000000000000000
0x7ffc91264c70:	0x00007fbcf8c56308	0x0000000000000000
0x7ffc91264c80:	0x0000556fb756e950	0x0000556fb756ebf0
0x7ffc91264c90:	0x00007fbcfdd5afd0	0x00007fbcf1b742b8
stack at fault RSP:
0x7ffc91264ca0:	0x00007ffc91264d20	0x00007ffc91264d28
0x7ffc91264cb0:	0x0000000000000000	0x0000556fb74aa740
0x7ffc91264cc0:	0x00007ffc91264cf0	0x00000000b74aacba
0x7ffc91264cd0:	0x00007fbcf8c56308	0x00007fbcf1e21328
0x7ffc91264ce0:	0x0000000000056ea0	0x00007fbcf1b742b8
0x7ffc91264cf0:	0x00007fbcf46776e0	0x0000000000012c46
0x7ffc91264d00:	0x00007ffc91264d50	0x0000556fb7493f8e
0x7ffc91264d10:	0x0000556fb73f5bf0	0x0000556fb73f5bf8
0x7ffc91264d20:	0x0000000000000000	0x0000000000000001
0x7ffc91264d30:	0x00007fbcf1c00048	0x000000000000000f
0x7ffc91264d40:	0x00007fbcf1404cf8	0x00007fbcf1404cd8
0x7ffc91264d50:	0x00007ffc91264dd0	0x0000556fb7493de8
search RhpGcProbeHijack on the fault stack:
Pattern not found.

# ======================================================================
# core.322722
# ======================================================================
RhpGcProbeHijack = 0x55ec683b7c20
fault IP  = 0x55ec6846c982
Node::SetOrAdd + 50
fault RSP = 0x7fffda837fe0
fault RBP = 0x7fffda838040
No symbol matches $frbp.
context (ip, rsp, rbp, 0, rbx, 0, r12, r13, r14, r15):
0x7fffda837f90:	0x000055ec6846c982	0x00007fffda837fe0
0x7fffda837fa0:	0x00007fffda838040	0x0000000000000000
0x7fffda837fb0:	0x00007f5f4acf44e0	0x0000000000000000
0x7fffda837fc0:	0x000055ec68530950	0x000055ec68530bf0
0x7fffda837fd0:	0x00007f5f4d199950	0x00007f5f412e4b28
stack at fault RSP:
0x7fffda837fe0:	0x00007fffda838060	0x00007fffda838068
0x7fffda837ff0:	0x0000000000000000	0x000055ec6846c740
0x7fffda838000:	0x00007fffda838030	0x000000006846ccba
0x7fffda838010:	0x00007f5f4acf44e0	0x00007f5f41cf5a48
0x7fffda838020:	0x0000000000056ea0	0x00007f5f412e4b28
0x7fffda838030:	0x00007f5f47e846b0	0x0000000000038362
0x7fffda838040:	0x00007fffda838090	0x000055ec68455f8e
0x7fffda838050:	0x000055ec683b7bf0	0x000055ec683b7bf8
0x7fffda838060:	0x0000000000000000	0x0000000000000001
0x7fffda838070:	0x00007f5f41c00048	0x0000000000000009
0x7fffda838080:	0x00007f5f41404c98	0x00007f5f41404c78
0x7fffda838090:	0x00007fffda838110	0x000055ec68455de8
search RhpGcProbeHijack on the fault stack:
Pattern not found.

# ======================================================================
# core.324376
# ======================================================================
RhpGcProbeHijack = 0x558d5f9a3c20
fault IP  = 0x558d5fa58982
Node::SetOrAdd + 50
fault RSP = 0x7fff125e4fb0
fault RBP = 0x7fff125e5010
No symbol matches $frbp.
context (ip, rsp, rbp, 0, rbx, 0, r12, r13, r14, r15):
0x7fff125e4f60:	0x0000558d5fa58982	0x00007fff125e4fb0
0x7fff125e4f70:	0x00007fff125e5010	0x0000000000000000
0x7fff125e4f80:	0x00007f680a2591e8	0x0000000000000000
0x7fff125e4f90:	0x0000558d5fb1c950	0x0000558d5fb1cbf0
0x7fff125e4fa0:	0x00007f68105cdc40	0x00007f6805317f80
stack at fault RSP:
0x7fff125e4fb0:	0x00007fff125e5030	0x00007fff125e5038
0x7fff125e4fc0:	0x0000000000000000	0x0000558d5fa58740
0x7fff125e4fd0:	0x00007fff125e5000	0x000000005fa58cba
0x7fff125e4fe0:	0x00007f680a2591e8	0x00007f6805db3b28
0x7fff125e4ff0:	0x0000000000056ea0	0x00007f6805317f80
0x7fff125e5000:	0x00007f680f254618	0x0000000000020746
0x7fff125e5010:	0x00007fff125e5060	0x0000558d5fa41f8e
0x7fff125e5020:	0x0000558d5f9a3bf0	0x0000558d5f9a3bf8
0x7fff125e5030:	0x0000000000000000	0x0000000000000001
0x7fff125e5040:	0x00007f6805c00048	0x000000000000000e
0x7fff125e5050:	0x00007f6805404cf8	0x00007f6805404cd8
0x7fff125e5060:	0x00007fff125e50e0	0x0000558d5fa41de8
search RhpGcProbeHijack on the fault stack:
Pattern not found.

# ======================================================================
# Code (same binary in all three runs; addresses from core.322433)
# ======================================================================
=== RhpThrowHwEx prologue (layout of the saved context) ===
Dump of assembler code for function RhpThrowHwEx:
   0x0000556fb73f58d0 <+0>:	mov    %rsp,%rax
   0x0000556fb73f58d3 <+3>:	and    $0xfffffffffffffff0,%rsp
   0x0000556fb73f58d7 <+7>:	xor    %rdx,%rdx
   0x0000556fb73f58da <+10>:	push   %r15
   0x0000556fb73f58dc <+12>:	push   %r14
   0x0000556fb73f58de <+14>:	push   %r13
   0x0000556fb73f58e0 <+16>:	push   %r12
   0x0000556fb73f58e2 <+18>:	push   %rdx
   0x0000556fb73f58e3 <+19>:	push   %rbx
   0x0000556fb73f58e4 <+20>:	push   %rdx
   0x0000556fb73f58e5 <+21>:	push   %rbp
   0x0000556fb73f58e6 <+22>:	push   %rax
   0x0000556fb73f58e7 <+23>:	push   %rsi
   0x0000556fb73f58e8 <+24>:	lea    -0x190(%rsp),%rsp
   0x0000556fb73f58f0 <+32>:	mov    %rdi,%rbx
   0x0000556fb73f58f3 <+35>:	mov    %fs:0x0,%rax
   0x0000556fb73f58fc <+44>:	lea    -0xd0(%rax),%rax
   0x0000556fb73f5903 <+51>:	mov    %rbx,%rdi
   0x0000556fb73f5906 <+54>:	mov    %rsp,%rsi
   0x0000556fb73f5909 <+57>:	xor    %rdx,%rdx
   0x0000556fb73f590c <+60>:	mov    %rdx,0x10(%rsi)
   0x0000556fb73f5910 <+64>:	movb   $0x1,0x19(%rsi)
   0x0000556fb73f5914 <+68>:	movl   $0xffffffff,0x1c(%rsi)
   0x0000556fb73f591b <+75>:	movb   $0x2,0x18(%rsi)
   0x0000556fb73f591f <+79>:	mov    0x78(%rax),%rdx
   0x0000556fb73f5923 <+83>:	mov    %rdx,(%rsi)
   0x0000556fb73f5926 <+86>:	mov    %rsi,0x78(%rax)
   0x0000556fb73f592a <+90>:	lea    0x190(%rsp),%rdx
   0x0000556fb73f5932 <+98>:	mov    %rdx,0x8(%rsi)
   0x0000556fb73f5936 <+102>:	call   0x556fb7469b00 <S_P_CoreLib_System_Runtime_EH__RhThrowHwEx>
   0x0000556fb73f593b <+107>:	int3
End of assembler dump.
=== Node::SetOrAdd prologue up to the faulting instruction (+50) ===
   0x556fb74aa950 <Node::SetOrAdd>:	push   %rbp
   0x556fb74aa951 <Node::SetOrAdd+1>:	push   %r15
   0x556fb74aa953 <Node::SetOrAdd+3>:	push   %r14
   0x556fb74aa955 <Node::SetOrAdd+5>:	push   %r13
   0x556fb74aa957 <Node::SetOrAdd+7>:	push   %r12
   0x556fb74aa959 <Node::SetOrAdd+9>:	push   %rbx
   0x556fb74aa95a <Node::SetOrAdd+10>:	sub    $0x38,%rsp
   0x556fb74aa95e <Node::SetOrAdd+14>:	lea    0x60(%rsp),%rbp
   0x556fb74aa963 <Node::SetOrAdd+19>:	mov    %rdi,-0x30(%rbp)
   0x556fb74aa967 <Node::SetOrAdd+23>:	mov    %r9d,-0x34(%rbp)
   0x556fb74aa96b <Node::SetOrAdd+27>:	mov    %rdi,%rbx
   0x556fb74aa96e <Node::SetOrAdd+30>:	mov    %rsi,%r15
   0x556fb74aa971 <Node::SetOrAdd+33>:	mov    %rdx,%r14
   0x556fb74aa974 <Node::SetOrAdd+36>:	mov    %rcx,%r13
   0x556fb74aa977 <Node::SetOrAdd+39>:	mov    %r8,%r12
   0x556fb74aa97a <Node::SetOrAdd+42>:	mov    0x10(%rbp),%r8
   0x556fb74aa97e <Node::SetOrAdd+46>:	mov    %r8,0x10(%rbp)
   0x556fb74aa982 <Node::SetOrAdd+50>:	movb   $0x0,(%r8)
   0x556fb74aa986 <Node::SetOrAdd+54>:	cmpq   $0x0,0x18(%rbx)
   0x556fb74aa98b <Node::SetOrAdd+59>:	je     0x556fb74aab25 <Node::SetOrAdd+469>
=== Program::Fill around the two calls (ContainsKey, SetOrAdd); return address of the fault is Fill+238 ===
   0x556fb7493f2c <Program::_Main___g__Fill_0_0+140>:	call   0x556fb74aa4b0 <Node::ContainsKey>
   0x556fb7493f31 <Program::_Main___g__Fill_0_0+145>:	test   %eax,%eax
   0x556fb7493f33 <Program::_Main___g__Fill_0_0+147>:	jne    0x556fb7493fe1 <Program::_Main___g__Fill_0_0+321>
   0x556fb7493f39 <Program::_Main___g__Fill_0_0+153>:	lea    0xdc0e8(%rip),%rdi        # 0x556fb7570028 <_ZTV6Object>
   0x556fb7493f40 <Program::_Main___g__Fill_0_0+160>:	call   0x556fb73f54c0 <RhpNewFast>
   0x556fb7493f45 <Program::_Main___g__Fill_0_0+165>:	mov    %rax,%rdx
   0x556fb7493f48 <Program::_Main___g__Fill_0_0+168>:	mov    0x8(%r14),%rdi
   0x556fb7493f4c <Program::_Main___g__Fill_0_0+172>:	mov    0x10(%r14),%rcx
   0x556fb7493f50 <Program::_Main___g__Fill_0_0+176>:	mov    0x18(%r14),%r8
   0x556fb7493f54 <Program::_Main___g__Fill_0_0+180>:	cmp    %dil,(%rdi)
   0x556fb7493f57 <Program::_Main___g__Fill_0_0+183>:	test   %r13,%r13
   0x556fb7493f5a <Program::_Main___g__Fill_0_0+186>:	je     0x556fb7494008 <Program::_Main___g__Fill_0_0+360>
   0x556fb7493f60 <Program::_Main___g__Fill_0_0+192>:	test   %rcx,%rcx
   0x556fb7493f63 <Program::_Main___g__Fill_0_0+195>:	je     0x556fb7494015 <Program::_Main___g__Fill_0_0+373>
   0x556fb7493f69 <Program::_Main___g__Fill_0_0+201>:	test   %r8,%r8
   0x556fb7493f6c <Program::_Main___g__Fill_0_0+204>:	je     0x556fb7494022 <Program::_Main___g__Fill_0_0+386>
   0x556fb7493f72 <Program::_Main___g__Fill_0_0+210>:	lea    -0x30(%rbp),%rsi
   0x556fb7493f76 <Program::_Main___g__Fill_0_0+214>:	mov    %rsi,(%rsp)
   0x556fb7493f7a <Program::_Main___g__Fill_0_0+218>:	lea    -0x28(%rbp),%rsi
   0x556fb7493f7e <Program::_Main___g__Fill_0_0+222>:	mov    %rsi,0x8(%rsp)
   0x556fb7493f83 <Program::_Main___g__Fill_0_0+227>:	mov    %r13,%rsi
   0x556fb7493f86 <Program::_Main___g__Fill_0_0+230>:	xor    %r9d,%r9d
   0x556fb7493f89 <Program::_Main___g__Fill_0_0+233>:	call   0x556fb74aa950 <Node::SetOrAdd>
   0x556fb7493f8e <Program::_Main___g__Fill_0_0+238>:	incl   0x34(%r14)
   0x556fb7493f92 <Program::_Main___g__Fill_0_0+242>:	cmp    %rax,0x8(%r14)
   0x556fb7493f96 <Program::_Main___g__Fill_0_0+246>:	je     0x556fb7493faa <Program::_Main___g__Fill_0_0+266>
   0x556fb7493f98 <Program::_Main___g__Fill_0_0+248>:	lea    0x8(%r14),%rdi
   0x556fb7493f9c <Program::_Main___g__Fill_0_0+252>:	mov    %rax,%rsi
=== Fill frame chain from SetOrAdd's saved rbp: [fill rbp], [fill rbp+8] ===
0x7ffc91264d50:	0x00007ffc91264dd0	0x0000556fb7493de8
Program::Main__ + 744
=== Node::ContainsKey ===
Dump of assembler code for function Node::ContainsKey:
   0x0000556fb74aa4b0 <+0>:	push   %rbp
   0x0000556fb74aa4b1 <+1>:	mov    %rsp,%rbp
   0x0000556fb74aa4b4 <+4>:	test   %rsi,%rsi
   0x0000556fb74aa4b7 <+7>:	je     0x556fb74aa4d0 <Node::ContainsKey+32>
   0x0000556fb74aa4b9 <+9>:	test   %rdx,%rdx
   0x0000556fb74aa4bc <+12>:	je     0x556fb74aa4dd <Node::ContainsKey+45>
   0x0000556fb74aa4be <+14>:	call   0x556fb74aac80 <Node::Search>
   0x0000556fb74aa4c3 <+19>:	cmpq   $0x0,0x18(%rax)
   0x0000556fb74aa4c8 <+24>:	setne  %al
   0x0000556fb74aa4cb <+27>:	movzbl %al,%eax
   0x0000556fb74aa4ce <+30>:	pop    %rbp
   0x0000556fb74aa4cf <+31>:	ret
   0x0000556fb74aa4d0 <+32>:	lea    0xc2271(%rip),%rdi        # 0x556fb756c748 <__FrozenSegmentStart+59880>
   0x0000556fb74aa4d7 <+39>:	call   0x556fb7438800 <System_Collections_Immutable_System_Collections_Immutable_Requires__FailArgumentNullException>
   0x0000556fb74aa4dc <+44>:	int3
   0x0000556fb74aa4dd <+45>:	lea    0xc2284(%rip),%rdi        # 0x556fb756c768 <__FrozenSegmentStart+59912>
   0x0000556fb74aa4e4 <+52>:	call   0x556fb7438800 <System_Collections_Immutable_System_Collections_Immutable_Requires__FailArgumentNullException>
   0x0000556fb74aa4e9 <+57>:	int3
End of assembler dump.
=== Node::Search: the two self-recursive tail-call epilogues ===
Dump of assembler code from 0x556fb74aac80 to 0x556fb74aacf8:
   0x0000556fb74aacce <Node::Search+78>:	add    $0x8,%rsp
   0x0000556fb74aacd2 <Node::Search+82>:	pop    %rbx
   0x0000556fb74aacd3 <Node::Search+83>:	pop    %r14
   0x0000556fb74aacd5 <Node::Search+85>:	pop    %r15
   0x0000556fb74aacd7 <Node::Search+87>:	pop    %rbp
   0x0000556fb74aacd8 <Node::Search+88>:	jmp    0x556fb74aac80 <Node::Search>
   0x0000556fb74aace9 <Node::Search+105>:	add    $0x8,%rsp
   0x0000556fb74aaced <Node::Search+109>:	pop    %rbx
   0x0000556fb74aacee <Node::Search+110>:	pop    %r14
   0x0000556fb74aacf0 <Node::Search+112>:	pop    %r15
   0x0000556fb74aacf2 <Node::Search+114>:	pop    %rbp
   0x0000556fb74aacf3 <Node::Search+115>:	jmp    0x556fb74aac80 <Node::Search>
End of assembler dump.
=== all threads ===
Thread 7 (Thread 0x7fbcc993b6c0 (LWP 322437)):
#1  0x00007fcc9510228a in clock_nanosleep () from /usr/lib/libc.so.6
#2  0x00007fcc95112bb7 in nanosleep () from /usr/lib/libc.so.6
#3  0x0000556fb73e0850 in GCToOSInterface::Sleep (sleepMSec=<optimized out>) at /__w/1/s/src/runtime/src/coreclr/gc/unix/gcenv.unix.cpp:359
#4  0x0000556fb743000a in WKS::gc_heap::allocate_soh (gen_number=gen_number@entry=0, size=size@entry=224, acontext=acontext@entry=0x7fbcc993b5f8, flags=flags@entry=0, align_const=align_const@entry=7) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:2985
#5  0x0000556fb73e716b in WKS::gc_heap::try_allocate_more_space (acontext=0x7fbcc993b5f8, size=224, flags=0, gen_number=0) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:4014
Thread 6 (Thread 0x7fbcc913a6c0 (LWP 322438)):
#1  0x00007fcc9510228a in clock_nanosleep () from /usr/lib/libc.so.6
#2  0x00007fcc95112bb7 in nanosleep () from /usr/lib/libc.so.6
#3  0x0000556fb73e0850 in GCToOSInterface::Sleep (sleepMSec=<optimized out>) at /__w/1/s/src/runtime/src/coreclr/gc/unix/gcenv.unix.cpp:359
#4  0x0000556fb743000a in WKS::gc_heap::allocate_soh (gen_number=gen_number@entry=0, size=size@entry=176, acontext=acontext@entry=0x7fbcc913a5f8, flags=flags@entry=0, align_const=align_const@entry=7) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:2985
#5  0x0000556fb73e716b in WKS::gc_heap::try_allocate_more_space (acontext=0x7fbcc913a5f8, size=176, flags=0, gen_number=0) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:4014
Thread 5 (Thread 0x7fbcc89396c0 (LWP 322439)):
#1  WKS::my_get_size (ob=0x7fbcfa07b4c0) at /__w/1/s/src/runtime/src/coreclr/gc/gcinternal.h:1179
#2  WKS::gc_heap::background_sweep () at /__w/1/s/src/runtime/src/coreclr/gc/background.cpp:4007
#3  0x0000556fb7412372 in WKS::gc_heap::gc1 () at /__w/1/s/src/runtime/src/coreclr/gc/collect.cpp:162
#4  0x0000556fb7429a2f in WKS::gc_heap::bgc_thread_function () at /__w/1/s/src/runtime/src/coreclr/gc/background.cpp:3167
#5  0x0000556fb73f7d38 in GCToEEInterface::CreateThread(void (*)(void*), void*, bool, char const*)::$_0::operator()(void*) const (argument=<optimized out>, this=<optimized out>) at /__w/1/s/src/runtime/src/coreclr/nativeaot/Runtime/gcenv.ee.cpp:646
Thread 4 (Thread 0x7fcc94fff6c0 (LWP 322434)):
#1  0x00007fcc950a908a in ?? () from /usr/lib/libc.so.6
#2  0x00007fcc950abc70 in pthread_cond_wait () from /usr/lib/libc.so.6
#3  0x0000556fb74050aa in UnixEvent::Wait (this=0x556fe191e1c0, milliseconds=<optimized out>) at /__w/1/s/src/runtime/src/coreclr/nativeaot/Runtime/unix/PalUnix.cpp:352
#4  0x0000556fb73f63ff in FinalizerStart (pContext=0x556fe191e1c0) at /__w/1/s/src/runtime/src/coreclr/nativeaot/Runtime/FinalizerHelpers.cpp:70
#5  0x00007fcc950ac93f in ?? () from /usr/lib/libc.so.6
Thread 3 (Thread 0x7fbcca93d6c0 (LWP 322435)):
#1  0x00007fcc9510228a in clock_nanosleep () from /usr/lib/libc.so.6
#2  0x00007fcc95112bb7 in nanosleep () from /usr/lib/libc.so.6
#3  0x0000556fb73e0850 in GCToOSInterface::Sleep (sleepMSec=<optimized out>) at /__w/1/s/src/runtime/src/coreclr/gc/unix/gcenv.unix.cpp:359
#4  0x0000556fb743000a in WKS::gc_heap::allocate_soh (gen_number=gen_number@entry=0, size=size@entry=144, acontext=acontext@entry=0x7fbcca93d5f8, flags=flags@entry=0, align_const=align_const@entry=7) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:2985
#5  0x0000556fb73e716b in WKS::gc_heap::try_allocate_more_space (acontext=0x7fbcca93d5f8, size=144, flags=0, gen_number=0) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:4014
Thread 2 (Thread 0x7fbcca13c6c0 (LWP 322436)):
#1  0x00007fcc9510228a in clock_nanosleep () from /usr/lib/libc.so.6
#2  0x00007fcc95112bb7 in nanosleep () from /usr/lib/libc.so.6
#3  0x0000556fb73e0850 in GCToOSInterface::Sleep (sleepMSec=<optimized out>) at /__w/1/s/src/runtime/src/coreclr/gc/unix/gcenv.unix.cpp:359
#4  0x0000556fb743000a in WKS::gc_heap::allocate_soh (gen_number=gen_number@entry=0, size=size@entry=160, acontext=acontext@entry=0x7fbcca13c5f8, flags=flags@entry=0, align_const=align_const@entry=7) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:2985
#5  0x0000556fb73e716b in WKS::gc_heap::try_allocate_more_space (acontext=0x7fbcca13c5f8, size=160, flags=0, gen_number=0) at /__w/1/s/src/runtime/src/coreclr/gc/allocation.cpp:4014
Thread 1 (Thread 0x7fcc953bd880 (LWP 322433)):
#1  0x00007fcc95044d48 in raise () from /usr/lib/libc.so.6
#2  0x00007fcc950257c1 in abort () from /usr/lib/libc.so.6
#3  0x0000556fb73df929 in SystemNative_Abort () at /__w/1/s/src/runtime/src/native/libs/System.Native/pal_threading.c:369
#4  0x0000556fb7444682 in FailFast (message=..., exception=..., errorSource=..., reason=<optimized out>, pExAddress=<optimized out>, pExContext=1) at /_/src/runtime/src/coreclr/nativeaot/System.Private.CoreLib/src/System/RuntimeExceptionHelpers.cs:371
#5  0x0000556fb7443ff5 in GetRuntimeException (id=322433) at /_/src/runtime/src/coreclr/nativeaot/System.Private.CoreLib/src/System/RuntimeExceptionHelpers.cs:99

@JulieLeeMSFT

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

@agocke

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

[NativeAOT] Recognize recursive tail-call epilogues on Unix x64 (#135990

Navigation Menu

Repository navigation

NativeAOT linux-x64: GC return-address hijack corrupts the caller's frame when a thread is suspended in the epilogue of a self-recursive fast tail call #135912

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Analysis

Suggested fixes (any one of them)

Related issues

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

NativeAOT linux-x64: GC return-address hijack corrupts the caller's frame when a thread is suspended in the epilogue of a self-recursive fast tail call #135912

Description

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Analysis

Suggested fixes (any one of them)

Related issues

Evidence

Activity

dotnet-policy-service commented on Oct 16, 2026

dotnet-policy-service commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions