[i-Vladimir-i]

--shared-expert-arena: one engine loads the file, the others wait and attach

#129 added the shared arena as a primitive and left two things to the caller: who loads the file ("no
leader/follower population protocol yet") and its lifetime. Today every engine that names the file reads the whole
pack into it again at start - 55 GiB for UD-IQ4_XS, written into pages another engine may be reading - and nothing
keeps two loads apart. This adds that part, inside the engine, so two engines of one pack (one per GPU, each with its
own expert cache and K/V; #1461's pair is one such setup) keep one RAM copy and load it once.

What it changes (src/core/pinned.cu, pinned.hpp, expert_source.cpp/.hpp, generate.cpp; the selftest;
DETAILS.md; measurements in bench/results/2026-10-13-community-r9700-shared-expert-arena/)

• Who loads: two one-byte OFD locks on the file (Linux 3.15; they belong to the open file, so the kernel drops them
  when an engine exits or dies). FILL is held by the engine that loads the file until the load passed its checks; an
  engine started meanwhile waits on it (shared arena ...: engine N is loading it, waiting) and then attaches - no
  read of the pack, attached 55.43 GiB another engine loaded, nothing read. USER is held shared while an engine maps
  the file.
• Lifetime: the file stays when the last engine exits, so a restart attaches too; deleting it frees the RAM. An
  engine that dies while loading leaves the file to the next one, which loads it again. A file of another pack, size or
  format is replaced when no engine maps it (USER taken exclusively, then unlinked - never truncated under a mapping)
  and refused while one does.
• Format V2: the header is one 2 MiB page, so the arena starts 2 MiB-aligned in the file and in memory and
  hugetlbfs can hold it (the V1 comment said it could not). V1 files are replaced like any other format when unused.
• Pages: the file is allocated up front (posix_fallocate: a tmpfs smaller than the arena is an error naming the
  size, not a SIGBUS part-way through the load), mapped before the GPU registration (MADV_POPULATE_*, Linux 5.14),
  and the loading engine makes its 2 MiB pages whole with MADV_COLLAPSE (Linux 6.1) while they are still empty. The
  start reports N of M GiB on 2 MiB pages; on AMD, when part of it is on 4 KB pages, it says why that matters.
• Why 2 MiB pages matter on AMD: hipHostRegister does not pin host memory on ROCm 7.14.1 (VmPin 0); the
  kernel's compaction moves 4 KB pages, and every move stops the process's GPU queues while the driver maps the range
  again. With 16 GiB registered, one forced compaction: 194-295 ms of stopped queues on 4 KB pages, 0-18 ms on a tmpfs
  huge=always; CPU reads and host->device copies were the same on both (table in DETAILS.md, method and the other
  backings - hugetlbfs, hipHostMalloc, hipMemCreate on the host - in the bench folder).

What it leaves alone: without --shared-expert-arena nothing changes - PinnedArena without a file takes the old
path, shared_publish() is a no-op, and ArenaExpertSource's load is the same code one level deeper. The SYCL tree
keeps its own V1 copy of the option: a V1 engine refuses a V2 file (size and header do not match) rather than
misreading it, and since it takes no locks its V1 file counts as unused here - unlinked, not truncated, so it keeps
its mapping. Linux only: the code is under __linux__; on Windows (as before) and on other platforms, macOS
included, a shared file is refused with "implemented on Linux only".

Checked

• HIP build (gfx1201, ROCm 7.14.1): ctest 108/109, the one failure ple_parity as before (needs the Q2_0 model).
  pinned_shared_test, rewritten for the protocol, runs the other engines as child processes: the second waits while
  the first loads and attaches once it publishes; an attach in the same process; another pack and another size
  refused while mapped; a restart attaches; a file of another pack held by another process refused, replaced once
  that process is gone; a loader killed before publishing leaves the file to the next one. Run on /dev/shm and on a
  hugetlbfs mount (STRATA_TEST_ARENA_DIR): both report the file whole on 2 MiB pages. CUDA: builds and links with
  nvcc 13.4.92, sm_120, 0 errors; not run on an NVIDIA GPU.
• End to end, R9700 32 GB, Unsloth UD-IQ4_XS (55.4 GiB of experts), --spec 4 --mtp, 48 greedy tokens, tmpfs
  size=58G,huge=always,noswap:

run	arena checksum (STRATA_VERIFY_ARENA=1)	tokens	start to exit
default arena (no flag)	92a4fc0d243d7402	reference	105 s
shared file, fresh: loads it	92a4fc0d243d7402	same	108 s
shared file again: attaches	92a4fc0d243d7402	same	68 s
two engines 3 s apart on a fresh file (2,000-slot caches): one loads, one waits and attaches	92a4fc0d243d7402 both	same as each other	111 s, 109 s

The load itself is about 32 s here (1.7-1.8 GiB/s from the file cache); the rest of a start is the same either way.
With the smaller caches the tokens differ from the reference from the 6th on, both engines alike - which experts a
card holds changes the rounding, as the docs say for other cache sizes. The fresh file of the second run was on 2 MiB
pages for 50.6 of 55.4 GiB (the machine had just freed a 55 GiB arena; MADV_COLLAPSE could not make the rest
whole), the fourth for all of it after MADV_COLLAPSE.

• Two engines generating at the same time, one per GPU (the R9700 x16 and a second R9700 on PCIe 3.0 x4), one copy of
  the experts in RAM (69 GB of 125 GB still available with both running), 512 greedy tokens each:

arena file	x16 alone	x4 alone	both at once, two runs: x16 / x4
tmpfs huge=always	63.3 tok/s	42.2 tok/s	63.9 / 41.3, 66.9 / 42.1
hugetlbfs (vm.nr_overcommit_hugepages)	67.2 tok/s	-	66.8 / 37.4, 66.9 / 41.9

Same checksum everywhere; each engine's tokens the same alone and in the pair. A forced compaction during the pair's
decode stopped the x16 engine's queues 0 ms in three of four runs and the x4 engine's 0.6-0.7 s in each, on tmpfs and
hugetlbfs alike - memory other than the arena, not traced here. The hugetlbfs pages (28,384) went back to the system
when the file was deleted. The engine runs on hugetlbfs were made before a fix of the start line's count (hugetlb
pages are in Shared_Hugetlb, not Rss: it said 0 of 55.44 GiB); the selftest on a hugetlbfs mount
(STRATA_TEST_ARENA_DIR) now shows the file whole on 2 MiB pages.

Not tested: NVIDIA at run time (cudaHostRegister pins the file's pages there); the server (serve/) running two
engines (#1461's desk is one); kernels older than 6.1 (MADV_COLLAPSE then reports itself unavailable in the start
line).

🤖 Generated with Claude Code

[i-Vladimir-i]

Rebased on v0.1.44 (ed4773eb), mergeable again. Two commits now:

1. The PR's commit, conflicts resolved. They were with pin by need (arena: pin by need (STRATA_PIN_BY_NEED=1) and a PCIe share floor (STRATA_PCIE_MIN1=1), +5-7 % prompt read on Windows #1905) in pinned.hpp, pinned.cu and
   expert_source.cpp: both sides kept. The ByNeed constructor passes the shared-file state to reserve() like the
   other constructors (pin by need never takes a shared file - one layout for every process - so the two do not meet
   at run time), and the padded-layout arguments of the readers moved into the "not attached" branch.
2. The --batch hangs during startup on Linux/CUDA (3x RTX 3060, 0.1.43): 100% kernel time, GPU idle, the expert cache is never allocated #2126 rule for the shared file. The engine that loads the file now asks for 2 MiB pages (MADV_HUGEPAGE,
   then MADV_COLLAPSE) only when free 2 MiB blocks could back half of the arena; otherwise it says so in the start
   line, and STRATA_ARENA_THP=1 asks anyway. Without it the shared path would have kept the per-fault compaction
   --batch hangs during startup on Linux/CUDA (3x RTX 3060, 0.1.43): 100% kernel time, GPU idle, the expert cache is never allocated #2126 removed from the private arena. Not on hugetlbfs (its pages are reserved); an engine that attaches allocates
   nothing and is unchanged. Seen here: the rule held back on a box with 0 GiB in free 2 MiB blocks (55 GiB free, all
   split, the rest in the file cache), and a huge=always tmpfs still put 55.1-55.2 of 55.4 GiB on 2 MiB pages - the
   MADV_COLLAPSE it skipped would have added the last ~0.3 GiB.

Checked again on the rebased branch (R9700, gfx1201, ROCm 7.14.1):

• ctest 111/112, the one failure ple_parity as before (needs the Q2_0 model); pinned_shared_test passes on
  /dev/shm and on a hugetlbfs mount (STRATA_TEST_ARENA_DIR, the file whole on 2 MiB pages).
• UD-IQ4_XS, --spec 4 --mtp, 48 greedy tokens, tmpfs huge=always: the arena checksum 92a4fc0d243d7402 in every
  run; default arena, a fresh shared file (loads it, 86 s) and the same file again (attaches, 66 s) give the same
  tokens as each other and as before the rebase. Two engines 3 s apart on a fresh file: one loads, the other waits
  ("another engine is loading it, waiting") and attaches without reading; the same checksum in both.
• One difference from the first round: in that pair (2,000-slot caches, both decoding on the same card at once) the
  second engine's tokens left the first's at token 46 of 48; before the rebase the pair matched to the end. The same
  configuration run alone - default arena twice, shared file loading, shared file attaching - gives the first
  engine's tokens every time, so the file is not the cause; with the small cache the adaptive tier's swaps are the
  likely place where two engines on one card can change each other's timing. Not traced further.

🤖 Generated with Claude Code

[shahrokhzargarpour]

Hi! We also touch pinned.cu, but on the other OS: our branch adds chunked 2 MiB large pages on Windows (VirtualAlloc MEM_LARGE_PAGES per chunk), under #ifdef _WIN32.

Your --shared-expert-arena is Linux-only (MAP_SHARED file + OFD locks + /proc/self/smaps), so the two changes are disjoint by platform.

The textual overlap (free_huge_bytes / PinnedArena) is only because both patch the same regions; the struct fields are additive.

Proposal: keep both platform branches in one pinned.cu/pinned.hpp and merge - would you be open to coordinating that?

#2510

[qiuzan2001]

Found by reading (no Mac here to build): src/core/pinned.cu now includes <sys/vfs.h> (line 35) and calls posix_fallocate (line 218) and fstatfs/f_type (line 248) in the non-_WIN32 branch, with no __linux__ guard. The Mac engine compiles this file as C++ (metal/CMakeLists.txt:92 and :101; metal/include/metal_compat/linux/mman.h exists for that), and macOS has no <sys/vfs.h> (statfs is in <sys/mount.h>) and no posix_fallocate; metal/include provides no shim for either. So I expect the Metal build to stop at the include. Putting open_shared and its helpers under #if defined(__linux__), with a "not available" message elsewhere as the Windows branch has, would keep the Mac build compiling. The description lists HIP and CUDA builds only.

[i-Vladimir-i]

Thanks, right: the Metal build compiles pinned.cu, and the new code used <sys/vfs.h>, fstatfs and posix_fallocate without a Linux guard. In 12e85a1 the whole shared-file path (the OFD locks, the smaps reader, open_shared, the unlock in shared_publish) is under __linux__; elsewhere a shared file is refused with "implemented on Linux only", as on Windows. The same commit also checks the error path's ftruncate result, which gcc's warn_unused_result flagged.

Checked: HIP (gfx1201) builds and pinned_shared_test passes; CUDA (nvcc 13.4, sm_120) builds and links, 0 errors, no warnings left in pinned.cu. I have no Mac to build Metal, but nothing Linux-only is left outside __linux__ (checked line by line). SYCL is not touched (it has its own pinned.dp.cpp).

[i-Vladimir-i]

Agreed, the overlap is only textual: your chunked large pages are under _WIN32, and the shared arena is Linux only - since 12e85a1 explicitly under __linux__, with a clear "not available" on other platforms. So pinned.cu keeps three branches, _WIN32 / __linux__ / other, and the new PinnedArena fields are additive. Whichever lands first, the other rebases; if yours goes in first, I will rebase #2271 onto it.

[Niko1221]

Thanks. This saves RAM and start time when two engines serve one pack; it does not change tokens per second, so it is outside this speed round and we have not ported it. It would need a rebase and a two-engine test on one box.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.