[eelgaev]

aarch64 / NVIDIA DGX Spark (GB10): build, run and set up with unified memory

This PR makes Strata build and run on aarch64, and ./setup.sh work end to end on an NVIDIA DGX Spark. It is experimental and community-tested, in the same spirit as the AMD HIP and sm_70 builds: one DGX Spark, Qwen3.8-Flash-Next IQ2_XS installed by setup. x86 builds are unchanged.

Speed (IQ2_XS, setup's own run-iq2_xs.sh: 128K context, --kv int8)

All 24,576 experts fit in the GPU's expert cache (33.0 GiB), so the CPU computes none.

Prompt	Context	Prefill	Decode	MTP acceptance
2K	0	928 tok/s	58.7 tok/s	78%
8K	0	1,356 tok/s	62.0 tok/s	81%
2K	16K	1,421 tok/s	60.5 tok/s	83%
8K	16K	1,450 tok/s	58.2 tok/s	79%
2K	64K	1,505 tok/s	58.9 tok/s	80%
8K	64K	1,515 tok/s	54.9 tok/s	75%

llama-benchy 0.4.0, 3 runs after a warm-up, --no-cache, one request at a time, 128 tokens decoded per request. Prefill and decode are from the server log (strata serve: prompt ...). benchy's pp column cannot time Strata's server, because the first stream chunk is sent before the prompt is read; details are in docs/DGX_SPARK.md.

What changes

The change splits naturally into three parts. Happy to send them as separate PRs if you prefer.

1. Build and engine on aarch64

• src/kernels/cpu/portable.cpp replaces the AVX-only kernel files on non-x86 (CMake chooses by CMAKE_SYSTEM_PROCESSOR). It contains the activation quantizer (expert.cpp's scalar loop) and the BF16 router dot (NEON). The AVX multi-token kernels report themselves unsupported, so native experts use ggml-cpu's NEON vec_dot.
• Q2_0 expert rows (the down projections of every setup model) go through ggml-cpu on non-x86 (q2_own_kernels()). x86 keeps q2_rows_any.
• include/strata/platform/cpu_relax.hpp: _mm_pause / _mm_sfence on x86 (as before), yield / dmb oshst on aarch64.
• -ffp-contract=off for C/C++ on non-x86 GCC/Clang, as an x86 build without -mfma effectively has, so host roundings match x86 and ggml. (quantize_act_parity's Q8_K case failed without it.)
• GGML_CPU_ARM_ARCH turns GGML_NATIVE off: gcc's -mcpu=native finds no features on GB10's Cortex-X925/A725, which leaves ggml-cpu without dotprod/i8mm.

2. Expert cache sizing on unified memory

• device_free_bytes() is used by both the cache sizing and ExpertCache::open's check. When the driver reports that the GPU shares host memory (cudaDevAttrIntegrated), it counts MemAvailable less 6 GiB (STRATA_UMA_HEADROOM_GIB) as free. cudaMemGetInfo counts only free RAM, and with --mmap-experts the page cache is mostly the GGUF's own pages. NVIDIA's DGX Spark Porting Guide (§5.5) recommends the same; swap is not counted.
• Without it, the cache stopped short of all experts, or was refused outright after a large file filled the page cache (45.29 GiB wanted, 44.64 GiB "free", 118 GiB available).
• hip_compat: cudaDevAttrIntegrated → hipDeviceAttributeIntegrated.

3. setup

• A unified memory GPU is detected from the driver (CU_DEVICE_ATTRIBUTE_INTEGRATED via libcuda, matched to nvidia-smi by PCI address; memory.total [N/A] only as a fallback) and gets the system's RAM as its memory.
• aarch64: no ready-made engine (the release is x86-64), so it compiles with GGML_CPU_ARM_ARCH from /proc/cpuinfo (engine and strata-vision); the CUDA repository key comes from sbsa; cpu_info() handles the absence of an x86 flags line; no AVX2 requirement.
• Unified memory: --mmap-experts (an expert arena in RAM would be a second copy of the cache in the same memory) and no KV streaming.

Plus docs/DGX_SPARK.md and a README pointer next to AMD HIP.

Tested on a DGX Spark

• ./setup.sh --model IQ2_XS --gguf-dir ... --vision no --yes: detects the GB10, compiles, packs, writes the start script.
• Correctness: coding, factual and arithmetic prompts answered correctly at 66-77 tok/s.
• The CPU path: with --expert-cache 2048 the CPU computed ~22 of ~29 experts per layer, Q2_0 rows included. Output was identical, token for token, to the all-GPU run (18.8 vs 81.9 tok/s).
• ctest: 51 of 52 pass. ple_parity needs the Q2_0 model's second shard, which was not present. expert_multi_test (the AVX-512 Q2_0 kernel) is now registered on x86 only.
• tools/test_setup_*.py: pass, except test_setup_choices, which fails the same way on unmodified main on this machine (find_nvcc). test_setup_unsloth simulates an x86 PC and now pins setup.ARM to False, as it already pins the CPU and GPU.

Not covered

• Models other than IQ2_XS were not run. All of their expert formats (IQ1_M, IQ2_XXS/XS/S, IQ3_XXS/S, IQ4_XS/NL, Q2_0; Q4_K/Q5_K/Q5_1/Q8_0 for the manual imports) have a ggml NEON path.
• Images (--vision): built with the ARM flags, not tested.
• The HIP build was not compiled (no ROCm here); the one HIP-side change is the attribute mapping above.
• Strata's x86 multi-token CPU kernels have no NEON version. On a Spark the CPU computes no experts, so this only matters with a reduced cache. I'm happy to port them as a follow-up if you want ARM CPUs to carry experts.

🤖 Generated with Claude Code

[eelgaev]

Updated the code and tested with Unsloth's UD-Q4_K_XL with the latest main, works pretty well!

[eelgaev]

Rebased to latest main, better numbers now.

IQ2_XS

Prompt	Context	Prefill before	Prefill after	Δ	Decode before	Decode after	Δ	MTP accept before → after
2K	0	928	957	+3.1%	58.7	55.7	-5.1%	78% → 72%
8K	0	1,356	1,357	+0.1%	62.0	59.7	-3.7%	81% → 74%
2K	16K	1,421	1,410	-0.8%	60.5	63.7	+5.3%	83% → 73%
8K	16K	1,450	1,503	+3.7%	58.2	57.3	-1.5%	79% → 80%
2K	64K	1,505	1,538	+2.2%	58.9	56.2	-4.6%	80% → 77%
8K	64K	1,515	1,548	+2.2%	54.9	56.4	+2.7%	75% → 75%
mean		1,362	1,386	+1.7%	58.9	58.2	-1.2%

UD-Q4_K_XL

Prompt	Context	Prefill before	Prefill after	Δ	Decode before	Decode after	Δ	MTP accept before → after
2K	0	610	590	-3.3%	39.2	38.4	-2.0%	76% → 74%
8K	0	1,127	1,117	-0.9%	41.5	39.2	-5.5%	75% → 72%
2K	16K	1,089	1,058	-2.8%	35.5	41.8	+17.7%	73% → 80%
8K	16K	1,183	1,173	-0.8%	42.3	39.7	-6.1%	78% → 78%
2K	64K	1,166	1,165	-0.1%	36.6	39.4	+7.7%	74% → 79%
8K	64K	1,195	1,187	-0.7%	38.2	41.5	+8.6%	75% → 81%
mean		1,062	1,048	-1.3%	38.9	40.0	+2.9%

Before / After the rebase

	Before	After
IQ2_XS chat decode (3 short prompts)	66–77 tok/s	68–79 tok/s
UD-Q4_K_XL chat decode	41–51 tok/s	41–51 tok/s
IQ2_XS small cache (2,048 slots) vs all-GPU, CLI	18.8 vs 81.9 tok/s	19.7 vs 82.9 tok/s
Output small cache vs all-GPU	identical	identical
UD-Q4_K_XL GPU memory	82.2 GiB	82.2 GiB
ctest	51 of 52	60 of 61

[Niko1221]

Fixed in 0.1.40: the free-memory figure for an integrated GPU (a unified-memory chip such as Strix Halo or the DGX Spark) is now MemAvailable less 6 GiB (STRATA_UMA_HEADROOM_GIB changes it) instead of cudaMemGetInfo's MemFree. The integrated-GPU check is cached per device, so a box with an APU and a discrete card gets the right answer for each. It applies to both backends, on Linux. The aarch64 build break and the rest of the PR are not part of this change.

[uncle-daddy-jp]

FYI for Spark users of this branch: on a DGX Spark (GB10) with the branch at 44df8fd, prompts were read at ~90 tok/s (IQ3_S) because the 32 prefill Stager threads inherit the host thread's core-0 pin and yield-spin there. The fix is in the prefill code shared with main (not in this branch's changes): #1056 / #1057 make the waits sleep, and the same binary then reads at ~1,200-1,400 tok/s with ~1 core busy. Thanks for the port, it built and ran first time here.