NUMA-aware placement: keep hot pools on the GPU's node, let cold pools (expert file tail, conversation caches) use the other socket — opt-in #1900

On dual-socket machines, memory on the other CPU socket is roughly 2x the latency away (QPI/UPI), and this engine currently has no NUMA-aware placement: the recommended practice is to pin the whole process to the GPU's node by hand (systemd CPUAffinity + NUMAPolicy=bind/NUMAMask, or numactl). Two references: #674 measured engine-wide binding to the GPU node at about +5% over default; #1300 notes there is no supported way to confine the pool to one NUMA node.

That works, but it throws away the other socket's RAM: on machines where both sockets have plenty of memory, cold data (the expert file tail not held in VRAM/RAM, conversation-cache snapshots) must live in the GPU node's RAM or be re-read from disk, while the second node sits idle.

Suggestion (opt-in, default behavior unchanged — the shape is the maintainers' call):

1. A manual flag placing designated cold pools on a chosen NUMA node (e.g. the expert file-tier pages and conversation caches);
2. Or an auto mode: read the topology at startup; keep hot pools (resident expert complement, KV streaming buffers, embeddings) on the GPU's node, and place cold pools on another node when it has free memory. In either case, honoring the rule that a new feature is opt-in and the default path stays byte-identical.

Or, if a maintainer would rather settle on the design, we can write the implementation — happy to prepare the PR once the shape (flag naming, scope, semantics) is decided.

Why it could help:

• The cold expert tail is ~13.5% of expert reads in our logs (≈4.4 GiB in the current profile-rank tail); serving it from socket-2 RAM instead of SSD reduces stalls;
• Memory pressure becomes per-node instead of everything squeezed into the GPU node's memory;
• It gives the "confine to a NUMA node" story an official answer (bench: community report, 2x Tesla V100-PCIE-32GB (sm_70, CUDA 12 source build): IQ3_S at 512K with --layer-split, and what the new decode profiler says about where tok/s comes from #1300).

Not proposed, to preempt confusion:

• Not a global interleave — hot paths must not be diluted (community interleave runs showed decode +4% but prompt -5%, and were not kept);
• Not CPU-pool core limiting (the glm5-next: GLM-5.3-Flash support intial attempt #1315 concerns); this is memory placement only;
• Not about device placement across sockets (Strata on 2x RX 7900 XTX (HIP, AVX-only CPU): Memory access fault by GPU node-2 ... Page not present with concurrent requests Dates below are day/month (2026), times are local (CEST, UTC+2). #1712 was a different topic).

Implementation notes (for whoever picks this up):

• Page-cache pages land on the node of the thread that first touches them, so cold reads would need their own pinned buffers or reader threads with their own policy;
• It must coordinate with external pinning: with a blanket cgroup/systemd binding to the GPU node, the engine cannot place anything on the other node — the docs should say so, and an auto mode would need blanket pinning lifted;
• numa_balancing can disturb static placement.

Measured on: 2-socket Xeon E5 v4 (~94 GB total; 64 GB attached to the GPU's socket), Tesla V100-32G, Flash-Next UD-IQ4_XS, v0.1.42 — hot pools: 27 GiB resident experts + 1.55 GiB KV host buffers + 644 MiB embeddings; cold candidates: ≈4.4 GiB expert tail + 4 GiB conversation caches; other socket ~10 GiB reclaimable.