Repository navigation
TP > 1: every rank keeps torch's default intra-op thread count, so a multi-rank weight load busy-waits itself to a standstill on a small-core host (19 min -> 61 s) #371
Description
Summary
Nothing in python/freetoken/ sets torch.set_num_threads() on the load path, so every spawned TP
rank (server/launch.py:165) inherits torch's default of one intra-op thread per logical CPU. At
tp_size = N that puts N x cpu_count OpenMP threads on the host's physical cores, and libgomp's
default OMP_WAIT_POLICY=ACTIVE makes every idle thread busy-wait at each fork/join.
On a 6-core host at --tp-size 2 that is 24 threads on 6 cores, and each ~400 KB Tensor.copy_ in
the expert placement loop costs ~8.4 ms instead of ~0.04 ms. The cold load takes 19 minutes
instead of 61 seconds. The disk is idle throughout and both ranks sit at ~600 % CPU.
The trigger is tp_size x cpu_count > physical cores, so it is a property of the host shape, not of
any vendor or model. A large-core host at TP=2 will likely never see it.
The measurement, part 1: the mechanism, CPU only
A standalone harness, no GPU passed to the container, no model loaded, no server started. It imports
freetoken.models.weight.iter_expert_tensors_parallel and replays the six placement writes of
models/nvfp4_banks.py::load_nvfp4_expert_source_banks_parallel verbatim, in two spawn children
started the way server/launch.py starts ranks. Checkpoint is Qwen3.8-Flash-Next NVFP4: 193 expert
shards, 63.3 GiB, H=2560 I=640 E=512 L=48.
96 of the 193 expert shards, 31.66 GiB per rank, two ranks:
| arm | rank wall | rate |
|---|---|---|
as shipped (12 threads/rank, ACTIVE) |
604.84 s | 54 MiB/s |
OMP_WAIT_POLICY=PASSIVE |
15.94 s | 2034 MiB/s (37.9x) |
torch.set_num_threads(3) |
12.82 s | 2529 MiB/s (47.2x) |
Thread-count sweep, 20 shards, two ranks:
| threads per rank | rank wall |
|---|---|
| 1 | 3.98 s |
| 2 | 3.17 s |
| 3 | 2.98 s |
| 4 | 3.00 s |
| 6 | 3.55 s |
| 12 (the default) | 131.85 s |
Every count from 1 to 6 removes the collapse; only the default triggers it. The cliff is exactly at
2 ranks x 12 > 12 logical CPUs.
The tell that it is barrier overhead and not copy cost: with the default, all six placement
writes cost ~21.5 s each over 2560 calls — ~8.4 ms per call — and the write that moves one eighth
the bytes costs the same as the one that moves eight eighths. A copy whose cost is independent of its
size is not a copy cost.
What it is not. Two concurrent ranks move 13.19 GiB with 1.82 s each inside preadv (~7.2 GB/s
aggregate), so the disk is not the term. One process running the same placement shape finishes in
2.74 s, so the shape is not the term either. Only two ranks at once with the default thread count
collapses.
PASSIVE costs the single-rank case nothing: 2.76 s with it against 2.94 s without, one process, 20
shards.
The measurement, part 2: the real cold load, both GPUs
One env line on the container, OMP_WAIT_POLICY=PASSIVE. Same image, same command, no rebuild, no
code change. Control is the same box's own load ten hours earlier.
| term | as shipped | PASSIVE |
|---|---|---|
| startup -> dense weights -> PLE | 7 s | 7 s |
| expert read | 1147 s | 32 s |
| pin settle (exposed once the read shrinks) | 3 s | 3 s |
| CUDA-graph capture | 18 s | 18 s |
| relay -> ready | 0 s | 1 s |
| load to ready | 1175 s | 61 s |
The engine's own expert progress bar: 63.3G/63.3G [00:32<00:00, 2.48GB/s] against the control's
[19:07<00:00, 59.7MB/s].
Decode afterwards is unchanged — 35.2 / 35.2 / 35.7 / 35.7 tok/s, the same band as before the change
— so PASSIVE is not paid back at serve time even though it is process-wide and applies to every
ATen CPU parallel region for the server's life.
Suggested fix
server/launch.py::_run_scheduler, after import torch and before Scheduler(args):
if args.tp_info.size > 1:
clamp = max(1, len(physical_core_cpus()) // args.tp_info.size)
if clamp < torch.get_num_threads():
logger.info_rank0(
f"torch intra-op threads: {torch.get_num_threads()} -> {clamp} "
f"(tp_size={args.tp_info.size})"
)
torch.set_num_threads(clamp)physical_core_cpus() already exists at moe/cpu_executor.py:92, and cpu_executor.py:246-253
already does exactly this clamp-and-log — it just runs after the expert load, so it does not help
here. The two compose: the later clamp only ever lowers the count.
Measured, the clamp is slightly better than the env var (47.2x against 37.9x), which on our load is
worth ~6 s out of 61. The reason to prefer it is not the 6 s — it is that it is tp_size-aware and
protects every host, where an env var protects one deployment.
Happy to open this as a PR if you would like it in that form.
Scope, and what we did not measure
- Measured on our tree, which carries local patches: an intermediate-dim expert shard for TP
(that is Offloaded MoE ignores tensor parallelism: TP=2 is a regression, and the fix is not expert parallelism #62's subject), feat(qwen4_exp): stream the PLE n-gram table from disk #311's disk PLE, and a chunked-multimodal-prefill patch. The mechanism
does not depend on any of them — the harness imports your reader and replays your placement
writes, and the trigger is only "N spawned ranks each defaulting tocpu_countintra-op threads". - On upstream
maineach TP rank loads the full expert set rather than half, so if anything this
should be worse there than here, not better. We have not run upstreammainat TP=2 to confirm
that and are not claiming it. - Measured on the offloaded-MoE expert-bank load path. Any ATen CPU parallel region during a
multi-rank load should be exposed the same way; we only measured this one. - No quality claim — this box has no fidelity instrument, and nothing here touches numerics.
- Possibly adjacent, explicitly not claimed as the same bug: ft serve hangs at 96% weight load on Threadripper 1950X + RTX 5060 Ti #261 (
ft servehangs at 96 %
weight load, load average > 10, box unresponsive) is single-rank, so this mechanism does not apply.
But "CPU pegged, disk idle, machine unresponsive" is the same signature, and
OMP_WAIT_POLICY=PASSIVEis a free thing to ask that reporter to try.
Platform
AMD Ryzen 5 7400 (6 physical cores / 12 logical) · AMD Radeon AI PRO R9700 x2 (gfx1201),
Linux, 122 GiB host RAM · PyTorch 2.11.0+rocm7.14.0, HIP 7.14.60850, RCCL 2.30.4 · FreeToken
main@4b94bdc3 + #132/#133/#134 + #311 + our sharding work.
The core count is the whole story here; the GPUs are incidental.
Activity
gdevenyi commentedon Sep 5, 2026
Measured the opposite on a large-core host — which is the case your write-up predicts will never see this, so this is confirmation rather than contradiction. Posting it because it constrains the shape of the fix: a constant would badly regress hosts like this one.
Box: 2 x Intel Xeon Gold 6526Y, 2 sockets x 16 cores x 2 threads = 64 logical / 32 physical, 2 x RTX 6000 Ada, TP=2.
torch.get_num_threads()reports 32 per rank and neitherOMP_NUM_THREADSnorOMP_WAIT_POLICYis set, so as shipped we run 64 OpenMP threads on 32 physical cores — 2x oversubscription against your 4x.Harness: two
spawnchildren, each replaying the placement write ofnvfp4_banks.py(contiguousTensor.copy_of[640, 2560]bf16 slices into one big host buffer), 400 shards = 640 MB per rank, warmed, worst-rank wall. Two independent runs:torch.set_num_threads(16)= physical // tp_sizetorch.set_num_threads(8)torch.set_num_threads(3)(your best arm)OMP_WAIT_POLICY=PASSIVESo the two arms that buy you 37.9x and 47.2x cost 2.4x and 3.1x here. Either as an unconditional change would turn a 61-second load into three minutes on this class of host.
The rule that satisfies both hosts is
physical_cores // tp_size. It is neutral here (16, within noise of the default) and on your 6-core box it yields 3 — which is the exact value your sweep found best. That is a one-line policy with no host detection beyond what torch already knows, and it degrades to today's behaviour at TP=1 on any host wherephysical >= cpu_count. I would not ship a constant, and I would not shipPASSIVEglobally: libgomp's spin is doing real work when the threads have cores to sit on.Two caveats on my numbers so they are not over-read:
Which supports your framing exactly: the trigger is
tp_size x cpu_count > physical cores, and the fix should be computed from that ratio rather than pinned to a number.🤖 Generated with Claude Code
https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
andrewhosf commentedon Sep 11, 2026
Repro confirmation + a workaround from a different host (FreeToken v0.1.2).
Host: Threadripper PRO 3945WX (12C/24T), 224 GB DDR4-3200 ECC (220 GiB usable; 7 of 8 channels populated), 2× NVIDIA CMP 170HX 64 GB (sm_80, HBM2e; x16 link at Gen2), Linux. Load: 300B-A-class MoE,
ft serve --tp-size 2 --moe-backend offload, ~137G expert pool in host RAM. With tp=2, torch's default intra-op count here was 48 threads on 12 cores (1 per logical CPU per rank, as described in this issue).What the oversubscription looks like from outside (serial path): the load reaches ~70% pool fill, then the host descends into full user-space starvation — SSH banner timeouts, node exporters unresponsive,
systemd-oomdstarting to reap unrelated services, journald going quiet; the kernel itself still answers pings. One instance required a hard reset; a later reproduction showed the same signature (stalled, then no recovery within ~8 minutes). On the serial path it is compounded bysafe_openreads filling the page cache while the expert pool grows — near the capacity wall the spin and the reclaim pressure combine into a standstill.Workaround that has fully stabilized multi-rank loads here (no patch):
per-rank threads 6 → 24 total on 12C/24T;
PASSIVEturns the intra-op busy-wait into a sleeping wait. With this (plus a cold page-cache start before serving), four subsequent multi-rank load attempts produced zero host-level starvation — every attempt either progressed or was aborted cleanly by an external watchdog (PSIfullavg10 +MemAvailable, graceful SIGTERM of the workload). The watchdog pattern is what makes unattended runs safe; happy to share it if useful.Happy to test any candidate fix for this issue on the same box.
gdevenyi commentedon Sep 11, 2026
Tested your scoping claim — "a large-core host at TP=2 will likely never see it" — on a host at the other end of the range. It holds, and the default is already near-optimal there.
Host: 2 × Xeon Gold 6526Y, 32 physical / 64 logical cores, TP=2, serving
nvidia/Qwen3.8-Flash-Next-NVFP4(qwen4_exp, 512 experts × 48 layers) fromdeploy/chatdnp. Two spawned ranks each replaying the host-side expert-placement copy — contiguous slice writes into a large host buffer, the waynvfp4_banksdoes — 400 shards of ~1.6 MB per rank:set_num_threads(phys // tp)= 16set_num_threads(8)set_num_threads(3)(your fix's value here)OMP_WAIT_POLICY=PASSIVETwo things worth extracting from that.
Torch's default is physical cores, not logical. It picked 32, not 64, so at TP=2 this host runs 2 × 32 = 64 threads on 32 physical cores — 2× oversubscription, not the 4× the logical count suggests. At 2× the fork/join cost is invisible. Your 6-core host at TP=2 is 12 threads on 6 cores by the same arithmetic, which is also 2× — so the ratio alone does not predict the pathology, and the absolute core count matters. Worth saying in the issue, because
tp_size × cpu_count > physical coresas a trigger condition would flag this host too, and here there is nothing to fix.OMP_WAIT_POLICY=PASSIVEis actively harmful at this size — 3.3× slower than leaving it alone. If the fix ends up setting that globally rather than clamping threads, it would be a real regression on large-core hosts. A thread clamp is the safer shape: atphys // tpwe measure 1.04×, i.e. neutral-to-marginally-positive, so it costs nothing here while fixing your case.For context on why I went looking: I was hunting launch-time wins on this box. Our cold start is 145 s and breaks down as PLE table read 78 s, expert bank build 38 s, prefill warmup 14 s, weight load and FP8 quantise ~13 s, CUDA-graph capture 2 s. Thread oversubscription turned out not to be a contributor here — but on your host it clearly is, and the bank build is exactly the phase that would suffer.
🤖 Generated with Claude Code
https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt