[drrros]

Overview

Support pipeline parallelism with MoE experts in host RAM

Additional information

When part of the MoE weights is in host RAM (-ot, --cpu-moe, --n-cpu-moe, or fit when the model does not fit in VRAM), llama.cpp turns pipeline parallelism off (!has_tensor_overrides() in llama-context.cpp). All the experts are then uploaded through the first device that can (GPU0 on CUDA) and the GPUs run strictly one after another.

This PR keeps pipeline parallelism working with the weights in host RAM. Each GPU uploads the experts of its own layers, consecutive ubatches overlap across devices. Long prompt processing is x1.5-2 faster on my 3x RTX PRO 4000, short prompts and generation stay about the same.

Benchmarks

3x RTX PRO 4000 Blackwell (PCIe 5 x16, P2P), EPYC 9274F, 384 GB DDR5, CUDA 13.3. Common settings -fa on -b 4096 -ub 4096 -lm none, CUDA_SCALE_LAUNCH_QUEUES=4x. PP is for the 1st / 2nd request of a about 50k-token RFC 7540.

setup	master	this PR
Qwen3.8-Flash-Next Q8_0, server, fit preset	1009 / 1019	1808 / 2102
Qwen3.5-35B-A3B, -ncmoe 999, server	2705 / 2725	5486 / 5901
Qwen3.8, llama-bench fit, pp8192 / pp32768 / tg64	1464 / 1334 / 46.2	1749 / 2474 / 47.1
Qwen3.5-35B-A3B fully in VRAM, server	7659 / 8196	7520 / 8226
DeepSeek-V4-Flash Q8, server, fit	625	1128
DeepSeek-V4-Flash Q8, llama-bench fit, pp32768 / tg128	711 / 24.0	1484 / 24.2

Commits:

1. ggml : upload user inputs once per device before the splits. In pipeline mode every graph input (KQ mask and friends) was copied n_copies times per device and the copies were held for the whole graph. Now one copy per device, uploaded async on the stream of the split that uses it. Alone it does not change the speed, but it cuts the compute buffers from 9133 to 2988 MiB per GPU (Qwen3.5-35B in VRAM, c=262144, and the saving grows with the context).

2. llama : keep graph topology constant with pipeline parallelism. With host weights the executed graph is not always the worst-case one, and then every ubatch of a prompt reallocated the compute buffers, which synchronizes all devices. So after a smaller graph the worst-case graph is reserved again. This is what makes the host-weights case fast at all: without this commit the branch gives 983/991 instead of 1809/2095 on the Qwen3.8 server bench.

3. ggml : pipeline parallelism with weights in host memory. The main part. One copy of a host weight per device instead of 4 held for the whole graph, the allocator reuses it between splits, each split backend writes it in stream order. Ops with host weights run on the device of their own layer through the graph callback, not all on GPU0. The pipeline is now enabled with overrides, but only when the host weights are spread over the devices - on the master fit layout (all host weights on the last GPU) it stays off and runs at master speed. Reading the MoE ids waits for the split backend and would stall the graphs in flight, so on large batches the whole expert tensor is uploaded instead (>= 16 ids per expert with a graph in flight, else a full batch with >= 64). Also fixes ggml_backend_sched_reserve_size to reset after split_graph so that the fit probes see the backend assignments made by the callback.

4. common : balanced layer slices for fit with pipeline parallelism. Master fit stacks all host weights on the last device (Qwen3.8 gets -ts 7,4,38), which is exactly the one-GPU upload this whole thing is about. When the pipeline conditions hold, fit now gives every GPU an equal slice of layers and converts them to full layers per device as they fit (-ts 16,16,17).

the CUDA top-k radix select from the earlier version of the branch is dropped, #28713 landed and already covers that path in master.

Corresponding discussion thread: #29964

Requirements

• I have read and agree with the contributing guidelines
  AI usage disclosure: YES, paired with opus\fable

[am17an]

can you post some perf results on the PR? I can try to reproduce them and test

[drrros]

@am17an there are some perf test in #29964 discussion thread, but i will make some benchmarks on fresh master vs PR and also with --moe-cache-mib option (thank you for that one)

[drrros]

@am17an Rebased and did multiple benches:
Hardware: 3x RTX PRO 4000 Bw, all on pcie 5 x16, EPYC 9274F, 384 GB DDR5 (12 channels), CUDA 13.3.
Common settings CUDA_SCALE_LAUNCH_QUEUES=4x, -lm none, -fa on, -b 4096 -ub 4096 for Qwen3.8 also --lazy-mode off.
Raw test results was kind of a mess for readability, asked llm to clean up, numbers are correct:

Test types:

• Server: RFC 7540 prompt of 52421 tokens without the prompt cache. PP is given for the 1st / 2nd request.
• TG: greedy generation on a short prompt, either 256 or 1024 tokens.

#	Model and setup	Test	master	branch	Notes
1	Qwen3.8-Flash-Next Q8_0, server with the preset (fit, -fitt 1024, c=262144)	PP, t/s	1009 / 1019	1808 / 2102	fit: master -ts 7,4,38, branch 16,16,17
2	same + --moe-cache-mib 6144	PP, t/s	981 / 987	1687 / 2049	master fit 7,21,21
3	same + --moe-cache-mib 12288	PP, t/s	937 / 949	1636 / 1953	master fit 27,17,5
4	row 1 setup	TG 256, t/s	42.0	41.6
5	row 2 setup (cache 6 GB)	TG 256, t/s	29.2	28.1	cache hit rate 56% / 55%
6	row 3 setup (cache 12 GB)	TG 256, t/s	30.4	33.0	cache hit rate 66% / 70%
7	Qwen3.5-35B-A3B Q6_K fully in VRAM, server, c=65536, -ngl 99 --fit off	PP, t/s	7659 / 8196	7520 / 8226	the #29184 regression is gone from master (#29986)
8	Qwen3.5-35B-A3B, -ncmoe 999, server, c=65536, -lm none	PP, t/s	2705 / 2725	5486 / 5901	experts in pinned host memory (CUDA_Host)
9	same with the default load mode (mmap)	PP, t/s	2029 / 2052	2088 / 2101	experts in CPU_Mapped; no gain
10	Qwen3.8, llama-bench, fit (-fitt 1024/1024/1024)	pp8192 / pp32768 / tg64, t/s	1464 / 1334 / 46.2	1749 / 2474 / 47.1	-r 2
11	same without CUDA_SCALE_LAUNCH_QUEUES	pp8192 / pp32768 / tg64, t/s	1466 / 1335 / 46.2	1383 / 1824 / 46.6	master does not depend on it (no pipeline)
12	Qwen3.5-35B-A3B, llama-bench, -ngl 99 -ncmoe 999	pp8192 / pp32768, t/s	3311 / 3014	4717 / 6642	master -r 1, branch -r 2
13	Qwen3.8, server, own fit with -fitt 1024, no cache	TG 1024, t/s	41.7	41.6	layouts: master 7,4,38, branch 16,16,17
14	same with -fitt 3072 (2 GB more free VRAM per GPU), no cache	TG 1024, t/s	41.6	40.5	layouts: master 6,6,37, branch 16,16,17
15	branch layout from row 14, --fit off with the llama-fit-params output, no cache	TG 1024, t/s	40.6	40.5	same layout on both
16	row 15 + --moe-cache-mib 6144 (about 2 GB per GPU)	TG 1024, t/s	28.8	28.8	hit rate 57.35% on both, same counters; PCIe rx 25-30 GB/s vs 0.2 GB/s without the cache
17	own fit + --moe-cache-mib 6144	TG 1024, t/s	29.7	-	master fit 7,21,21, cache 3 + 3 GB on CUDA1/2, hit rate 60.6%
18	row 1 setup, commit 3 as in the draft PR (pad n_outputs 0 to 1 + re-reserve with the current n_outputs)	PP, t/s	1009 / 1019	1798 / 2075	fails test-recurrent-state-rollback (new test_shared_seq_reserve)
19	same, without commit 3	PP, t/s	1009 / 1019	983 / 991
20	same, only the n_outputs padding	PP, t/s	1009 / 1019	974 / 978
21	same, padding + re-reserve of the worst-case graph	PP, t/s	1009 / 1019	1799 / 2083
22	same, only the re-reserve of the worst-case graph (the new commit 3)	PP, t/s	1009 / 1019	1809 / 2095	passes the test

So with --moe-cache-mib TG slower for me (both on master and on this branch).
Below is Claude's addition if It could be of any help:

Details

Notes on rows 13-17 (why the MoE cache slows down generation here):

• Freeing VRAM for the cache costs little. It is -0.2% to -2.6% (rows 13-15).
• The slowdown comes from the cache itself (row 16).
  • At a 57% hit rate, each token misses about 166 experts of about 5 MiB each, so about 830 MiB per token goes over PCIe.
  • The misses of a layer are uploaded only after its router runs.
  • During generation one GPU works at a time.
• Without the cache, the CPU computes the same experts directly from DDR5, which is faster here.

Tests on the branch head (95c7ce509):

• every commit builds with CUDA and no warnings;
• ctest 60/60, including test-recurrent-state-rollback with GGML_SCHED_DEBUG_REALLOC=1;
• test-backend-ops -o TOP_K 531/531, -o ARGSORT 98/98;
• test-llama-archs: no failures. This includes the "Host experts", "MoE cache" and "MoE cache, layer split" configs. The "Host experts" model has a single MoE layer, so the pipeline is not enabled in that test.
• With a 6 GB cache, master at the branch layout (pipeline off) and the branch (pipeline on) produce identical greedy text.

[drrros]

also pictures of nvtop while running llama-bench:
master:

branch:

[am17an]

Cache rate needs to be above 70-80% for the speed-up from what I saw. But otherwise the PP results look quite good to me. Can you clean up the PR and make it ready for review?

[am17an]

Also measure some real workload other than llama-bench, as random tokens wouldn't have lower hit rates than something in distribution.

[drrros]

Also measure some real workload other than llama-bench

Bench results above except 10,11,12 are from server mode, processing RFC 7540 prompt of 52421 tokens (just copypasting part from head of rfc page)

Can you clean up the PR and make it ready for review?

Will do

In #29964 there was a discussion that in some configurations this PR can lower performance, my plan was to introduce new option for -fit something like balanced (to not change "on" behavior) - leave that for follow up pr, or implement in current?

[drrros]

Removed first commit as all required changes already in master via #28713 did a clean up, fixed long comments, updated description, marking as ready for review

[mixa3607]

• Master: 23b0202a189c44a54625aadcb37a946dd1d6278d.
• PR head: f81e182b340c452adb4725cd6a44c5d6734baea1.
• PR merge base: 5e1d74043eb84f5f82471a1c5d4c8171a4c0fcba (not identical to tested master).
• Hardware: 4 x gfx906 MI50, 32 GiB each; 76 logical CPUs, 2 NUMA nodes.
• ROCm 7.14, HIP graphs and RCCL enabled, Release, max-ilp, identical build settings.
• Model: DeepSeek-V4-Flash-Vision-orcarouter MXFP4, snapshot e262f88e8b1fdd390a2c0cad2af436dce4c8fa0e, all 4 shards; mmproj loaded.
• Context 131072; batch/ubatch 2048; 32/32 threads; NUMA interleave; layer split 25/25/25/25.
• HIP_VISIBLE_DEVICES=1,0,3,2; HSA_FORCE_FINE_GRAIN_PCIE=1; HSA_XNACK=0.
• Server /completion, one 512+16 warmup, then 3 repetitions per prompt depth, each generating 256 tokens.
• Identical tokenized server README prompts, seed 1234, ignore_eos=true, cache_prompt=false.
• PP counts and TG=256 verified for every response. TG is measured after the indicated prompt depth, not at an empty context.
• Mean +/- sample standard deviation, tokens/second; startup and HTTP overhead excluded.

llama args

command:
- numactl
- --interleave=all
- /app/llama-server
args:
- --model
- /root/.cache/huggingface/hub/models--orcarouter--DeepSeek-V4-Flash-Vision-Uncensored-GGUF/snapshots/e262f88e8b1fdd390a2c0cad2af436dce4c8fa0e/DeepSeek-V4-Flash-Vision-Uncensored-MXFP4-00001-of-00004.gguf
- --mmproj
- /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-Vision-Exp-GGUF/snapshots/87ce15810931a270f165e5e50439cd446f6b4ba5/mmproj-F16.gguf
- --alias
- DeepSeek-V4-Flash-Vision-orcarouter-nocache-rocm
- --ctx-size
- '131072'
- --batch-size
- '2048'
- --ubatch-size
- '2048'
- --parallel
- '1'
- --threads
- '32'
- --threads-batch
- '32'
- --device
- ROCm0,ROCm1,ROCm2,ROCm3
- --main-gpu
- '0'
- --n-gpu-layers
- all
- --split-mode
- layer
- --tensor-split
- 25,25,25,25
- --override-tensor
- (blk.0.ffn_(up|down)_exps.weight|blk.1.ffn_(up|down)_exps.weight|blk.2.ffn_(up|down)_exps.weight|blk.3.ffn_(up|down)_exps.weight|blk.4.ffn_(up|down)_exps.weight|blk.5.ffn_(up|down)_exps.weight)=CPU,(blk.11.ffn_(up|down)_exps.weight|blk.12.ffn_(up|down)_exps.weight|blk.13.ffn_(up|down)_exps.weight|blk.14.ffn_(up|down)_exps.weight|blk.15.ffn_(up|down)_exps.weight|blk.16.ffn_(up|down)_exps.weight)=CPU,(blk.22.ffn_(up|down)_exps.weight|blk.23.ffn_(up|down)_exps.weight|blk.24.ffn_(up|down)_exps.weight|blk.25.ffn_(up|down)_exps.weight|blk.26.ffn_(up|down)_exps.weight|blk.27.ffn_(up|down)_exps.weight)=CPU,(blk.33.ffn_(up|down)_exps.weight|blk.34.ffn_(up|down)_exps.weight|blk.35.ffn_(up|down)_exps.weight|blk.36.ffn_(up|down)_exps.weight|blk.37.ffn_(up|down)_exps.weight)=CPU
- --load-mode
- mlock
- --cache-ram
- '32768'
- --spec-type
- none
- --temp
- '1.0'
- --top-p
- '1.0'
- --min-p
- '0.01'
- --no-context-shift
- --cache-idle-slots
- --jinja
- --metrics
- --props
- --slots
- --offline
- --fit
- 'off'
- --host
- 0.0.0.0
- --port
- '8080'
- --log-colors
- 'off'

Prompt	Master PP	PR PP	PP change	Master TG256	PR TG256	TG change
512	81.98 +/- 0.91	83.24 +/- 0.44	+1.5%	14.44 +/- 0.12	14.65 +/- 0.21	+1.4%
8192	149.68 +/- 0.04	191.60 +/- 0.05	+28.0%	13.74 +/- 0.07	14.00 +/- 0.06	+1.9%
32768	122.47 +/- 0.01	312.46 +/- 0.04	+155.1%	13.09 +/- 0.01	13.30 +/- 0.10	+1.5%

[aminya]

Testing this with Qwen 3.8 Flash Next with or without MTP enabled, unfortunately, the prefill speed is halved in a dual 3090/3060 GPU setup. Previously, it was ~1200 without MTP and ~600t/s with MTP (#27428) but with this PR it goes down to ~300t/s using the exact same flags regardless if MTP is enabled or not.
All flags are llama-server defaults, no behaviour override.
Looking at the VRAM distribution, I could see that the VRAM is no longer fully utilized even with --fit-target set to something small like 100 MB. It leaves out ~8GB on my GPU for some reason.

[sasa7812]

How do you get 1200 on 3090? Can you share the secret (config)?

[aminya]

Here it is: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF, the pruned version of Flash Next, which fully fits my VRAM.

[sasa7812]

Ah... got it, it is fully resident. Thank you for the link

[drrros]

@aminya not sure how to debug this, can you show what llama-fit-params generates for your setup?

[drrros]

@am17an @ggerganov PTAL