Skip to content

[Linux][multi-GPU] 8 GiB cudaHostRegister cap causes ~3x Q2_0 prefill slowdown #253

Closed
Closed
[Linux][multi-GPU] 8 GiB cudaHostRegister cap causes ~3x Q2_0 prefill slowdown#253

Description

@bettercallcaleb
bettercallcaleb
opened on Sep 30, 2026

Summary

On Linux, Strata's multi-GPU path appears to apply the same 8 GiB CUDA host-registration cap that the source comments describe as a Windows/WDDM workaround.

On my Q2_0 setup this pushes most of the expert arena onto the unpinned staging path and causes a severe prefill regression.

A local patch that keeps the 8 GiB cap on Windows but uses unrestricted cudaHostRegisterPortable on Linux improved 32K-context prefill from ~665 tok/s to ~2,110 tok/s and reduced per-chunk wait copy time from ~2 seconds to tens of milliseconds.

Environment

  • Linux
  • Strata engine 0.1.28
  • Revision: bbaaabb4643bb7873d4cef9d48b5dcf96e6cbff4
  • Model: Qwen3.8-Flash-Next, Q2_0
  • Context: 32768
  • KV: int8
  • CPU: Ryzen 5 7600
  • RAM: ~128 GB
  • RTX 4090 24 GB
  • RTX 3060 12 GB
  • GPU order for this A/B: 3060 -> 4090
  • --layer-split auto selected K=2
  • 3060 runs layers 0-1
  • 4090 runs layers 2-47

PCIe measurements:

  • RTX 4090 H2D: 26.90 GB/s
  • RTX 3060 H2D: 6.40 GB/s
  • P2P: unavailable both directions

Stock behavior

Startup:

cudaHostRegister limited to 8 GiB for CUDA1
12 slices pinned (7 GiB)
layer split auto: K=2

Benchmark:

1K:  prefill 597.8 tok/s
4K:  prefill 612.8 tok/s
32K: prefill 664.9 tok/s

With STRATA_PREFILL_TIMING=1, representative 2048-token chunks on the 4090 stage:

GPU timeline 3374 ms ... wait copy 2644 ms (78.4%)
GPU timeline 2946 ms ... wait copy 2208 ms (74.9%)
GPU timeline 3030 ms ... wait copy 2004 ms (66.1%)
GPU timeline 3237 ms ... wait copy 2448 ms (75.6%)

Full ~28K prompt:

wall 42674 ms

Controls

memlock

Raised RLIMIT_MEMLOCK to unlimited.

Performance was effectively unchanged:

before: 32K prefill 654.2 tok/s
after:  32K prefill 652.5 tok/s

So mlock() itself does not solve the issue.

PCIe

Pinned-memory CUDA copy benchmark:

RTX 4090 H2D 26.90 GB/s
RTX 3060 H2D  6.40 GB/s
P2P access: NO both directions

The secondary GPU PCIe link is behaving normally.

Prefill chunk size

4090-only with explicit --prefill 2048:

32K prefill 2814.4 tok/s

So the 2048-token chunk size does not explain the dual-GPU ~665 tok/s result.

Suspected cause

Current code applies the 8 GiB registration cap whenever multi_gpu is true:

const uint64_t pin_limit =
    (o.expert_cache_remote[0] > 0 || multi_gpu) ? (8ull << 30) : 0;

The surrounding source comment describes this as a Windows/WDDM workaround.

On Linux, this appears to leave most expert pages outside CUDA page-locked registration, causing the prefill path to spend most of its time in wait copy.

Local patch

I changed only the cap selection:

#ifdef _WIN32
const uint64_t pin_limit =
    (o.expert_cache_remote[0] > 0 || multi_gpu) ? (8ull << 30) : 0;
#else
const uint64_t pin_limit = 0;
#endif

Same machine, same model, same GPU order, and same K=2 split:

1K:  prefill  785.6 tok/s
4K:  prefill 1581.2 tok/s
32K: prefill 2109.8 tok/s

Full ~28K prompt:

stock:   42674 ms
patched: 13359 ms

Representative patched chunks:

GPU timeline 803 ms ... wait copy 35 ms (4.3%)
GPU timeline 822 ms ... wait copy 40 ms (4.9%)
GPU timeline 876 ms ... wait copy 35 ms (4.0%)
GPU timeline 827 ms ... wait copy 34 ms (4.1%)

So the Linux-only full-registration change improves this workload by about 3.2x and removes the multi-second expert-copy waits.

Possible fix

If the 8 GiB cap is specifically needed for Windows/WDDM, would it make sense to keep it under _WIN32 while leaving Linux unrestricted and relying on the existing registration fallback if full registration fails?

I can provide additional raw timings or test a proposed change.

Activity

  1. cha0yang commented on Sep 30, 2026

    @cha0yang
    cha0yang
    on Sep 30, 2026

    Independent confirmation on a 2x16 GB Blackwell box — plus one caveat that 36 GB of VRAM hides.

    Environment

    • Linux, engine 0.1.29 (stock + the patch below), [120] arch
    • Model: Qwen3.8-Flash-Next IQ3_XXS (unpruned, 512 experts), 76 GB GGUF
    • Context 262144, --kv k8v4, --mtp, --spec 4
    • CPU: Ryzen 7 9700X (full AVX-512), 64 GB DDR5-5600
    • 2x RTX 5060 Ti 16 GB, --layer-split auto -> CUDA0 layers 0-27, CUDA1 layers 28-47
    • --pcie-frac 0.00 (calibrated)

    The patch

    Same line as yours, in src/program/generate.cpp:

    #if defined(_WIN32)
    const uint64_t pin_limit = (o.expert_cache_remote[0] > 0 || multi_gpu) ? (8ull << 30) : 0;
#else
    const uint64_t pin_limit = 0;   /* #253: the 8 GiB cap is a WDDM workaround */
#endif

    The host side also needs memlock raised, since the arena is ~40 GB. Before / after:

    mlock failed (raise ulimit -l); cudaHostRegister limited to 8 GiB for CUDA1; 10 slices pinned (7 GiB)
expert arena: cudaHostRegister PORTABLE ok; MAP_HUGETLB unavailable (no hugetlb pool configured?); using 4 KB pages

    (/etc/security/limits.conf: zhuerpiao - memlock unlimited. It is silently required for the
    unpatched build too, but only shows up as mlock failed in the log.)

    Results

    Same weights, same context, same box; only the pin_limit differs.

    prompt before (8 GiB cap) after gain
    8,065 tok 865 tok/s 1,674 tok/s +94%
    32,062 tok 821 tok/s 2,141 tok/s +161%

    Decode is unchanged (~70 tok/s), expert-cache hit rate is unchanged (87.8%), and the outputs
    read the same to me. Your 32K number (2,110) and mine (2,141) agree to within 1.5% on
    completely different hardware — that looks like the cap was the whole story, not a quirk of
    your 3060/4090 split.

    (The 8K "after" number also carries a manual --prefill, because auto no longer starts at all
    once the arena is fully pinned. With --prefill 512, the smallest chunk that starts, it is already
    1,110 tok/s — so the pinning alone is worth at least +28% on 8K, and the rest is the chunk size.)

    Caveat: pinning the full arena costs VRAM

    cudaHostRegisterPortable maps the arena into the GPU's address space, and the driver
    reserves VRAM for those mappings. On 2x16 GB that is enough to break startup: the expert
    cache shrinks, --prefill auto still picks a 2048-token chunk, its buffers no longer fit,
    and the engine exits before it is ready.

    expert cache auto: 8.95 GiB free, 700 MiB reserved -> 3124 slots
layer split auto: CUDA1 ... 4.30 GiB for experts        <- ~7 GiB before the patch
strata serve: prefill: device buffers for a chunk of 2048 tokens do not fit
strata serve: the GPU has too little free VRAM for the prompt path: ... or read prompts in smaller chunks (--prefill 512)
RuntimeError: the engine exited before it was ready

    With 36 GB across two cards this is invisible, so it may be worth a note: on 16 GB cards the
    patch trades VRAM for pinning, and auto's estimate does not seem to account for the VRAM
    the mappings take. Workaround is to pin the chunk size instead of using auto — sweep on
    this box, 8K prompt:

    --prefill tok/s
    512 1,110
    1024 1,576
    1536 1,694 best here
    2048 — engine fails to start

    So the win is real and large, but on smaller cards it is a win with a manual --prefill,
    not the free lunch it looks like on a 24 GB card. Thanks — the write-up with the numbers made
    this easy to reproduce.

    (Separately filed #268 for an O(n^2) BPE loop in tools/strata_tokenizer.py, also CJK-only.)

  2. Niko1221 commented on Sep 30, 2026

    @Niko1221
    Niko1221
    on Sep 30, 2026
    Owner

    Confirmed - the 8 GiB cap is a Windows (WDDM) workaround that also applies on Linux. 0.1.31 applies it only under WDDM (Windows and WSL2), adds STRATA_ARENA_PIN_GIB (N GiB, or 0 for the whole arena), and steps the prompt chunk down instead of exiting when the whole arena is pinned. We're measuring it on a 2-GPU AMD box too.

  3. Niko1221
    added a commit that references this issue on Oct 1, 2026

    #253: the 8 GiB arena pin cap only under WDDM; STRATA_ARENA_PIN_GIB; …

    ed6c124

  4. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Niko1221
    on Oct 1, 2026
    Owner

    Fixed in 0.1.31: the 8 GiB cap applies only under WDDM (Windows, WSL2); Linux pins the whole arena again, and STRATA_ARENA_PIN_GIB=N sets a cap on any OS. The split's prompt chunk steps down instead of exiting when the stages' caches shrink. Thanks for the measurements!

  5. Niko1221
    closed this as completedon Oct 1, 2026
  6. MAHDI-AQ
    added 3 commits that reference this issue on Oct 4, 2026

    --split-skip-if-fits (opt-in): an auto split runs on the main card al…

    4035ceb

    --split-skip-if-fits (opt-in): an auto split runs on the main card al…

    ae25b96

    --split-skip-if-fits (opt-in): an auto split runs on the main card al…

    4db0f0c
  7. sergqwer
    added a commit that references this issue on Oct 4, 2026

    Start: register the expert arena per layer on a thread ahead of the r…

    59d97b3
  8. Niko1221
    added 2 commits that reference this issue on Oct 6, 2026

    Start: register the expert arena per layer on a thread ahead of the r…

    3aae3a2

    #253: the 8 GiB arena pin cap only under WDDM; STRATA_ARENA_PIN_GIB; …

    9f8fa90

  9. 3 remaining items

  10. anon761
    added a commit that references this issue on Oct 6, 2026

    Niko1221#253: the 8 GiB arena pin cap only under WDDM; STRATA_ARENA_P…

    22c69c9
  11. sskver
    added a commit that references this issue on Oct 7, 2026

    Start: register the expert arena per layer on a thread ahead of the r…

    97a2586
  12. k0valik
    added 10 commits that reference this issue on Oct 15, 2026

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): include/strata from ac922

    a23bcef

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/core from ac922

    1c72985

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/prefill from ac922

    69598f0

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/program from ac922

    1e26c8d

    replay(hub.lumenfield.work-lighttransport-Strata-glm53f): src/prefill from glm53f

    71e9fad

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): include/strata from ac922

    55297b8

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/core from ac922

    9075a5d

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/prefill from ac922

    e9c0181

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/program from ac922

    17dda1e

    replay(hub.lumenfield.work-lighttransport-Strata-glm53f): src/prefill from glm53f

    bb8e4ac
  13. chimpera
    added a commit that references this issue on Oct 17, 2026

    peer: create the peer device's context before the arena's whole-range…

    da593fd
  14. DarthCY-K
    added 2 commits that reference this issue on Oct 20, 2026

    peer: create the peer device's context before the arena's whole-range…

    5a7be59

    arena, layer split: each stage registers its own layers in its own CU…

    bdf85ce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions