Repository navigation
PCIe probe can read ~3x low and pin pcie_frac far below the calibrated value #485
Description
probe_pcie_h2d_gbps() (src/program/generate.cpp) measures one 256 MiB x4 pinned burst after a single warmup and derives pcie_frac from it. On the same machine the same binary read the link at very different rates across starts, and a low sample pins pcie_frac well below what tools/calibrate.py picks.
Environment
- NVIDIA RTX A3000 12 GB Laptop (sm_86), PCIe x16 Gen4 (
current_link_width=16,16.0 GT/s) - Intel Core i7-12850HX (8 P + 8 E, AVX2, no AVX-512), 128 GB RAM
- Engine 0.1.34, native IQ3_XXS pack (39.97 GiB arena)
Measurements (same host, across starts)
PCIe probe: 18.5 GB/s host->device -> pcie_frac 0.39 (default 0.55)
PCIe probe: 6.9 GB/s host->device -> pcie_frac 0.15 (default 0.55)
PCIe probe: 5.8 GB/s host->device -> pcie_frac 0.12 (default 0.55)
tools/calibrate.py on the same host (decode tok/s, --spec-min-p fixed):
--pcie-frac |
0.0 | 0.2 | 0.35 | 0.55 | 0.75 |
|---|---|---|---|---|---|
| tok/s | 27.0 | 30.1 | 32.7 | 32.5 | 30.8 |
So the auto value the probe produced (0.12-0.15) is ~5 tok/s below the calibrated 0.35, and 0.35 also edges out the 0.55 default.
Likely cause
One 256 MiB x4 burst (1 GiB) after one warmup, on the default stream, with no retry/median and no exclusivity check: concurrent copy-engine work, ASPM/low link power state, or a context that has not ramped can depress the single sample, and the result is then used as a hard tuning input.
Suggested fix
Prime the link and take the median of several bursts (a host-side fix is on the fix/pcie-probe-median branch of my fork), and/or read the trained link width/speed (NVML nvmlDeviceGetCurrPcieLinkWidth / ...LinkSpeed) instead of one bandwidth sample. Note also the bw / 26 formula's discontinuity at 20 GB/s (19.9 -> 0.42, 20.0 -> 0.55), which a noisy sample can straddle.
A full write-up of this host (hugepages, probe, calibration) is in bench/results/2026-10-02-rtx-a3000-i7-12850hx/.
Activity
meelonjisoo-commits commentedon Oct 2, 2026
Data point from a different direction on this, in case it helps scope the fix.
Our box: RTX 3080 (gen4 x16) + Tesla V100-PCIE-32GB sitting in the chipset gen3 x4 slot. The probe reads the V100's link at 2.8 GB/s ->
pcie_frac 0.00, and that reading looks correct for the slot (gen3 x4 = 3.94 GB/s theoretical), not a mis-read.What the
0.00fallback costs, and what it does not (0.1.35, IQ3_XXS, 32K cold prompt; startup logs otherwise identical apart from the probe line):Solo, the fallback is enormous. The same V100 at 2.8-2.9 GB/s ->
0.00-> 305 t/s prefill. Moved to the CPU-attached x16 slot (probe 13.1 GB/s ->0.28), same binary and config: 729 t/s = 2.4x, with the only log difference being the probe and thepcie_fracderived from it.In a layer split, the fallback costs nothing. On a 3080+V100 split, forcing the second stage's policy to the values a wider slot would select changed nothing at all:
--pcie-fracforced (all stages)(vs 549.5 at the probe-derived
0.00.) With a balanced placement, that stage's cache covers ~100 % of its own layers' profile, so the PCIe share has almost nothing to route during prefill. The split's 2.9x loss there turned out to be the placementK, not the link — filed as #509.So the calibration matters most exactly where cache coverage is incomplete (a single card on a narrow link, or a stage whose layers do not fit its VRAM), and a low probe reading is not automatically the explanation for a slow split.
Two suggestions if either is useful:
pcie_fracit derived — that alone lets a user tell a mis-read from a genuinely narrow link.--pcie-fracis a single value that overrides every stage's own probe (pcie_givenshort-circuits the per-stage branch), so raising it for the narrow second card also lowers the fast card's setting. A per-stage override (or reusing the existing request-levelpcie_frackey for serve requests) would make that measurable.PCIe probe (#485): best of four timed bursts, a continuous share, the…
Engine 0.1.37: a silent engine is restarted (#481), Windows AMD count…
Niko1221 commentedon Oct 2, 2026
Fixed in 0.1.37: the probe times four 256 MiB bursts and takes the fastest (a disturbance only slows a copy), and the PCIe share now follows the measured speed smoothly below 20 GB/s (
base x min(1, GB/s / 20)) instead of jumping. Above 20 GB/s nothing changes. The probe line lists every burst. Thanks for the data!PCIe probe (#485): best of four timed bursts, a continuous share, the…
Engine 0.1.37: a silent engine is restarted (#481), Windows AMD count…
PCIe probe (Niko1221#485): best of four timed bursts, a continuous sh…
Engine 0.1.37: a silent engine is restarted (Niko1221#481), Windows A…
PCIe probe (Niko1221#485): best of four timed bursts, a continuous sh…
Engine 0.1.37: a silent engine is restarted (Niko1221#481), Windows A…