generate: probe the real host->device bandwidth for pcie_frac

[pipeob0]

Independent of #43 — no stacking needed. It applies on plain main: it adds one function and replaces the pcie_frac default line, while #43 touches only the expert kernels run on notice a few lines below. Verified by cherry-picking this commit onto origin/main with no conflicts.

Why

pcie_frac's defaults (0.55 native / 0.2 canonical) were measured on a PCIe 4.0 x16 link (~26 GB/s).
The comment right above them says the copy engine's share has to shrink with the link — but nothing measured
the link, and a x8 card in a x8 slot is very common: it is what a 16 GB card with a big expert cache lands on
on most AM4/AM5 boards. There the share was being set to 2× what the wire can carry, so the GPU waits for DMA
that has not arrived. The only user-facing knob was --pcie-frac, i.e. a per-user guess.

What

Probe the host→device bandwidth once at startup, the way the engine actually uses it — DMA reads from
pinned host memory, 4 × 256 MiB, copy engine warmed — then scale the default by it:

o.pcie_frac = std::min(base, std::max(0.05, base * (bw / 26.0)));

Clamped so it can only shrink the share relative to the measured link and never below 0.05, and never above
the current default (a x16 machine keeps exactly today's behaviour). --pcie-frac still wins when given, and
if the probe cannot run at all the default is kept and the log says so.

On this box (RTX 5060 Ti 16 GB at PCIe 4.0 x8):

strata generate: PCIe probe: 14.1 GB/s host->device -> pcie_frac 0.30 (default 0.55)

Cost and risk

One-time at startup: ~1 GB of copies over a 14 GB/s link ≈ 75 ms, plus a 256 MiB pinned buffer and a 256 MiB
device allocation that are freed again before the arena is built. If either allocation fails, the probe
returns "could not run" and nothing changes. The probe runs after CUDA context setup, so it measures the same
path the expert fetches use.

Behaviour change is confined to machines whose link is slower than x16 PCIe 4.0 — the population the current
default is wrong for. It is also self-documenting: the log now states the measured bandwidth and both the
probed and the stock value, so a report can say whether the share was probed or guessed.

Not claimed: an end-to-end speed number for this. It moves the copy-engine share on x8 links; the round-time
effect on this machine was inside the noise of the other work in progress, so I am not attaching a number
that the method (±5% acceptance-driven round counts) cannot support.

[Niko1221]

Thanks. Two things before I take it: 0.1.14 moves the native packs' PCIe share from DMA to the copy kernel (#31: the host could hang inside cudaMemcpyAsync), so the share is now read by SMs over the link rather than the copy engine

• the probe still measures the right thing, but the scaling needs a new measurement. Could you run your x8 card on 0.1.14 with --pcie-frac 0.55 vs 0.30 (fixed prompts, ms/round as in kernels: AVX2 multi-token i-quant row kernels (an AVX2-only CPU decoded every i-quant row once per token) #43)? If the lower share wins there, I'll take the probe. (On x16 PCIe 4.0 the probe also lowers the share slightly - 25 vs 26 GB/s - so a threshold, e.g. only below 20 GB/s, would keep today's behaviour there.)

[pipeob0]

Ran it on 0.1.14 as asked — same box (RTX 5060 Ti 16 GB at PCIe 4.0 x8, Ryzen 7 5700X3D), fixed prompt, --spec 4, engine stopped so the link is not shared, 6 interleaved pairs (0.55, 0.30, 0.55, …), ms/round = decode ms / drafting rounds.

	--pcie-frac 0.55	--pcie-frac 0.30
ms/round (median of 5, cold run dropped)	60.53	50.16
range over the 5	54.68 - 61.75	48.58 - 51.57
per-pair delta (0.55 minus 0.30)	+10.23, +10.37, +6.10, +10.18, +9.38, +10.50 — all six pairs favour 0.30
gate/up (CPU pool) ms/round	9.52	13.85
down (CPU pool) ms/round	3.98	5.57
rows phases	22.0 GB/s	22.7 GB/s
CPU pool call ms/round	14.82	20.76
pcie experts line, share	141/256	77/256
distinct experts per layer read over PCIe	3.30	1.60

The lower share wins, by ~10.2 ms/round (~17%). The engine's own instrumentation shows why, and it is the mechanism you described: at 0.55 it streams 3.30 experts per layer over the wire, at 0.30 only 1.60. The CPU pool pays for that — gate/up +4.3, down +1.6, CPU pool call 14.8 → 20.8 ms/round — and the round is still 10.2 ms faster, so the GPU's wait on the link shrinks by more than the pool grows. The rows throughput barely moves (22.0 → 22.7 GB/s), i.e. the pool is not the constraint; the wire is.

The probe on this box now prints PCIe probe: 13.7 GB/s host->device -> pcie_frac 0.29, which lands on the value that just measured best.

On your threshold point — agreed, and I'd take it. Measured 25-26 GB/s on x16 PCIe 4.0 means today's default is the right one there, and scaling it down by 25/26 is a behaviour change for no measured gain. So:

if (bw >= 20.0) o.pcie_frac = base;   // x16-class link: keep exactly today's default
else o.pcie_frac = std::min(base, std::max(0.05, base * (bw / 26.0)));

Say the word and I'll push that as a second commit on this branch (the 20 GB/s cut-off sits between the 13.7 measured here and the 25-26 you cite, so both populations keep the behaviour that measures right for them).

Two caveats, stated plainly: 5 runs per arm after dropping the cold one, and this machine drifts ~6% slower under ~40 min of sustained AVX load — the per-pair interleaving is what I lean on (6/6 same sign), not the medians. The 17% is well above that drift, but I am not claiming a third decimal.

[Niko1221]

Thanks @pipeob0 - your 6/6 A/B on the x8 card was convincing. Merged in v0.1.20 with the follow-ups you offered: from 20 GB/s up the measured default stays (x16 links), the comment now says the share is read by the SMs over the link (the copy kernel since 0.1.14), and the probe runs for native packs only (the canonical pack's 0.2 was never measured against the link). --calibrate from 0.1.19 still overrides it with a measured value.