[GSD-12874] [BMG / Arc Pro B70] Is cross-root-port P2P expected? L0 dev-to-dev copy returns 0x70000003; kernel refuses at P2PDMA (host bridge not whitelisted) — and can the capability be made queryable?

Pre-submission Checklist

I have searched existing issues and discussions for similar questions
I have checked the documentation and FAQ / programmers-guide

Question Category

API / Development

Other Category

No response

Question

English isn't my first language, so I drafted this with AI help — but every result was measured by me on the hardware described. Full writeup +
reproducer: https://github.com/TSUMUGI-XE/b70-dual-tp2

Confound ruled out (control done). The test host normally runs pcie_acs_override=downstream,multifunction (for an unrelated VFIO passthrough),
which can make the kernel refuse peer P2P via pci_p2pdma_distance() and mimic this signature. I re-ran on a control boot with the override
removed: peer copy still returns 0x70000003, host-staged still passes — identical to baseline (control
log). The override is not the cause. On that boot the
kernel logged the reason: xe 0000:07:00.0: cannot be used for peer-to-peer DMA as the client and provider (0000:03:00.0) do not share an upstream bridge or whitelisted host bridge.

Summary

On a workstation with two Intel Arc Pro B70 (BMG), a Level-Zero direct device-to-device copy between the cards (single process, per-device L0
contexts) returns 0x70000003 (ZE_RESULT_ERROR_OUT_OF_DEVICE_MEMORY) instead of doing the peer copy. Clean error, not a hang. A host-staged
copy of the same buffers (dev0 → host → dev1) succeeds — so allocations/queues are fine, only the cross-device peer path fails.

This may well be expected behavior. I ruled out my own host config (control above); the kernel then refuses because the two cards are on separate
root ports with no shared PCIe switch and the host bridge isn't on pci_p2pdma_whitelist. So I'm not claiming a clear bug — I'm asking: (1) should
BMG + a consumer Intel host bridge route P2P through the root complex (whitelist gap to fix, or topology limit to document)? and (2) regardless, can
the unsupported case be reported as a queryable capability instead of OUT_OF_DEVICE_MEMORY, so frameworks stop probing-by-failure?

Environment

GPUs: 2× Intel Arc Pro B70 (BMG), separate PCIe root ports
Kernel: Linux 7.x, xe driver — reproduced on two builds (stock Ubuntu HWE 7.0.0-22 and a self-built 7.1-rc)
compute-runtime / Level-Zero: 1.15.x (NEO 26.x, IGC 2.34.x)
Single process, ONEAPI_DEVICE_SELECTOR=level_zero:*, ZES_ENABLE_SYSMAN=1; IOMMU/VT-d on, GPUs in separate IOMMU groups

Reproducer

~80-line single file: repro/b70_p2p_copy_probe.cpp. Build/run:

source /opt/intel/oneapi/setvars.sh                                                                                                                    
icpx -fsycl -O2 b70_p2p_copy_probe.cpp -lze_loader -o b70_p2p_copy_probe                                                                               
ONEAPI_DEVICE_SELECTOR=level_zero:* ZES_ENABLE_SYSMAN=1 ./b70_p2p_copy_probe

Output:

--- TEST 1: L0 direct copy dev0 -> dev1 ---                                                                                                            
  zeCommandListAppendMemoryCopy -> 0x70000003 (FAILED)                                                                                                 
--- TEST 2: host-staged copy dev0 -> host -> dev1 ---                                                                                                  
  checksum: PASS                                                                                                                                       
RESULT: l0_p2p=BROKEN(ze-error)  host_staged=WORKS

Negative controls (what is NOT the cause)

Host platform swap (different CPU/chipset, separate root ports, Gen4 x8/x8, separate IOMMU groups) → identical failure. PCIe bandwidth / root-port
layout / IOMMU grouping ruled out. Reproduces across two kernels.
pcie_acs_override removed control boot → still 0x70000003, kernel names the P2PDMA refusal (above). Override ruled out. Neither GPU exposes a
p2pdma sysfs node even with ACS off.

Related (not duplicates — different HW or failure mode)

intel/compute-runtime [GSD-12641] USM device allocation fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY when Level Zero context spans 2 discrete Arc A770 GPUs (regression from 25.40 to 26.09) #916 — same OUT_OF_DEVICE_MEMORY class on a 2-GPU Level-Zero context, but A770 (DG2) and urUSMDeviceAlloc, not a peer copy.
intel/compute-runtime [GSD-12696] [Xe2/BMG-G31] multi-rank MPI + Level Zero: resource_info.cpp:15 abort after upgrading CR 26.05 → 26.14 (Ubuntu 26.04, xe driver) #922 — same dual-B70 (BMG-G31) multi-GPU Level-Zero frontier, but a multi-process zeInit abort, not a single-process peer
copy.
vllm-project/vllm #41663 — the downstream TP=2 symptom on the same hardware.

Ask

Is BMG + a consumer Intel host bridge eligible for pci_p2pdma_whitelist (does the silicon route P2P through the root complex), or is
cross-root-port P2P genuinely unsupported here? Whitelist it, or document it?
Either way, can compute-runtime report the capability (a zeDeviceCanAccessPeer-style query) so frameworks choose host-staging deliberately
instead of probing-by-failure?
Is 0x70000003 (OUT_OF_DEVICE_MEMORY) the intended code for "peer access refused by the kernel P2PDMA layer", or does it mask the more specific
reason the kernel already names?

Happy to run additional diagnostics or driver builds — I have the two B70 set up and can iterate.

Additional Notes

No response

Do we have an answer from intel on this?

@mjsabby

Follow-up — and to @mjsabby's question above: no technical reply from Intel on my side yet (it's tracked internally as [GSD-12874]), but I have a concrete new result that I think answers my own Ask #1, so I'll add it here.

Short version: I forced the consumer host bridge into pci_p2pdma_whitelist to see if it's eligible. The kernel then allows the peer copy, but the root complex silently corrupts it — the command returns SUCCESS and the data is wrong.

Setup. Same 2× Arc Pro B70 (BMG), on separate CPU root ports, consumer Rocket Lake-S host bridge 8086:4c43. I built a kernel with that host bridge added to pci_p2pdma_whitelist[] and booted with intel_iommu=off (headless, GPUs otherwise idle).

Result.

The kernel P2PDMA gate now passes — the earlier cannot be used for peer-to-peer DMA ... do not share an upstream bridge or whitelisted host bridge refusal is gone, and a direct dev0→dev1 zeCommandListAppendMemoryCopy returns ZE_RESULT_SUCCESS (0x0) instead of 0x70000003.
But the destination buffer is fully wrong — checksum FAIL on 100% of elements (errs = 4194304 / 4194304). The same buffers copied host-staged (dev0 → host → dev1) verify correct (checksum PASS).
So with the whitelist forced open, the operation reports success while the data does not actually arrive — silent corruption.
The hardware itself is healthy: the PCIe link measures ~14 GB/s H2D/D2H (≈13.96 / 14.13), i.e. full Gen4 x8 per card, and host-staged copies are correct. The only thing that fails is the direct cross-root-port peer copy. (To be clear: that 14 GB/s is host↔device link bandwidth — not a peer-path figure; the peer path moved no valid data, so I'm making no claim about peer throughput.)

Interpretation. Forcing the whitelist makes the kernel allow the mapping, but the RKL-S root complex doesn't correctly route the cross-root-port peer write. The L0/xe side does issue the copy (no 45/39, no capability rejection) — the failure is purely in the peer data path through the root complex. This is consistent with pci_p2pdma_whitelist deliberately excluding consumer client host bridges.

For contrast (not measured by me, just to scope it): Intel's own Platform Evaluation Guide for Multi-ARC notes no expected P2P restriction on AMD hosts, and there are community reports of multi-Arc P2P working on AMD EPYC. So this looks specific to consumer Intel single-IIO host bridges, not Arc P2P in general — Xeon/validated platforms are a separate case I'm not commenting on.

Two practical questions:

Is cross-root-port P2P simply unsupported on consumer (RKL-S and similar) single-IIO hosts — i.e. a topology limit worth documenting for Arc Pro multi-GPU, with an external PCIe switch (both GPUs under one upstream bridge) being the only path on these platforms?
Independent of (1): since forcing the path yields ZE_RESULT_SUCCESS + corrupt data rather than a clean refusal, can the stack detect/refuse this case (a queryable peer-access capability, or a hard error) so a user can't unknowingly run a workload on a path that returns wrong results? Silent corruption is worse than the original OUT_OF_DEVICE_MEMORY.

I can share the one-line whitelist patch, the ~80-line probe, and the sanitized boot log if any of that is useful.

(English isn't my first language, so I drafted this with AI help — but every result above was measured by me on the hardware described.)

@TSUMUGI-XE

Thanks @TSUMUGI-XE. I commented on your repo asking a similar question. Is it worth checking out consumer AMD platform and seeing if it works.

You also specifically call out cross root. Have you tried in a single root and it DOES work or also doesn't work? In other words is there any kernel driver work needed here? I will also clone this repo and ask Claude.

I think it will be tough for Intel to say or reverse course on the fact they themselves probably didn't allow list their client skus for p2p dma.

That said it is concerning that your est did not pass when you have server class skus.

@mjsabby

Thanks @mjsabby — good questions.

Single-root: haven't been able to test it yet, and that's the key gap. Both B70s sit on separate CPU root ports, so I can't make them single-root without changing topology. Linux pci_p2pdma already allows P2P unconditionally when two devices share an upstream PCIe switch — no driver work needed there. The refusal is specific to cross-root-port (no shared switch, host bridge not whitelisted). So I'd expect single-root / behind-a-switch to work at the kernel layer; the open question is whether the data then routes. I'm getting a PEX88096 switch board to put both cards under one upstream bridge and will post ze_peer numbers.

"Kernel driver work needed?" For the shared-switch case, no — the path is already there. For cross-root-port on this consumer host, forcing it (whitelisting the host bridge) produces silent corruption, so the kernel's refusal is correct — it's not a missing driver, it's the root complex not routing peer TLPs.

On the SKU point: the GPUs aren't the problem — the B70s issue the copy and host-staged works. The failing piece is the consumer host's root complex (Rocket Lake-S, 8086:4c43), which isn't in pci_p2pdma_whitelist. On a whitelisted/server host (the list has Skylake-E / Xeon Scalable) or behind a switch, the same GPUs should route P2P.

AMD: agreed, worth checking — I'm on an Intel host so can't test it myself, but Intel's own Multi-ARC guide notes no expected P2P restriction on AMD hosts, and there are community reports of multi-Arc P2P on EPYC. So this looks like a consumer-Intel-root-complex thing, not an Arc-wide limit.

Version caveat (in fairness): my data is from kernel 7.0.0-22 — I stay on 7.0 because 7.1 regresses my Wi-Fi — and I haven't tested released 7.1 or the latest UMF myself. I know it's been reported that the dual-B70 TP=2 path needs Linux ≥7.1; worth noting the one clean working setup I've seen is a Threadripper HEDT with multiple root complexes (TP=2 works across different RCs there, but still crashes on the same RC). The 7.1 xe changelog shows no P2P/multi-GPU changes, so I don't expect a kernel bump alone to flip the cross-root-port verdict on a consumer single-root-complex host — but I'll confirm that with the switch rather than assume it.

Separately — I have a host-staged TP=2 workaround (routes the all-reduce through CPU/gloo so it never touches GPU P2P; works regardless of root-complex topology) plus the ze_peer/checksum + pci_p2pdma distance/whitelist probe scripts I used to characterize all this. Perf is modest but real: on a dense 27B it ran ~1.2x single-stream vs a single GPU and scaled with batch; on an MoE it was capacity-only (single-stream was actually slower than one card). Happy to share both if useful — heads up, I'm traveling for a bit, so I'll post the cleaned-up patch + scripts once I'm back.

Will report back once the switch is in.

Yes, your repo was very useful to me for investigations, so please continue. The goal is to not through CPU/gloo for me, but yes your experiment proves it is somewhat usable, although I suspect Intel probably wants to get it fixed.

BTW, you don't need EPYC right? All zen chipsets are allow listed.

Another quesiton to ponder, if Windows supports it, given Intel controls the drivers there more strongly than having to wait for upstream Kernel.

Quick follow-up on the AMD question — I went and checked the actual Linux pci_p2pdma whitelist (drivers/pci/p2pdma.c), and it's asymmetric in a way that's directly relevant:

AMD: blanket-enabled — if (cpu_vendor == AMD && family >= 0x17) ⇒ any Zen-or-newer host is trusted for p2pdma, no per-device list.
Intel: an explicit device-ID allowlist of Xeon E5/E7 + Skylake-E only. Consumer parts (mine is Rocket Lake-S, 8086:4c43) aren't in it, so the distance check returns -1 and P2P is refused up front.

So on a Zen+ box your two B70s should clear the kernel gate that mine fails — and the kernel only blanket-trusts AMD because the fabric actually routes peer TLPs (vs. my consumer Intel RC, which even when forced open returns the buffer as all-1s = a clean Unsupported-Request refusal, not a transfer). That makes consumer AMD a genuinely worthwhile test. If you try it, I'd love to know whether p2p_access flips to 1 and a real cross-card copy verifies (not just returns success).

I'm taking the other branch: a PEX (PLX) switch on my Intel box to put both cards under one shared upstream bridge and bypass the root complex entirely — but it's ~2-3 weeks out. So let's split it: you on AMD, me on the switch, and compare notes. Whichever lands first tells us a lot.

@TSUMUGI-XE

@TSUMUGI-XE yes I'm going to try with AMD host, your repo's code should be a quick minimal repro to test right?

Quick answers:

Repro: yes — it's a single ~85-line SYCL/Level-Zero file (repro/b70_p2p_copy_probe.cpp), build icpx -fsycl -O2 ... -lze_loader, runs in seconds. On AMD the key check is that the copy verifies (contents match), not just that it returns success — on my consumer Intel box the call "succeeds" but the destination comes back all-1s (a PCIe Unsupported-Request, i.e. the RC didn't actually move the bytes). So the thing to watch: does zeDeviceCanAccessPeer / p2p_access report 1, and does the copied buffer match the source?

Windows: I don't think it escapes the wall on consumer Intel. The all-1s is a hardware Unsupported-Request from the root complex — that's below the OS, so a Windows driver (even with Intel's tighter control there) hits the same silicon that won't route peer TLPs across root ports. Where the RC does route (AMD Zen / Xeon / behind a PCIe switch), Windows P2P should be fine — but Linux works there too. So Windows is a driver-exposure question, not a fix for the consumer-Intel-RC routing limit.

Looking forward to your AMD numbers — that's the cleanest test of the whole thing.

AM5 works for P2P, I guess the best setup for developers not looking to fork $$$ on NVIDIA is use AMD CPUs and Intel GPUs. What a world!

Does putting both B70 under a single PCIe switch route cross-card P2P on a consumer Intel host?

Following up. On my consumer Intel host (Rocket Lake-S, 8086:4c43) cross-card B70 P2P doesn't work: the kernel refuses it (0x70000003 / pci_p2pdma_distance = −1 — consumer Intel RCs aren't in the whitelist), and forcing the whitelist open returns ZE_RESULT_SUCCESS but corrupt data. The same two cards verify clean cross-card P2P on an AMD host (third-party, checksum PASS both directions) — so this is the consumer Intel root complex, not the cards.

My plan is a PCIe switch to put both cards under one shared upstream bridge, so peer TLPs route within the switch and never reach the root complex (per the kernel p2pdma docs: two devices under a common switch → distance 4 → PCI_P2PDMA_MAP_BUS_ADDR).

Question: is that the expected-to-work topology for B70↔B70 P2P on a consumer Intel host — and is there any ACS caveat I should clear on the switch (e.g. a downstream-port ACS redirect forcing traffic back through the host bridge)? A confirmation would de-risk it before I test the hardware — and once both cards sit under the switch, will zeDeviceCanAccessPeer report capable and the copy route correctly? A one-line note of the supported topology in the docs would also save the next person.

Switch result: B70↔B70 P2P verifies clean on a consumer Intel host — but only with ACS redirect off on the switch

Following up on my own switch question with hardware in hand. I re-homed both B70 under a single Broadcom/LSI PEX880xx (Gen4) switch — both cards on downstream ports of one shared upstream bridge — on the same consumer Intel host (Rocket Lake-S, 8086:4c43). Upstream to the root complex is Gen4 x16.

lspci -tv (both cards under one upstream 01:00.0):

-01.0-[01-18]----00.0-[02-18]--+- … -[0e]----00.0  Battlemage G31
                               |- … -[12]----00.0  Battlemage G31
                               \-1c.0-[18]----00.0  Broadcom / LSI PEX880xx PCIe Gen 4 Switch

The switch topology alone was not sufficient — there's a second gate: ACS redirect on the switch's own downstream ports. Both states, same cards, same box:

ACS redirect enabled (the switch's power-on default):

cross-card copy runs at 10.89 GB/s — so it's on a fast/direct-looking path, not the ~3 GB/s host-staged one …
… but the data is wrong: checksum FAIL, buffer returns all-0xFFFFFFFF — the exact same silent corruption as the bare consumer RC. The switch has ReqRedir+/CmpltRedir+, so peer TLPs are redirected up to the root complex and corrupted there.

ACS redirect cleared on the switch downstream ports (setpci ECAP_ACS Control = 0):

checksum PASS, clean both directions (write-then-read, 0xAB over 256 MiB, 268435456 / 268435456 bytes correct).
dmesg clean — no 0x70000003, no engine reset.
peer bandwidth (ze_peer, both cards under the switch, 256 MB):

                              1 MB    16 MB   256 MB
unidirectional write (each)   24.1    28.2    28.5   GB/s   (one direction at a time; d0->d1 and d1->d0 symmetric)
bidirectional  write          46.9    55.8    56.5   GB/s   (both directions at once; ~2x, PCIe full-duplex)
bidirectional  read           41.2    49.4    50.0   GB/s
host-staged fallback on this box: ~3 GB/s

i.e. ~28.5 GB/s per direction, and ~56 GB/s aggregate with both directions running at once (Gen4 x16 upstream) — the same order as the AM5 third-party numbers earlier in the thread (~28.6 GB/s/dir there on Gen5 x8), and ~9× the host-staged path this box falls back to when P2P is refused. The bidirectional figure is the one that matters for TP=2 all-reduce.

To be clear, that's the switch's physical ACS, not the kernel pcie_acs_override= knob (which only changes IOMMU grouping, not how the switch routes TLPs). So the working recipe on a consumer Intel host is: both cards under a common PCIe switch and ACS redirect disabled on the switch downstream ports. With either one missing you get ZE_RESULT_SUCCESS + silently corrupt data.

This matches the AMD third-party datapoint from earlier in the thread (clean on Zen, corrupt on bare consumer Intel): the consumer RC is what mangles peer writes, and keeping the TLPs switch-local (ACS-off) sidesteps it entirely.

Follow-through: a real TP=2 all-reduce workload runs cleanly on this path. With the switch-local ACS-off peer path confirmed clean, I ran an actual tensor-parallel workload over it — vLLM serving a 27B model with tensor-parallel-size=2 across the two B70 (XCCL / oneCCL peer all-reduce). It initializes in ~90 s, serves, and a concurrency sweep completes 100% with clean output and no hang or engine reset. With CUDA-graph decode it sustains real serving throughput — up to ~919 tok/s aggregate at N=64, roughly 2.5× the same model on a single card:

concurrency N    aggregate tok/s   (27B, TP=2 peer all-reduce, CUDA-graph decode)
  1                42
  8               237
 32              655
 64              919

So beyond the raw copy benchmark, the clean switch-local peer path carries a real collective serving workload correctly — which is exactly why the routing question matters: with the corruption present (ACS-on), that same collective would be silently operating on mangled peer data.

(Orthogonal aside, in case it saves someone time: getting the collective to start on this many-interface host also required pinning oneCCL's OFI/TCP bootstrap to loopback — FI_TCP_IFACE=lo — a oneCCL/libfabric interface-selection quirk unrelated to the peer-TLP routing question here.)

Which brings the original ask back, sharper: at every stage before the fix, zeCommandListAppendMemoryCopy returned ZE_RESULT_SUCCESS on data that was silently wrong — nothing in the API distinguished "peer copy landed" from "peer copy got redirected and corrupted." A queryable capability — does this device→device path actually route peer TLPs on this platform/topology? — would let a runtime (or oneCCL) refuse or fall back instead of returning success on garbage. Is that detectable today from L0/the driver, or is silent-corruption-on-unsupported-topology the current expected behavior?

(Happy to share the repro + both checksum logs if useful.)

@TSUMUGI-XE

@TSUMUGI-XE thank you for the detailed investigation, and apologies for the delayed response.

Your results indicate that P2P works when both B70 GPUs share a PCIe switch and ACS redirect remains disabled. However, with ACS redirect enabled, pci_p2pdma_distance() should reject the path through the non-whitelisted host bridge.

Could you run p2p-distance with ACS redirect enabled on the external switch and share the full output? Please also confirm that the kernel used for this test did not contain the host-bridge whitelist modification or another whitelist bypass.

I agree that ZE_RESULT_ERROR_OUT_OF_DEVICE_MEMORY is misleading when peer access cannot be established. We are investigating this error path so that an unsupported P2P configuration can be reported more accurately.

@ola2308

@ola2308 Thanks — I ran p2p-distance in both ACS states on the same box. Your expectation is
exactly what it reports, and the result does not depend on any whitelist modification.

Both B70 sit under one PLX switch, so their common upstream bridge is the switch (0000:0a:00.0),
not the root complex.

ACS redirect ENABLED (ECAP_ACS+0x6.w=000c on every ReqRedir-capable bridge):

ACS redirect is enabled on: 0000:0d:01.0, 0000:0b:00.0, 0000:11:01.0, 0000:0b:10.0
  -- P2P traffic is routed back through the host bridge/RC.
  host bridge 0000:00:00.0 (8086:4c43): NOT whitelisted

map_type = PCI_P2PDMA_MAP_NOT_SUPPORTED
distance = -1   (internal topology hop-count: 8)

ACS redirect DISABLED (switch downstream ports cleared — the configuration my ze_peer PASS was
measured in):

Direct PCIe P2P is possible: no ACS redirection on the path.

map_type = PCI_P2PDMA_MAP_BUS_ADDR
distance = 8

So pci_p2pdma_distance() behaves as you described: with redirect on, the path falls back to the
host bridge, 8086:4c43 is not whitelisted, and the verdict is NOT_SUPPORTED. P2P is only possible
because the traffic stays switch-local.

On the whitelist question: p2p-distance implements the stock pci_p2pdma_whitelist[] (no
8086:4c43 entry) and prints host bridge 0000:00:00.0 (8086:4c43): NOT whitelisted in the
redirect-on run — so the BUS_ADDR verdict above comes from the shared switch bridge, not from a
whitelist entry. In the interest of full disclosure: I do have a local kernel build that adds
8086:4c43 to that table, from an earlier experiment testing whether the consumer RC could route
peer TLPs at all (it could not — the data came back silently corrupted, which is what led me to the
switch). It is irrelevant to the switch-local path, but I'm happy to re-run both measurements on a
stock distro kernel if you want that removed as a variable.

Thanks for looking at the ZE_RESULT_ERROR_OUT_OF_DEVICE_MEMORY path — a distinct "peer access not
supported in this topology" error would have saved a lot of guessing here.

Full p2p-distance run (ACS on → measure → restore → verify)

=== 対象 bridge (ACS ReqRedir capable) ===
  00:01.0
  00:06.0
  00:1b.0
  00:1c.4
  00:1c.7
  02:00.0
  02:04.0
  02:08.0
  02:0c.0
  04:10.0
  07:00.0
  07:10.0
  0b:00.0
  0b:10.0
  0d:01.0
  0d:02.0
  11:01.0
  11:02.0
  15:14.0
  15:15.0
=== BEFORE ACSCtl (plx-acs-off 後=0000 のはず) ===
  00:01.0 ECAP_ACS+0x6=0000
  00:06.0 ECAP_ACS+0x6=0000
  00:1b.0 ECAP_ACS+0x6=0000
  00:1c.4 ECAP_ACS+0x6=0000
  00:1c.7 ECAP_ACS+0x6=0000
  02:00.0 ECAP_ACS+0x6=001d
  02:04.0 ECAP_ACS+0x6=001d
  02:08.0 ECAP_ACS+0x6=0000
  02:0c.0 ECAP_ACS+0x6=001d
  04:10.0 ECAP_ACS+0x6=0000
  07:00.0 ECAP_ACS+0x6=0000
  07:10.0 ECAP_ACS+0x6=0000
  0b:00.0 ECAP_ACS+0x6=0000
  0b:10.0 ECAP_ACS+0x6=0000
  0d:01.0 ECAP_ACS+0x6=0000
  0d:02.0 ECAP_ACS+0x6=001d
  11:01.0 ECAP_ACS+0x6=0000
  11:02.0 ECAP_ACS+0x6=001d
  15:14.0 ECAP_ACS+0x6=0000
  15:15.0 ECAP_ACS+0x6=0000
=== ACS redirect ON (RR+CR=0x000c) ===
  acs-on: 00:01.0 -> 000c
  acs-on: 00:06.0 -> 000c
  acs-on: 00:1b.0 -> 000c
  acs-on: 00:1c.4 -> 000c
  acs-on: 00:1c.7 -> 000c
  acs-on: 02:00.0 -> 000c
  acs-on: 02:04.0 -> 000c
  acs-on: 02:08.0 -> 000c
  acs-on: 02:0c.0 -> 000c
  acs-on: 04:10.0 -> 000c
  acs-on: 07:00.0 -> 000c
  acs-on: 07:10.0 -> 000c
  acs-on: 0b:00.0 -> 000c
  acs-on: 0b:10.0 -> 000c
  acs-on: 0d:01.0 -> 000c
  acs-on: 0d:02.0 -> 000c
  acs-on: 11:01.0 -> 000c
  acs-on: 11:02.0 -> 000c
  acs-on: 15:14.0 -> 000c
  acs-on: 15:15.0 -> 000c
=== ★p2p_distance WITH redirect ON (期待: NOT_SUPPORTED) ===
provider: 0000:0e:00.0 -> 0000:0e:00.0
client:   0000:12:00.0 -> 0000:12:00.0

PCIe path:
0000:0b:00.0 (1000:c010)  [ACS redirect ON: blocks direct P2P]
  +-0000:0c:00.0 (8086:e2ff)
    +-0000:0d:01.0 (8086:e2f0)  [ACS redirect ON: blocks direct P2P]
      +-0000:0e:00.0 (8086:e223)
        +-0000:0a:00.0 (1000:c010)  <-- common upstream bridge
          +-0000:12:00.0 (8086:e223)
            +-0000:11:01.0 (8086:e2f0)  [ACS redirect ON: blocks direct P2P]
              +-0000:10:00.0 (8086:e2ff)
                +-0000:0b:10.0 (1000:c010)  [ACS redirect ON: blocks direct P2P]

ACS redirect is enabled on: 0000:0d:01.0, 0000:0b:00.0, 0000:11:01.0, 0000:0b:10.0 -- P2P traffic is routed back through the host bridge/RC.
  (kernel hint: pci=disable_acs_redir=0000:0d:01.0;0000:0b:00.0;0000:11:01.0;0000:0b:10.0)
  host bridge 0000:00:00.0 (8086:4c43): NOT whitelisted

map_type = PCI_P2PDMA_MAP_NOT_SUPPORTED
distance = -1  (as returned by pci_p2pdma_distance_many; the internal topology hop-count is discarded and -1 is returned whenever any client is PCI_P2PDMA_MAP_NOT_SUPPORTED)
  (internal topology hop-count, not exposed by the kernel API: 8)
=== RESTORE (plx-acs-off.sh) ===
  acs-off: 00:01.0
  acs-off: 00:06.0
  acs-off: 00:1b.0
  acs-off: 00:1c.4
  acs-off: 00:1c.7
  acs-off: 02:00.0
  acs-off: 02:04.0
  acs-off: 02:08.0
  acs-off: 02:0c.0
  acs-off: 04:10.0
  acs-off: 07:00.0
  acs-off: 07:10.0
  acs-off: 0b:00.0
  acs-off: 0b:10.0
  acs-off: 0d:01.0
  acs-off: 0d:02.0
  acs-off: 11:01.0
  acs-off: 11:02.0
  acs-off: 15:14.0
  acs-off: 15:15.0
plx-acs-off: cleared 20 bridges (2026-08-13 08:59:11)
=== VERIFY 復元後 (期待: BUS_ADDR) ===
provider: 0000:0e:00.0 -> 0000:0e:00.0
client:   0000:12:00.0 -> 0000:12:00.0

PCIe path:
0000:0b:00.0 (1000:c010)
  +-0000:0c:00.0 (8086:e2ff)
    +-0000:0d:01.0 (8086:e2f0)
      +-0000:0e:00.0 (8086:e223)
        +-0000:0a:00.0 (1000:c010)  <-- common upstream bridge
          +-0000:12:00.0 (8086:e223)
            +-0000:11:01.0 (8086:e2f0)
              +-0000:10:00.0 (8086:e2ff)
                +-0000:0b:10.0 (1000:c010)

Direct PCIe P2P is possible: no ACS redirection on the path.

map_type = PCI_P2PDMA_MAP_BUS_ADDR
distance = 8  (as returned by pci_p2pdma_distance_many)
=== 完了・redirect OFF に復元済み ===

@TSUMUGI-XE

@TSUMUGI-XE What we need to verify is that we don't get the silent corruption with a stock kernel and the local switch in ACS redirect mode. Instead p2pdma should be refused. The theory is that you see the silent corruption in the ACS redirect enabled case because even though you have a local switch, ACS redirects traffic through the host bridge which doesn't support it. So essentially you see the same error as without the switch.

@thellstrom-os

@thellstrom-os Done — I re-ran it on a stock kernel, and your expectation holds.

Booted stock 7.0.0-22-generic (no whitelist patch), re-enabled ACS redirect at runtime with
setpci on the switch bridges, and ran both states on that one boot:

	ACS redirect ON	ACS redirect OFF (control)
`pci_p2pdma_distance`	`PCI_P2PDMA_MAP_NOT_SUPPORTED`	`PCI_P2PDMA_MAP_BUS_ADDR`, distance 8
`zeDeviceCanAccessPeer`	0→1 = 0, 1→0 = 0	0→1 = 1, 1→0 = 1
256 MB dev-to-dev copy	refused — `ZE_RESULT_ERROR_OUT_OF_DEVICE_MEMORY`	28.00 GB/s
checksum	(nothing transferred)	PASS, 268435456 / 268435456 bytes, 0 corrupt

No silent corruption in the redirect case — it is a clean refusal, which matches your reading that
ACS sends the traffic back through the host bridge.

Two notes on my earlier posts, since both could mislead:

ACS here is not disabled by a kernel parameter. pcie_acs_override= is on my cmdline but is a
no-op on this build; a systemd unit clears the redirect with setpci after boot. So the redirect
is active during early boot, which is where the xe refusal message comes from.
The NOT_SUPPORTED verdict I posted on Aug 13 came from p2p-distance, which models the stock
pci_p2pdma_whitelist[] in userspace — not from my patched kernel's own decision.

That answers both halves of my original question: the refusal is expected, and
zeDeviceCanAccessPeer does report it. Happy for this to be closed.

@TSUMUGI-XE

@TSUMUGI-XE thank you for the thorough testing and for confirming the behavior on stock kernel. We're glad that the silent corruption isn't present there and that fallback is working correctly as expected.

Since your original question have been answered and you're good with the current state, I'll go ahead and close this issue. Please feel free to reach out or open a new issue if you have any further questions or run into any other problem in future.

Navigation Menu

Repository navigation

[GSD-12874] [BMG / Arc Pro B70] Is cross-root-port P2P expected? L0 dev-to-dev copy returns 0x70000003; kernel refuses at P2PDMA (host bridge not whitelisted) — and can the capability be made queryable? #935

Pre-submission Checklist

Question Category

Other Category

Question

Summary

Environment

Reproducer

Negative controls (what is NOT the cause)

Related (not duplicates — different HW or failure mode)

Ask

Additional Notes

Does putting both B70 under a single PCIe switch route cross-card P2P on a consumer Intel host?

Switch result: B70↔B70 P2P verifies clean on a consumer Intel host — but only with ACS redirect off on the switch

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[GSD-12874] [BMG / Arc Pro B70] Is cross-root-port P2P expected? L0 dev-to-dev copy returns 0x70000003; kernel refuses at P2PDMA (host bridge not whitelisted) — and can the capability be made queryable? #935

Description

Pre-submission Checklist

Question Category

Other Category

Question

Summary

Environment

Reproducer

Negative controls (what is NOT the cause)

Related (not duplicates — different HW or failure mode)

Ask

Additional Notes

Activity

mjsabby commented on Jun 19, 2026

TSUMUGI-XE commented on Jun 19, 2026

mjsabby commented on Jun 19, 2026

TSUMUGI-XE commented on Jun 19, 2026

mjsabby commented on Jun 19, 2026

mjsabby commented on Jun 19, 2026

TSUMUGI-XE commented on Jun 19, 2026

mjsabby commented on Jun 19, 2026

TSUMUGI-XE commented on Jun 19, 2026

mjsabby commented on Jun 28, 2026

TSUMUGI-XE commented on Jun 28, 2026

Does putting both B70 under a single PCIe switch route cross-card P2P on a consumer Intel host?

TSUMUGI-XE commented on Jul 6, 2026

Switch result: B70↔B70 P2P verifies clean on a consumer Intel host — but only with ACS redirect off on the switch

ola2308 commented on Aug 12, 2026

TSUMUGI-XE commented on Aug 13, 2026

thellstrom-os commented on Aug 18, 2026

TSUMUGI-XE commented on Aug 18, 2026

ola2308 commented on Aug 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions