GPUDirect RDMA (dmabuf) READ from GPU BAR1 fails on Intel Arrow Lake-S / Z890 - writes work, reads fail with CQE error, no PCIe error anywhere; Z890 missing from chipset_info.c #1253

Summary

On Intel Arrow Lake-S (Core Ultra 9 285K, ASUS Pro WS Z890-ACE SE), any
NIC-initiated DMA READ of GPU BAR1 memory registered via the dmabuf path fails
deterministically. DMA WRITEs to the same BAR1 addresses work at full line rate.
The identical software stack on a Threadripper PRO node works in both directions
at 180 Gb/s.

We have reduced this to a single-node loopback reproducer (NIC port 1 reads GPU
memory served by NIC port 0 on the same machine) and traced it to the PCIe
transport between two CPU root ports. The kernel-side dmabuf mapping is provably
correct (verified with bpftrace: the NIC receives the exact BAR1 bus address).
No AER/DevSta error is logged on GPU, NIC, or either root port when the read
fails - the failure is silent at PCIe level and surfaces only as an RDMA
completion error.

Notably, chipset_info.c has entries with workaround flags for consumer Intel
platforms Z690 (Alder Lake) and Z790 (Raptor Lake), but no entry for Z890/Arrow
Lake. We tested the Z790-class rBAR workaround (BAR1 resized to 256M) - it does
not help, so this appears to be a different, new failure class.

System details

GPU BAR1: 32 GB resizable BAR at 0xa0_0000_0000 (above 4G).
nvidia-peermem not loadable (inbox kernel lacks ib_register_peer_memory_client);
all tests use the upstream dmabuf path (ib_umem_dmabuf).

Reproducer (single node, deterministic)

# Server: GPU memory via dmabuf on port 0
ib_read_bw -d rocep3s0f0 -x 3 --report_gbits --use_cuda=0 --use_cuda_dmabuf -D 3 -p 20001
# Client: reads from server's GPU memory via port 1 (loopback through switch)
ib_read_bw -d rocep3s0f1 -x 3 --report_gbits -D 3 -p 20001 10.99.8.1

Result on Arrow Lake, 100% of runs:

Failed status-remote operation error (11): wr_id 0 syndrom 0x89

Same command with ib_write_bw (NIC writes into GPU memory): ~59 Gb/s, no error.
Same commands with host RAM instead of GPU memory: 97.6 Gb/s, no error.
Same commands on the Threadripper PRO control node: 180 Gb/s, no error.

Evidence collected

1. Both verb directions fail identically when the NIC must READ GPU memory:

   • ib_read_bw, GPU memory on server (NIC = responder): client sees
     remote operation error, syndrome 0x89.
   • ib_write_bw, GPU memory on client (NIC = requester reading its local
     source buffer from VRAM): local protection error, syndrome 0x51.
   • Interpretation: same failure (NIC DMA read of BAR1), seen from the two
     verb roles. Writes into BAR1 always succeed.

2. The dmabuf mapping handed to the NIC is correct (bpftrace on
   dma_buf_map_attachment return): sg0: dma=0xa000600000, len=131072

   • exactly BAR1 base + allocation offset. reg_user_mr_dmabuf,
     ib_umem_dmabuf_map_pages, mlx5r_umr_update_mr_pas all return 0.

3. No PCIe error is recorded anywhere at failure time. GPU AER (UESta/CESta),
   NIC AER + Device Status (cleared before test), and both root ports
   (00:06.0/00:06.3) stay clean: no UR, no CA, no Completion Timeout, no
   Malformed TLP. The NIC appears to receive a formally valid completion and
   still fails the operation.

4. Exhaustive exclusion matrix (each tested, failure unchanged):

   • message size 64 B .. 128 KB (even a single 64-byte read fails)
   • QP types RC, DC, XRC; rdma_cm and non-CM connection setup
   • IOMMU off (kernel cmdline), ACS disabled on all root ports
   • NoSnoop / RelaxedOrdering toggled off on NIC and GPU (setpci)
   • NVreg_RegistryDwords="ForceP2P=0x11;PeerMappingOverride=1" (clean reboot)
   • BAR1 resized 32G -> 256M (sysfs resource1_resize) - rules out the
     Z790-class HAS_RESIZABLE_BAR_ISSUE workaround as a fix
   • cold vs warm GPU (8.5 GB allocated + active CUDA context)
   • two independent perftest builds (23.04 and git HEAD)
   • NIC FW config identical to working node (mstconfig byte-identical:
     MAX_ACC_OUT_READ=44, P2P_ORDERING_MODE=DEVICE_DEFAULT, etc.)
   • NIC FW health: devlink health clean, no fwtrace faults, AER clean

5. Sporadic NVRM allocation errors on the failing platform only (possibly
   related, not 1:1 correlated with the CQE failure):
   NVRM: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal @ mem_desc.c:1359 via nv_gpu_ops.c:4968
   (NV01_MEMORY_SYSTEM alloc). Never seen on the Threadripper control node.

6. chipset_info.c has no Arrow Lake / Z890 entry. Z690 and Z790 entries
   exist and set PDB_PROP_CL_HAS_RESIZABLE_BAR_ISSUE. Since the failure here
   is independent of BAR1 size, Z890 appears to need a different workaround
   class (or an erratum-level fix from Intel).

7. Reproduced on a second, independent Arrow Lake system ("vega"): Core
   Ultra 7 265K host, RTX PRO 4000 Blackwell, two single-port ConnectX-5 Ex
   (separate cards), running as a Q35 VM with GPU+NIC passthrough (VFIO) on the
   ARL host. Same loopback test: syndrome 0x89 on the GDR read; host-RAM read
   on the same path: 97.5 Gb/s clean. Different CPU SKU, different card
   layout, virtualized: identical failure signature. This makes a
   board/SKU-specific defect unlikely and points at the platform (root
   complex) itself.

Working hypothesis

The Arrow Lake-S root complex does not correctly route NIC-initiated non-posted
(read) TLPs between two CPU root ports when the target is a peer BAR above 4G,
while posted (write) TLPs on the same path route fine. The returned completion
is formally valid at PCIe level (no AER anywhere) but the data/completion
semantics are broken, causing the CX-5 to report a fatal MR/operation error.

Prior art: linux-rdma/perftest#289 shows the same signature (syndrome 0x51,
one direction fails) on a Z690 consumer platform; @drossetti suggested
ACS P2P redirection as the usual cause - excluded here (ACS fully disabled).
That case was never root-caused; the reporter gave up.

Questions for NVIDIA

1. Is GPUDirect RDMA (dmabuf path) expected to work across two root ports on
   consumer/workstation Intel platforms, or is a PCIe-switch topology the only
   supported configuration on these chipsets?
2. Is there a known Arrow Lake / Z890 erratum for inter-root-port non-posted
   P2P routing? Should Z890 get a chipset_info.c entry?
3. Are there driver-level knobs (registry keys, dmabuf mapping variants) that
   change how BAR1 pages are exposed for peer DMA reads that we haven't tried?
4. Any suggestions to capture the failing completion at the NIC (CX-5) level -
   e.g. vendor syndrome decode of 0x89/0x51 in this context - to pin down
   whether the completion carries poisoned data, wrong byte count, or wrong tag?

Attachments

Evidence bundle (lspci topology + -vv dumps, mstconfig, failing runs on both
ARL systems, Threadripper control, bpftrace script + output):
https://gist.github.com/TensorRaya/37c896bc9887767543d55e946e761e6d