[GSD-13010] Sporadic permanent GPU wedge on dual Arc Pro B70 (BMG G31): ccs/bcs engine reset + "Fault response: Unsuccessful -ENOENT/-EINVAL" under sustained Level-Zero inference load

Pre-submission Checklist

I am using the latest GPU driver version — currently on distro-provided 26.05.37020.3 (Ubuntu 26.04); latest release is 26.22.38646.4. Happy to retest on the latest release if you expect it to be relevant — the failure is easy for us to reproduce.
I have searched for similar issues and found none — closest field reports: [Bug]: XPU TP=2 on dual Intel Arc Pro B70 (Battlemage): GP fault + xe BCS engine reset reproduces in intel/vllm:0.17.0-xpu on Ubuntu 24.04 HWE 6.17 vllm-project/vllm#41663 (same dual-B70 hardware, GP fault + bcs engine reset) and [BUG] Tumbleweed + Intel Arc B580: OpenCL LAPLACIAN_ASSEMBLE triggers Xe Engine Reset (GT0: Engine reset: engine_class=ccs) darktable-org/darktable#20257 (same ccs engine-reset signature on Arc B580 via OpenCL, no vLLM/oneCCL involved).

System Information

GPU Hardware

2× Intel Arc Pro B70 (Battlemage G31, PCI ID 8086:e223, ASRock subsystem 6025)

DRI Devices Information

$ ls -ls /dev/dri/*
0 crw-rw----+ 1 root video  226,   0 Jul  4 18:12 /dev/dri/card0
0 crw-rw----+ 1 root video  226,   1 Jul  4 18:12 /dev/dri/card1
0 crw-rw----+ 1 root video  226,   2 Jul  4 18:12 /dev/dri/card2
0 crw-rw----+ 1 root render 226, 128 Jul  4 18:12 /dev/dri/renderD128
0 crw-rw----+ 1 root render 226, 129 Jul  4 18:12 /dev/dri/renderD129

$ ls -la /dev/dri/by-path/
pci-0000:45:00.0-card -> ../card0        (platform framebuffer)
pci-0000:c3:00.0-card -> ../card1
pci-0000:c3:00.0-render -> ../renderD128
pci-0000:c7:00.0-card -> ../card2
pci-0000:c7:00.0-render -> ../renderD129

GPU Detailed Information (lspci output)

$ sudo lspci -vvv -k -s c3:00.0   (c7:00.0 is identical)
c3:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
	Subsystem: ASRock Incorporation Device 6025
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Region 0: Memory at 11e00000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 10800000000 (64-bit, prefetchable) [size=32G]
	Kernel driver in use: xe
	Kernel modules: xe

Host platform: AMD EPYC 7413 (24c). GuC firmware: xe/bmg_guc_70.bin version 70.58.0.

Driver Version

intel-opencl-icd / libze-intel-gpu1 26.05.37020.3-1 (Ubuntu 26.04 distro packages)

Installed GPU Driver Packages

intel-opencl-icd          26.05.37020.3-1
intel-opencl-icd-legacy   24.35.30872.45-1
libze-intel-gpu-dev       26.05.37020.3-1
libze-intel-gpu-legacy1-1 24.35.30872.45-1
libze-intel-gpu1          26.05.37020.3-1
libze1:amd64              1.28.2-2

Driver Installation Details

Distro-provided packages from Ubuntu 26.04 LTS archives (no Intel PPA / no manual install). Workload runs inside Docker containers with --device /dev/dri passthrough; the failure reproduces with two different container userspaces (oneAPI 2025.3.2 and the intel/vllm 0.17 prebuilt image), which points away from a specific container userspace.

Linux Distribution

Other

Other Linux Distribution

Ubuntu 26.04 LTS

Kernel Version & Boot Parameters

$ uname -a
Linux battlematrix 7.0.0-27-generic #27-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 18 19:13:49 UTC 2026 x86_64 GNU/Linux

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.0-27-generic root=UUID=... ro quiet splash crashkernel=2G-4G:320M,...

Issue Description

Actual Behavior

Under sustained multi-request LLM inference (vLLM, tensor-parallel=2 across the two B70s, 8–12 concurrent requests, mixed 4k–32k contexts), roughly once every 2–6 hours one card takes a GPU page fault and the xe driver resets its compute/copy engines:

xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=29, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL

After the reset the userspace Level-Zero context is permanently wedged: the in-flight SYCL kernel never completes, the worker process never returns, and the serving engine dies (TimeoutError: RPC call to sample_tokens timed out → EngineDeadError). Only a full process/container restart recovers.

Key observations:

Both cards are affected (events captured on c3:00.0 and c7:00.0) → not a single defective unit.
8 application-level failures over July 2–4 each correlate to the second with a dmesg engine-reset event.
The dying scheduler step can be tiny: one captured crash dump shows 9 requests in decode, 9 total scheduled tokens, KV-cache at 14% — so this is not a batch-size/pressure problem. Incidence appears to scale with the number of kernel submissions over time, not with step size.
Ruled out on our side: model & quantization path (reproduces with int4/AWQ/FP8 weights), guided-decoding backend, vLLM graph capture (enforce-eager), vLLM async scheduling (disabled), vLLM version (intel/vllm 0.17 image and upstream v0.24 source build), scheduler oversubscription (max-num-seqs capped).
oneCCL "stable fallback" env from [Bug]: XPU TP=2 on dual Intel Arc Pro B70 (Battlemage): GP fault + xe BCS engine reset reproduces in intel/vllm:0.17.0-xpu on Ubuntu 24.04 HWE 6.17 vllm-project/vllm#41663 already applied in all runs (CCL_ENABLE_SYCL_KERNELS=0, CCL_ATL_TRANSPORT=ofi, CCL_ZE_IPC_EXCHANGE=sockets, CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0, SYCL_UR_USE_LEVEL_ZERO_V2=0) — it removed an earlier crash mode but not this one.
Same ccs engine-reset signature has been reported without vLLM/oneCCL entirely ([BUG] Tumbleweed + Intel Arc B580: OpenCL LAPLACIAN_ASSEMBLE triggers Xe Engine Reset (GT0: Engine reset: engine_class=ccs) darktable-org/darktable#20257, OpenCL on B580), suggesting driver/firmware level.

Expected Behavior

No GPU page faults / engine resets under sustained compute load; or, if an engine reset occurs, the Level-Zero context should recover (or at least fail fast with an error the application can handle) instead of wedging permanently.

Reproduction Rate

Always reproduces - 100% (within 2–6 hours of sustained load; see steps)

Steps to Reproduce

Host as described above (dual Arc Pro B70, xe driver, kernel 7.0.0-27).
Serve a ~27B LLM with vLLM v0.24 (XPU build from upstream Dockerfile.xpu, oneAPI 2025.3.2) with --tensor-parallel-size 2 --enforce-eager --max-model-len 32768, oneCCL env as above. (Also reproduces with the intel/vllm:0.17.0-xpu prebuilt image and a 70B AWQ model.)
Drive sustained concurrent load: 8–12 parallel /v1/chat/completions requests in a loop for hours (we use JSON-schema constrained outputs, mixed prompt sizes 4k–32k tokens).
Within 2–6 hours: dmesg shows the ccs/bcs engine reset + unsuccessful fault response on one of the cards, and the serving engine hangs permanently.

Minimal load generator (python, against the vLLM OpenAI endpoint):

import httpx, json, time, concurrent.futures as cf
SCHEMA = {"type":"object","properties":{"verdict":{"type":"string"},"reasoning":{"type":"string"}},
          "required":["verdict","reasoning"]}
def call(i):
    p = f"Analyze company #{i}. Respond with JSON matching: {json.dumps(SCHEMA)}"
    return httpx.post("http://127.0.0.1:8080/v1/chat/completions", timeout=600, json={
        "model":"local","messages":[{"role":"user","content":p}],
        "max_tokens":300,"response_format":{"type":"json_object"}}).status_code
while True:
    with cf.ThreadPoolExecutor(max_workers=12) as ex:
        print(time.strftime("%T"), list(ex.map(call, range(12))))

Regression Information

Is this a regression?

Yes, this is a regression — unknown: this dual-B70 configuration has shown the issue on every stack we tried; we have no known-good baseline.

Logs

API Call Logs

Not captured yet — the incident is frequent enough to instrument: we can provide ze_debug/API traces, py-spy/gdb stacks of the hung worker process, or xe debugfs state on request. Please tell us which traces are most useful.

strace Logs

Not captured (available on request, see above).

System Logs / dmesg Output

All engine-reset events captured July 2–4 (timestamps CEST). Every one coincides (±2 s) with an application-level engine death:

[Jul 2 23:40:09] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=43, state=0x289
[Jul 2 23:40:09] xe 0000:c3:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 2 23:53:37] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=48, state=0x289
[Jul 3 00:01:23] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=27, state=0x289
[Jul 3 00:01:23] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=32, state=0x289
[Jul 3 00:12:24] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=26, state=0x289
[Jul 3 00:18:56] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=43, state=0x289
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
[Jul 4 05:49:27] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=48, state=0x289
[Jul 4 10:51:14] xe 0000:c3:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=31, state=0x289
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL

Backtrace (if crash or hang occurred)

Application-side (vLLM EngineCore parent; the GPU worker itself hangs and never returns — we can attach py-spy/gdb to it at the next occurrence on request):

(EngineCore) ERROR [core.py:1233] EngineCore encountered a fatal error.
  File ".../vllm/v1/executor/multiproc_executor.py", line 387, in get_response
    status, result = mq.dequeue(timeout=dequeue_timeout)
  File ".../vllm/v1/executor/multiproc_executor.py", line 370, in get_response
    raise TimeoutError(f"RPC call to {method} timed out.") from e
TimeoutError: RPC call to sample_tokens timed out.
(APIServer) vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

Captured scheduler state at one crash (shows the dying step was 9 decode tokens across 9 requests, KV 14%):

SchedulerOutput(..., total_num_scheduled_tokens=9, ...)
SchedulerStats(num_running_reqs=9, num_waiting_reqs=0, kv_cache_usage=0.1439...)

Reproducer

Source Code / Reproducer

See "Steps to Reproduce" above — vLLM serve command plus the 15-line python load generator. Full container logs, complete dmesg, and our original forensics write-up are in #946 (filed before we were aware of the template — apologies; this issue supersedes it).

Triage datapoint from an independent dual-B70 rig (cross-report from vllm-project/vllm#41663, user Zumbasam, 2026-07-06):

On their stack — Ubuntu 24.04.4, kernel 6.17.0-23, GuC 70.44.1, two B70 on separate root ports — engine_class=ccs resets under SYCL-allreduce load are recoverable: no Fault response lines, the card survives, and with CCL_ENABLE_SYCL_KERNELS=0 (+CCL_ALLREDUCE=ring) they get 2-hour soaks with zero engine resets.

On our stack — kernel 7.0.0-27, GuC 70.58.0, compute-runtime 26.05 — the resets come with Fault response: Unsuccessful -ENOENT/-EINVAL and permanently wedge the Level-Zero context, even with CCL_ENABLE_SYCL_KERNELS=0 applied.

Same GPU model, same workload class → the permanent-wedge mode appears to correlate with the newer KMD / GuC firmware / compute-runtime combination.

We're happy to bisect on our rig (it reproduces within 2–6 h): which component would you like us to vary first — GuC firmware (70.58 → 70.44), kernel xe, or compute-runtime (26.05 → 26.22)? Any debug keys/traces you want captured during the next occurrence?

Additional data point — single-GPU B580, no oneCCL/TP, and a firmware A/B (70.58.0 → 70.65.0) that eliminated the instantly-reproducible faults

Reporting a data point that isolates a couple of variables the existing reports here and in the linked vLLM issues (#41663, vllm#48953) can't, since those are all dual-GPU + oneCCL + tensor-parallel.

Environment


GPU	1× Intel Arc B580 (Battlemage G21, `8086:e20b`) — single card
Host	AMD EPYC 74F3 (Zen 3 "Milan"), 512 GB DDR4
Kernel	7.0.0-27-generic (Ubuntu 26.04), `xe` driver 1.1.0
Workload	llama.cpp, SYCL/Level-Zero backend, single device — MoE hybrid (attention + KV cache on GPU, experts on CPU). No oneCCL, no tensor-parallelism, no vLLM.

Same signature, without any multi-GPU/oneCCL involvement

We hit the identical kernel signature reported here:

xe 0000:83:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:83:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, ...
xe 0000:83:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
xe 0000:83:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, ...

Note the bcs (copy engine) reset occurs on a single GPU with no inter-device copy traffic (no oneCCL, no IPC handles, no TP). That suggests the underlying xe/GuC fault is not solely the oneCCL stale-IPC-handle path — there's a single-GPU code path that produces the same ccs/bcs reset + -ENOENT.

Firmware A/B — this is the main point

Single controlled variable: only the GuC firmware changed; same kernel (7.0.0-27), same hardware, same workload.

GuC bmg_guc_70.bin v70.58.0: the ccs/bcs engine reset + -ENOENT/-EINVAL faults appeared almost immediately — at GPU init and within seconds of applying real compute load. Under sustained load it escalated to a hard wedge quickly.
GuC bmg_guc_70.bin v70.65.0 (2026-07-14, latest in linux-firmware; WHENCE: "GuC API/APB ver 70.65.0 for Battlemage"): the load-time faults are gone. It has since run ~1 hour of heavy sustained + concurrent load — including a single ~114k-token prefill plus concurrent generations with the GPU pinned at max clock — with zero Fault response/Engine reset lines.

I am explicitly not claiming the bug is fixed — the permanent wedge reported here is a multi-hour phenomenon and ~1 hour is well inside the window where even the old firmware could run clean. What changed is concrete: the previously instantly-reproducible faults no longer reproduce on 70.65.0. Can other affected users (B70/B50, dual-GPU) test whether GuC 70.65.0 changes their reproduction rate? The GuC changelog is a one-liner with no detail, so community A/B is the only way to tell if something in 70.58.0→70.65.0 touched this path.

How to upgrade the GuC firmware (what we did)

The GuC blob ships in linux-firmware, independent of the kernel — so you can test 70.65.0 without changing kernels. If your distro already packages it (Fedora shipped 70.65.0 in its ~20260519 linux-firmware), just update that package. Otherwise, drop the blob in manually:

# 0. current version
sudo dmesg | grep -i 'guc.*version'      # e.g. bmg_guc_70.bin version 70.58.0

# 1. fetch the blob from upstream linux-firmware (WHENCE confirms 70.65.0 for Battlemage)
curl -Lo /tmp/bmg_guc_70.bin \
  https://gitlab.com/kernel-firmware/linux-firmware/-/raw/main/xe/bmg_guc_70.bin

# 2. back up the current blob
#    (Ubuntu/Debian ship it zstd-compressed as bmg_guc_70.bin.zst)
sudo cp /lib/firmware/xe/bmg_guc_70.bin.zst ~/bmg_guc_70.bin.zst.bak

# 3. install it — Ubuntu/Debian (compressed firmware): recompress then replace
zstd -19 /tmp/bmg_guc_70.bin -o /tmp/bmg_guc_70.bin.zst
sudo cp /tmp/bmg_guc_70.bin.zst /lib/firmware/xe/bmg_guc_70.bin.zst
#    …distros using UNCOMPRESSED firmware instead: sudo cp /tmp/bmg_guc_70.bin /lib/firmware/xe/

# 4. refresh initramfs (xe fw can be early-loaded) and reboot
sudo update-initramfs -u                 # Fedora: sudo dracut -f
sudo reboot

# 5. verify after reboot
sudo dmesg | grep -i 'guc.*version'      # should now read 70.65.0

Rollback is just: restore the .bak, update-initramfs -u, reboot.

AMD-host mitigation (both reporters here are on EPYC)

On AMD platforms, before disabling the IOMMU we saw the xe engine-reset events coincide with AMD-Vi DMA timeouts that took the entire host down (unrecoverable, required a power cycle):

AMD-Vi: Completion-Wait loop timed out
IOMMU ...: IOTLB_INV_TIMEOUT device=0000:83:00.0

Booting with amd_iommu=off (verified: iommu_groups count 0, DMA via SWIOTLB) converted the outcome from whole-host lockup to a recoverable GPU reset. It does not fix the underlying xe/GuC fault, but for AMD-EPYC hosts it's the difference between "process restart" and "hard reboot." (Corroborated by an Arch thread on the same B580 + amd_iommu=off; note iommu.strict=1 made it worse there.)

Happy to provide

Full dmesg, xe debugfs/GT state, GuC log (xe.guc_log_level=3), or a minimal single-GPU Level-Zero repro if that's useful for narrowing this down.

add watchdog: detect and recover from Xe2 Level-Zero wedge

bound EngineCore shutdown after fatal error to prevent wedged-worker …

Cross-report, same fault class (ccs/bcs reset + Fault response: Unsuccessful -ENOENT/-EINVAL, no Timedout job) on a different stack:
single Arc Pro B60, OpenVINO GPU plugin/OpenCL, not vLLM/oneCCL. New
evidence this thread doesn't have: an allocation-log line naming the
faulting buffer.

Edit: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/9141 - kernel side.

The faulting GPU VA is the start of the runtime's own direct-submission
SEMAPHORE_BUFFER, not an application buffer. Kernel fault line paired
with the matching compute-runtime allocation-log line
(NEOReadDebugKeys=1 LogAllocationType=1 LogAllocationStdout=1), same run:

Faulted Address: 0x0000d556aa670000  FaultType: 0  FaultLevel: 4  EngineClass: 5 ccs
Fault response: Unsuccessful -ENOENT
Created Graphics Allocation of type SEMAPHORE_BUFFER  Size 65536  GPU VA 0xffffd556aa670000-0xffffd556aa67ffff

Sign-extended 48-bit address, exact match. allocateResources() marks it
evictable=false on submit -- UMD bookkeeping only, not a kernel pin (xe
docs: user BOs are never pinned). handleResidency() never re-validates the
ring/semaphore per dispatch, so an eviction under VRAM pressure could leave
a live semaphore wait pointed at a non-present page. Diagnosis, not confirmed.

Env
GPU	Arc Pro B60 (Battlemage G21), `8086:e211`, 24 GB
Kernel	`7.0.14-12`, distro build
compute-runtime	`26.27.39122.11`
GuC firmware	`70.65.0` -- we fault on 70.65.0, re: the A/B above
OpenVINO GPU plugin	`2026.4.0-22849-71640275d29`

Repro (98k-token single prefill, u8:i4 KV, paged attention):

config	result
VRAM near limit, host/other-GPU busy	1/10
VRAM near limit, quiet host	1/1 (845 s)
+ `EnableDirectSubmission=0`	4/4 (not discriminating -- quiet-host control also passes; +1.4% cost elsewhere)
headroom restored (bound partitions)	2/2 (830.7 s)

Ask: pin or re-validate the ring/semaphore BOs before a wait, or bind
with a residency fence -- see allocateResources()/makeResourcesResident(),
makeResidentWithinOsContext(), handleResidency(). Full block + citations
in the fold below.

Full fault block, allocation log, and source citations

[245349.964621] xe 0000:0f:00.0: [drm] Tile0: GT0:
	ASID: 998  Faulted Address: 0x0000d556aa670000
	FaultType: 0  AccessType: 0  FaultLevel: 4  EngineClass: 5 ccs  EngineInstance: 0
[245349.964636] Fault response: Unsuccessful -ENOENT
[245349.965738] Engine reset: engine_class=ccs, guc_id=8, state=0x289
[245349.966000] Engine reset: engine_class=bcs, guc_id=9, state=0x289
[245349.971177] xe 0000:0f:00.0: [drm] Tile0: GT0:
	ASID: 998  Faulted Address: 0x0000d556aa740000
	FaultType: 0  AccessType: 0  FaultLevel: 4  EngineClass: 3 bcs  EngineInstance: 0
[245349.971188] Fault response: Unsuccessful -ENOENT

Allocation log (same run):

COMMAND_BUFFER   Size 1114112  GPU VA 0xffffd556aa660000-0xffffd556aa76ffff  (17x64KiB, encloses both below)
RING_BUFFER      Size 327680   GPU VA 0xffffd556aa680000-0xffffd556aa6cffff
SEMAPHORE_BUFFER Size 65536    GPU VA 0xffffd556aa670000-0xffffd556aa67ffff  (= ccs fault address)
SEMAPHORE_BUFFER Size 65536    GPU VA 0xffffd556aa740000-0xffffd556aa74ffff  (= bcs fault address)

Same two addresses recur under 9 distinct ASIDs (946, 950, 954, 962, 965,
970, 974, 978, 998). Crash depth varies (~3.6k, ~12k, ~40k tokens into the
same 98k-token prefill across three traced runs) -- not a fixed depth-tied
defect on our side. A stale-binding trace on our own paged-attention
intermediates found 0 mismatches over 5,867 dispatch lines, ruling out our
buffer lifetime management.

Source citations (compute-runtime master
2446b526c3d4b958d8707392a271db5a252e20e6, fetched 2026-09-04):
shared/source/direct_submission/direct_submission_hw.inl lines 111
(allocateResources), 190 (makeResourcesResident, evictable=false);
shared/source/os_interface/linux/drm_memory_operations_handler_bind.cpp
line 102 (makeResidentWithinOsContext, flag only skips
updateResidencyTaskCount); shared/source/direct_submission/linux/ drm_direct_submission.inl line 205 (handleResidency, waits on paging
fence only, no re-validation); shared/source/os_interface/linux/xe/ ioctl_helper_xe.cpp line 1270 (getFlagsForVmBind, sets
DRM_XE_VM_BIND_FLAG_IMMEDIATE only -- bind timing, not pinning);
shared/source/xe2_hpg_core/hw_info_bmg.cpp line 36
(directSubmissionEngines enabled by default on ccs, ccs1, bcs).
docs.kernel.org/gpu/xe/xe_mm.html: "user BOs are evictable and user BOs
are never pinned by Xe."

Cross-referencing from the Xe kernel side (drm/xe#8390) — we reproduced the kernel-level
failure this issue tracks on a different B70 board and driver version, which may
narrow the scope for whoever triages this:

GPU: Intel Arc Pro B70 desktop (ASRock board, subsystem 1849:6025), PCI 0xe223 — full tower card, same form factor as the original B70 report
Kernel: 7.2.3-070203-generic (mainline build, xe ahead of the 7.0.0-31 used in the
original xe#8390 report), GuC 70.72.1, HuC 8.2.10, DMC 2.6
Workload: ollama 0.33.3 (Vulkan/ANV), gemma-4-12B Q4_K_M, single request, ~31k-token
prompt at ctx 65536 → sustained ~490s prefill
Result: 2/2 trips — Timedout job: engine ccs, guc_id=29 (same engine slot on every
trip), Xe device coredump has been created, llama-server killed by the driver.
Control (short prompt, same ctx 65536): clean. So it's the sustained long prefill at large
context, not the context size itself.
Timing shift: trip lands ~130s into prefill on 7.2.3 vs ~395s on 7.0.0-31 — earlier,
not later, as the driver moved forward. Not fixed by newer xe.
Evidence: full coredump (30.9MB raw, xz'd) attached to drm/xe kernel issue #8390, note 3659046 (https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/8390#note_3659046).

Happy to run targeted experiments on this hardware if it helps isolate — drm-tip bisect,
debug knobs, anything concrete.

@ralphsanders2001-ai

Cross-report: 4-way matrix (2 card pairs × 2 vLLM XPU images × 2 GuC firmwares), all 4 combinations reproduce this exact signature — plus a controlled card/slot swap

Posting to add a controlled matrix from a 4x Arc Pro B70 host, since this thread
already spans dual/single-GPU, vLLM/ollama/OpenVINO/OpenCL, and now GuC
70.58/70.65/70.72.1 without a clean firmware fix. Our reproducer is fast
(3-5 minutes under adversarial load vs. the 2-6h organic reproduction reported
above), which may help anyone trying to bisect further.

Environment


GPUs	4x Intel Arc Pro B70 (Battlemage G31, `8086:e223`)
Host	AMD EPYC 7642, Supermicro H12SSL-i
Kernel	`7.0.0-31-generic` (Ubuntu 24.04.4 LTS)
GuC firmware tested	packaged `70.58.0` and upstream `linux-firmware` `70.72.1` (WHENCE-confirmed), selected via `xe.guc_firmware_path`
vLLM XPU images tested	`vllm/vllm-openai-xpu:v0.28.0` (vllm 0.28.0+xpu, vllm-xpu-kernels 0.1.13.2, Level Zero UMD 26.05.37020.3) and `v0.29.0` (vllm 0.29.0+xpu, vllm-xpu-kernels 0.1.14.1, Level Zero UMD 26.27.39122.11) — torch 2.13.0+xpu and oneCCL unchanged between the two
Model / workload	Qwen3.8-27B, FP8, `--tensor-parallel-size 2`, `--max-model-len 8192`, `--max-num-seqs 8`; `vllm bench serve`, random 2048-in/256-out, `--request-rate inf --max-concurrency 8 --ignore-eos`, seed 42

The matrix

Card pair	vLLM image	GuC	Result
Pair A	v0.28.0	70.58.0	FAIL (confirmation-matrix cell, 56/100)
Pair B (different physical cards, swapped slots vs. Pair A's original run)	v0.28.0	70.58.0	FAIL (88/100)
Pair A	v0.29.0	70.58.0	FAIL (56/100)
Pair A	v0.29.0	70.72.1	FAIL (40/100)

All four combinations produce the identical fault family: ccs engine timeout
(Timedout job) with an xe devcoredump on one card, followed within seconds
by cascading ccs+bcs engine resets on both cards of the pair.
Representative excerpt (v0.29.0 / GuC 70.72.1 cell):

xe 0000:83:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=12, state=0x3
xe 0000:83:00.0: [drm] Tile0: GT0: Timedout job: seqno=252, lrc_seqno=252, guc_id=12, flags=0x20 in python3 [20097]
xe 0000:83:00.0: [drm] Xe device coredump has been created
xe 0000:83:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=16, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=18, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=22, state=0x289

We also physically swapped two of the four cards between PCIe slots in an
earlier round of this investigation (unrelated to the table above): the
originally-suspect card passed a 4-hour sustained pass in its new slot, and the
known-good card that took its old slot failed under the same general-model
workload. Combined with the matrix above, we can rule out a single defective
card or a single defective slot — the fault follows the workload class
(sustained load on this model/engine combination), not any specific piece of
hardware.

Notes relevant to points already raised in this thread

@ralphsanders2001-ai's GuC 70.72.1 / kernel 7.2.3 result (xe#8390) is
corroborated here on a completely different workload and kernel. We see
the same "not fixed by newer GuC" outcome on 70.72.1 with vLLM TP=2 on
kernel 7.0.0-31, vs. your ollama single-request long-prefill on 7.2.3. Two
unrelated workloads, two different kernel majors, same firmware, same
failure family — this seems to weaken firmware-version as the controlling
variable on its own.
@davetha's 70.65.0 single-GPU result: we haven't tested 70.65.0
specifically (only 70.58.0 and 70.72.1), so we can't directly confirm or
contradict the "faults gone at init" observation, but per your own caveat
and our 70.72.1 result, a firmware version alone doesn't appear sufficient
to close the multi-hour/sustained-load version of this bug.
AMD-Vi: on a separate solo-general run (different evidence set, not the
matrix above) we saw ten AMD-Vi IO_PAGE_FAULT events on one card in a
single 4K page, coincident with the same ccs timeout/reset — not the
Completion-Wait loop timed out host-lockup mode described upthread. We
have not disabled amd_iommu (per this thread's own caveat that it's
containment, not a fix, we've left it enabled).
@marfrit's SEMAPHORE_BUFFER / eviction-under-residency-pressure
diagnosis: plausible fit for what we see, but we have not correlated our
own devcoredumps against NEOReadDebugKeys=1 LogAllocationType=1 LogAllocationStdout=1 allocation logs. Happy to capture that pairing on our
next occurrence if it's useful — our reproducer above gets there in
3-5 minutes, which may be faster than other environments in this thread.

Let us know if a specific trace (ze_debug, xe.guc_log_level=3, GT/debugfs
state, allocation-log correlation) would help narrow this down further; our
reproduction rate is high enough to capture it on the next run.

Context: we previously reported ccs engine resets / "Timedout job" events on
Battlemage (BMG G31, Arc Pro B70, PCI 8086:e223) under sustained vLLM TP=2
load (this issue). We have since captured two first-fault devcoredumps with a
fresh-start harness (tracing armed before worker launch, allocation-create
logging enabled, dump preserved within ~0.6 s of detection). This comment
shares the new facts and asks three specific questions.

Two fault instances (both first-fault captures, not teardown noise)

Field	Fault A (earlier)	Fault B (latest)
Card	87:00.0	c3:00.0
Engine/class	ccs (5)	ccs (5)
guc_id / seqno	28 / 98	22 / 76
Worker	VLLM::Worker (live serving)	VLLM::Worker (live serving)
Kernel	Ubuntu 7.0.0-31-generic (xe, GuC submission, execlists off)	same
GuC firmware	xe/bmg_guc_70.bin 70.58.0	same
Queue state at fault	REGISTERED+ENABLED (0x3)	REGISTERED+ENABLED (0x3)
Policy	quantum 1000us, preempt 640000us (stock CONFIG defaults)	same
Dump "Timeout" field	0 ms (snapshot artifact, see below)	0 ms

Both occurred during normal serving (not teardown): later bcs/ccs resets with
fault VAs in a GPU_TIMESTAMP_DEVICE_BUFFER allocation only appear ~36 s after
the first fault, during container stop (states 0x289/0x2c9, i.e.
BANNED|KILLED|PENDING_DISABLE). Ordering: GuC CONTEXT_RESET_NOTIFICATION →
KMD "Engine reset" log → "Timedout job" log → devcoredump, all within ~50 ms.

Kernel-side reading (mainline v7.0 sources)

"Engine reset: engine_class=ccs ..." is printed by
xe_guc_exec_queue_reset_handler, dispatched only by
XE_GUC_ACTION_CONTEXT_RESET_NOTIFICATION — i.e. the firmware reset the
context and notified the driver; the subsequent "Timedout job" TDR ran with
skip_timeout_check (exec_queue_reset set), so no KMD timeout was measured.
The devcoredump's "Timeout: 0 (ms)" copies the drm-scheduler timeout value
at snapshot time (post-reset), not the configured policy; effective ccs
job_timeout_ms is the 5000 ms class default.
The engine head (ACTHD/BBADDR) sits inside a COMMAND_BUFFER allocation
created at startup; the timed-out batch address is inside a RING_BUFFER
allocation. We do not claim this identifies the stalled instruction or
dependency.

GuC event-log terminal pattern (structural decode only)

We decoded the devcoredump's [LOG] blob (ascii85; 36-byte buffer-state
headers at offsets 0/36/72 with markers 0xCABBA9E6_0xDEADFEED etc.; fixed
20-byte records [stamp][0][meta][fmt][payload]; no buffer overflow; one
32-bit stamp wrap; 1,908 and 2,827 records respectively).

The final ten records are structurally similar across both faults:
identical meta-code sequence
(0x8102 0x4ad 0x4b0 0x8101 0x4c6 0x4c6 0x4c6 0x84f6 0x8101 0x84f8),
7/10 positions fully equal, with the last three differing only in fmt/payload
(0x1c vs 0x16; 0x1040231b vs 0x104022ee) — presumably context-dependent
fields. The distance from the final record to the snapshot timestamp is
~857k ticks in both runs (upper bound only: the driver copies the log buffer
before reading GUC_PMTIMESTAMP_LO, so this includes capture latency; the
record timestamp's clock domain is unverified).

We do NOT claim this pattern is a reason-specific signature — it may be
ordinary reset handling.

Questions

Is this terminal pattern ordinary context-reset handling, or does any of it
encode the reset REASON (watchdog expiry vs. preemption failure vs. policy
violation)? Is there a public mapping of the record fields (what we call
meta/fmt/payload) for GuC 70.58?
What is the event-record timestamp's clock domain (19.2 MHz CS ref, GuC
global, other), and what is the authoritative record format definition?
Which single additional diagnostic would best distinguish initiating
causes here — e.g. a raised guc_log_level (which value), GuC crash-dump
region enablement, or a specific tracepoint set — given the constraint of
stock kernel + stock GuC firmware?

Additional reproduction data for Arc Pro B70 on Ubuntu 24.04 LTS.

This report is from 2026-09-18 JST.

Hardware and kernel:

GPU: Intel Arc Pro B70
PCI address: 0000:03:00.0
PCI ID: 0xe223
Kernel: 7.0.0-31-generic
Kernel driver: xe
GuC firmware: xe/bmg_guc_70.bin
GuC version: 70.72.1
GuC wanted version: 70.54.0
Application: llama-server
Intel Compute Runtime: 26.31.39395.13-1~~24.04~~ppa1

xe-b70-devcoredump-20260918.txt

Level Zero: 1.32.0-1~~24.04~~ppa1

The user-space Intel Compute Runtime and Level Zero package versions were not recorded at the time of the incident.

Observed kernel event sequence:

19:54:15:

BCS page fault response failed with -EINVAL.
ASID: 462
Faulted Address: 0x0000d556a8ac7000
FaultType: 0
AccessType: 0
FaultLevel: 0
EngineClass: 3 bcs
EngineInstance: 0

20:15:56:

Engine reset: engine_class=ccs, logical_mask=0x1, guc_id=2, state=0x3
Timedout job: seqno=198, lrc_seqno=198, guc_id=2, flags=0x20 in llama-server

20:19:18:

Fault response: Unsuccessful -ENOENT
Engine reset: engine_class=bcs, logical_mask=0x1, guc_id=6, state=0x289
Related fault record:
- ASID: 580
- Faulted Address: 0x0000d556aa411000
- FaultType: 0
- AccessType: 0
- FaultLevel: 4
- EngineClass: 3 bcs
- EngineInstance: 0

21:41:41:

Engine reset: engine_class=ccs, logical_mask=0x1, guc_id=2, state=0x3
Timedout job: seqno=3514, lrc_seqno=3514, guc_id=2, flags=0x20 in llama-server

21:43:00:

Engine reset: engine_class=bcs, logical_mask=0x1, guc_id=6, state=0x289
Fault response: Unsuccessful -ENOENT
Related fault record:
- ASID: 1236
- Faulted Address: 0x0000d556aa411000
- FaultType: 0
- AccessType: 0
- FaultLevel: 4
- EngineClass: 3 bcs
- EngineInstance: 0

The same BCS fault address, 0x0000d556aa411000, was observed with two different ASIDs.

The journal export interleaves multiline kernel messages, so the exact timestamp association of the ASID 580 and ASID 1236 fault records may not be fully preserved.

The attached Xe device coredump corresponds to the second CCS timeout.

Coredump summary:

Reason: Timedout job - seqno=3514, lrc_seqno=3514, guc_id=2, flags=0x20
Context: ccs2
Engine class: COMPUTE
Schedule state: REGISTERED|RESET|BANNED
LRC head: 9128
LRC tail: 9288
H2G CTB broken: 0
G2H CTB broken: 0
G2H outstanding: 0
VM snapshot result: -ENODEV
Coredump size: 522760 bytes
Uncompressed coredump SHA-256:
6f9f69a79d1ad456c7ebcbf1076be29ec2e326fe5af0f06058e37f54b729eb8b

No direct root-cause claim is made here.

The available data shows a BCS not-present read fault, unsuccessful page-fault responses (-EINVAL and -ENOENT), CCS context resets, BCS resets, and timed-out CCS jobs.

This reproduces the same general failure family described in this issue.

The complete coredump and the filtered kernel event log are attached.

Due to limited technical resources, I may not be able to provide further diagnostics. The attached files contain the complete data currently available from this incident.

Single card datapoint on 26.35.39758.10, and a caveat that matters: on this card the failure
follows the userspace stack, not just the driver version.

Host: one Arc Pro B70 (8086:e223, Sparkle, 0000:04:00.0), Meigao N5A mini PC, Ubuntu 26.04,
kernel 7.0.0-30-generic, xe, GuC firmware xe/bmg_guc_70.bin version 70.58.0 with GuC submission.
The same box also holds an RTX 3090 Ti.

Driver stack on the failing run:
intel-opencl-icd / libze-intel-gpu1 26.35.39758.10 (upstream release 2026-09-17)
intel-igc-core-2 / intel-igc-opencl-2 2.41.5
libze1 1.32.0-1~~26.04~~ppa1
libigdgmm12 22.10.0
intel-ocloc 26.31.39395.14 (see the packaging note at the end, this run had the compiler
one release behind the driver)

Workload: vLLM 0.25.1+xpu from a venv, fp16 merge of our own judge model, single card,
--max-model-len 4096 --max-num-seqs 12 --max-num-batched-tokens 16384 --gpu-memory-utilization
0.88 --enable-prefix-caching, two client processes, 12 requests in flight.

It served 2,793 requests in about two minutes and then the device died. Throughput right before
the fault was steady at about 1,079 prompt tokens/s and 577 generation tokens/s with 12 running
requests and KV cache at 0.2 percent.

Kernel sequence:

17:17:53 Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=12, state=0x3
17:17:53 Timedout job: seqno=4294967206, lrc_seqno=4294967206, guc_id=12, flags=0x20 in python [2320039]
17:17:53 Xe device coredump has been created
17:17:54 Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=16, state=0x289
17:17:54 Fault response: Unsuccessful -ENOENT

Userspace: RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST),
then EngineDeadError. vLLM dumped scheduler output for the 12 in flight requests, each with about
144 computed tokens and 1 to 15 output tokens, so this was a live steady state serving load and
not a startup or teardown artifact.

Two things worth stating precisely:

This is not the xpu-smi polling problem that closed [Bug]: Qwen/Qwen3.6-35B-A3B and Qwen/Qwen3.6-35B-A3B-FP8 issue while inferencing on Intel XPU (4xB70) vllm-project/vllm#50850. We do not run an
xpu-smi poll loop here. B70 telemetry on this host reads sysfs hwmon directly and nothing
spawns xpu-smi on a timer.
The card recovered on its own. The engine reset cleared it, XPU compute works again, xpu-smi
reports Device state: normal with 31.4 of 31.9 GiB free, no EXT4 or I/O errors, and the
3090 Ti in the same box never noticed. No reboot was needed.

The engine_class=bcs ... state=0x289 line is the signature we have been chasing since August.
Driver context on this same card and host:

26.27.39122.14 same vLLM XPU workload DEVICE_LOST loop, ccs engine resets about every 2 minutes
25.48.36300.8 same vLLM XPU workload 49,104 requests, 0 errors (2026-08-31)
26.35.39758.10 same vLLM XPU workload DEVICE_LOST after 2,793 requests (this post)

So 26.35 has not fixed this path. On the vLLM XPU side it is worse than 25.48 for us.

Now the caveat. Our llama.cpp SYCL path on the same driver looked clean in short windows and then
faulted repeatedly under the same 27B model. Same card, same 26.35.39758.10, same llama.cpp SYCL
build (10655, cb300598d), Qwen3.8-27B Q4_0:

earlier in the day, long context, 2 slots 123 requests, 0 errors, 0 kernel lines
24.7K token prompts, 2 slots 4 requests ok, 4 timed out, 98,784 prompt tokens,
server survived:

17:37:18  Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x3
17:37:18  Timedout job: seqno=369, lrc_seqno=369, guc_id=2, flags=0x20 in llama-server [2797580]
17:57:37  Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=6, state=0x289
17:57:37  Fault response: Unsuccessful -ENOENT

the same shape again server process died and left a coredump:

18:22:15  Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x3
18:22:15  Timedout job: seqno=4294967244, lrc_seqno=4294967244, guc_id=2, flags=0x20 in llama-server [3891561]

short prompts, 4 clients faulted as well:

18:40:50  Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x3
18:40:50  Timedout job: seqno=852, lrc_seqno=852, guc_id=2, flags=0x20 in llama-server [186117]
18:55:38  Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=6, state=0x289
18:55:38  Fault response: Unsuccessful -ENOENT

clean windows that same afternoon the 27B with short prompts for 1200s (276 requests,
0 errors) and the 4B model at high request rate

Tallies from the mixed soak on this card (6 hour budget, it stops early after 3 faulting windows):

17:57:39 window 1 27B long-context (24.7K prompts) crash_lines=4
18:13:51 window 2 27B small-prompt soak crash_lines=0
18:19:34 window 3 4B high-rate soak crash_lines=0
18:34:58 window 4 27B long-context (24.7K prompts) crash_lines=5
18:55:40 window 5 27B small-prompt soak crash_lines=4

5 windows run, 3 faulted.
faulting windows: 1 (27B long-context (24.7K prompts)), 4 (27B long-context (24.7K prompts)), 5 (27B small-prompt soak)
clean windows: 2, 3
=== SOAK END Fri Sep 18 06:55:41 PM EDT 2026 windows=5 faults=3 ===

That bcs state=0x289 with Fault response -ENOENT is the same pair as the vLLM fault above, reached
from a different userspace on the same driver, so the fault is not confined to one stack. The
faulting windows cover both prompt shapes at 27B (two long context windows and one short prompt
window) while the same build and driver passed a 123 request long context window and a 1200s short
prompt window earlier the same afternoon, so this is intermittent rather than tied to one shape.
That is also why a single clean soak should not be read as a fix.

For completeness, the older driver fails differently on this stack. On 25.48.36300.8 a bounded
allocation probe (six 4GB sycl::malloc_device buffers, each fully written with memset, nothing
else running, one card) fails within seconds, twice out of two on a freshly bound device:

Faulted Address: 0x0000d55936bff000
FaultType: 0, AccessType: 1, FaultLevel: 2
EngineClass: 3 bcs EngineInstance: 0
Fault response: Unsuccessful -EINVAL
Engine memory CAT error [18]: class=bcs, logical_mask: 0x1, guc_id=6
Timedout job: seqno=4294967170, lrc_seqno=4294967170, guc_id=6, flags=0x20 in b70_alloc_probe [308709]
Xe device coredump has been created
exec queue reset detected (x7)
userspace: UR_RESULT_ERROR_DEVICE_LOST

A second run of the same probe on the same driver hung instead of faulting, and the driver reset
it itself about three and a half minutes later ("Schedule disable failed to respond, guc_id=6",
then "trying reset from guc_exec_queue_timedout_job", then reset queued, started, done). The same
probe completed 6 of 6 cycles (24GB allocated, written and held) on 26.35.39758.10, twice out of
two, on devices bound the same way. The bcs -EINVAL here is close to what localyouser posted above
from llama-server on 26.31: same engine, same -EINVAL, different address and access type.

So bcs and ccs queue resets on this part are reachable from at least three submission patterns
(vLLM XPU serving, llama.cpp SYCL long context, and plain sycl::malloc_device plus memset with no
model and no compute kernels), and no driver version we have tested is clean across all of them.
Two of those reproduce quickly on a single card, which may be easier for your engine team than the
dual card setups in this thread. Happy to package the 20 line SYCL probe and the exact vLLM
recipe, and we have coredumps from the llama.cpp long context fault and from the allocation fault.

Separately, and probably a different ticket: getting vLLM to start at all on 26.35 needed eight
workarounds, and one of them is a real packaging gap worth flagging:

The kobuk PPA carries no intel-ocloc for the 26.35 line (the newest there is 26.31.39395.14),
so a PPA-only user ends up with a compiler one release behind the driver. The upstream 26.35
release does ship intel-ocloc_26.35.39758.10-0, so this run used the 26.31 ocloc only because
we were installing from the PPA. We plan to re-run with the matching 26.35 ocloc and will
report whether the fault changes, since a compiler and driver mismatch is exactly the sort of
thing that could matter here.
Triton's Intel backend dlopen()s the unversioned libze_loader.so, which only ships in
libze-dev; the runtime package installs libze_loader.so.1 only. We worked around it with a
symlink in the process's own library path.

Tell me if you want either of those filed on its own and I will write it up with logs.

Follow-up on the ocloc question, since I said above that we would re-run it.

We repeated the same single card vLLM run with the compiler matched to the driver instead of the
PPA's older one:

intel-opencl-icd / libze-intel-gpu1 26.35.39758.10-0
intel-igc-core-2 / intel-igc-opencl-2 2.41.5
intel-ocloc 26.35.39758.10-0 (from the 26.35 release deb, not the PPA)
libze1 1.32.0-1~~26.04~~ppa1
everything else as in the first post: vLLM 0.25.1+xpu, fp16 judge merge, --max-num-seqs 12,
two client processes, 12 requests in flight

Result: it still dies, and slightly sooner.

server ready 21:34:49
first fault 21:35:45 (about 56 s into serving, 2,292 requests completed)

21:35:45 Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x3
21:35:45 Timedout job: seqno=4294967199, lrc_seqno=4294967199, guc_id=2, flags=0x20 in python [3645744]
21:35:45 Xe device coredump has been created
21:35:45 Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=6, state=0x289
21:35:45 Fault response: Unsuccessful -ENOENT
userspace: RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
then vllm.v1.engine.exceptions.EngineDeadError

We kept the coredump from this run, so there is now one from the fully matched stack as well as the
earlier ones.

So the compiler being one release behind the driver was not the cause, and matching it does not
change the outcome. The missing ocloc for the 26.35 line is still a real packaging problem, it cost
us workarounds to start vLLM at all, but it is not this bug.

One more packaging detail from the same session, since it will bite anyone installing from the PPA.
Triton's Intel backend compiles a SPIR-V shim that includes level_zero/ze_api.h. That header only
ships in libze-dev, while the PPA installs libze1 (runtime) only, so the shim build fails with

triton/backends/intel/include/sycl_functions.h:12:10: fatal error:
level_zero/ze_api.h: No such file or directory

and it surfaces in vLLM as "Engine core initialization failed" with a g++ -lsycl -lze_loader link
error, which takes a while to trace back to a missing header. It only appears when the shim has to
be rebuilt (changing the ocloc version changes the cache key, which is how we hit it). We worked
around it by extracting the headers from the libze-dev deb into a private include directory and
pointing CPATH at it. Shipping those headers with the runtime package would remove a confusing
failure mode. Happy to file that on its own if you would rather track it separately.

One more comparison, this time at the userspace level rather than the driver level, because
upstream llama.cpp has since added a workaround that cites an issue in this tracker.

llama.cpp master now contains c9a5eeeb3 ("sycl: fix the B70 mem allocate error when >19.3GB", PR
#28953). It caps the SYCL buffer type maximum allocation on BMG-G31 at 60 percent of what the
driver reports:

if(is_bmg_g31_arch(device)) {
    //Todo, it's workaround for BMG-G31, which has a known issue with large allocations.
    //The max alloc size is reduced to 60% of the reported max alloc size.
    //remove it after https://github.com/intel/compute-runtime/issues/998 is fixed.
    max_alloc_size = max_alloc_size*0.6;
}

So we ran the same long context shape on both builds, alternating the order between rounds, on the
same card and the same driver 26.35.39758.10:

build A reference 10655 / cb300598d, built 2026-08-27, no B70 allocation cap
build B master 11042 / ec9281505, built 2026-09-18, includes the 60 percent cap
shape Qwen3.8-27B Q4_0, back-to-back 24,696 token prompts, 2 slots in flight, 900 s per window

window 1 reference clean 126 requests, 0 errors, 3,111,696 prompt tokens
window 2 master clean 115 requests, 0 errors, 2,840,040 prompt tokens
window 3 master FAULT 5 kernel lines
window 4 reference FAULT 4 kernel lines

window 3, master 22:25:17 Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x3
22:25:17 Timedout job: seqno=4294967190, lrc_seqno=4294967190, guc_id=2, flags=0x20 in llama-server [665062]
server process died, the client never completed a request
window 4, reference 22:44:04 Engine reset: engine_class=ccs, logical_mask=0x1, guc_id=2, state=0x3
22:44:04 Timedout job: seqno=1303, lrc_seqno=1303, guc_id=2, flags=0x20 in llama-server [964800]
23:00:39 Engine reset: engine_class=bcs, logical_mask=0x1, guc_id=6, state=0x289
23:00:39 Fault response: Unsuccessful -ENOENT

Each build passed one window and failed the next, so we stopped there. Coredumps were captured from
both faulting windows.

That is a negative result and it kills a hypothesis we had. The allocation cap does not prevent
these resets here. Taken with the earlier results in this thread (faults on 26.35 from two different
userspaces, and vLLM still dying with the compiler matched to the driver), the resets track neither
the driver version, nor the userspace build, nor the compiler version. On the evidence we have they
live in engine or queue management.

One caveat, and it cuts against our own earlier comment on #998: we cannot tell from our hardware
whether the 60 percent cap helps the case it was added for. We never reproduced the above 19.3GB
allocation failure on this card on 25.48 or 26.35, and the cap targets an allocation error, which is
a different symptom from the queue resets above.

One more elimination on 26.35.39758.10, and it closes the last lever we had.

Context: over in the llama.cpp thread (ggml-org/llama.cpp#27595 (comment))
the only setting that ever made this card stop faulting was disabling the Level Zero V2 submission
path (SYCL_UR_USE_LEVEL_ZERO_V2=0). On 26.27.39122.14 that gave a clean 20 minute window while the
stock path died at about 5.5 minutes, which is why the submission path got named as the trigger.

Re-ran it tonight on 26.35.39758.10, with IGC 2.41.5 and ocloc 26.35.39758.10 matched to the driver,
same single B70, same model and shape as the build A/B I posted above (Qwen3.8-27B Q4_0, back-to-back
24,696 token prompts, 2 slots in flight, 900 s window). Confirmed the flag was present in the server
process and that the work was genuinely resident on the card (24.1 GiB, 201 to 223 W) rather than a
CPU fallback. It faulted about 3.5 minutes in:

23:14:35  Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x3
23:14:35  Timedout job: seqno=1018, lrc_seqno=1018, guc_id=2, flags=0x20 in llama-server [1495371]
23:14:35  Xe device coredump has been created
23:24:57  Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=6, state=0x289
23:24:57  Fault response: Unsuccessful -ENOENT

Coredumps: b70_20260918_231435.bin for the ccs fault, b70_20260918_232457.bin for the bcs fault on
teardown of the window.

The client completed 33 requests with 2 errors before it went, and the server process itself survived
this window. So the off path is inside the same distribution as the on path here.

That makes four eliminations on our hardware: the resets follow neither the driver version, nor the
userspace build, nor the compiler version, nor the submission path. On this card whatever this is
sits below all four, and I have nothing left to bisect on it.

Box state afterwards, since these runs are reproducible: driver and ocloc 26.35.39758.10, device
rebound and enumerating as level_zero:gpu 1.17.39758+10, no test server running, and the unrelated
serving process on the other GPU in the machine untouched throughout.

@fbiricotti

4× Arc Pro B70, vLLM TP1 ×4: ablation (Graph / MTP / FP8 KV) plus a kernel A/B. Only the kernel change reduced the failures: 7.0.0-30 ≈ 1/h, 6.17.0-35 1 in 8 h, of a milder kind

Adding a controlled dataset, because this thread still lacks two things: an ablation of vLLM-side features, and a same-workload kernel comparison.

Environment


GPUs	4× Arc Pro B70 (`8086:e223`, subsystem `8086:1701`)
Host	AMD Threadripper PRO 5955WX, Gigabyte MC62-G40, Proxmox VE 9. GPUs passed through with VFIO to a Q35 guest
Guest	Ubuntu 24.04, kernel 7.0.0-30-generic vs 6.17.0-35-generic (the only variable in §3)
GuC	`bmg_guc_70.bin` 70.60.0 (dmesg: wanted 70.54.0), same on both kernels
Userspace (in container)	libze-intel-gpu1 / intel-opencl-icd 26.27.39122.11, IGC 2.38.2, libze1 1.32.0, torch 2.13.0+xpu
vLLM	`vllm/vllm-openai-xpu@sha256:f01e24f6…` (0.27.2rc1.dev77)
Workload	Qwen3.8-27B GPTQ-Int4 (G128) + MTP4 draft (BF16), FP8 KV, max-model-len 161000, gpu-memory-utilization 0.90, XPU Graph on. TP=1, one instance per card (`ZE_AFFINITY_MASK=N`), so no oneCCL and no cross-card traffic. Each card gets one stream in a loop: temp 0, 1024 output tokens, 8 rotating prompts

A watchdog on every card samples /metrics every 20 s. When a request is running but neither the generation nor the prompt token counter has moved for 90 s, it takes a py-spy dump --native before it kills the container.

1. Signature (kernel 7.0.0-30)

Two forms, both seen on all 4 cards:

Device lost: Engine memory CAT error [1]: class=ccs ×5–7 → Engine reset ccs state=0x249 → Timedout job … in VLLM::EngineCor → devcoredump → bcs reset → Faulted Address 0x0000d556aa4c7000 … EngineClass: 3 bcs → Fault response: Unsuccessful -ENOENT → UR_RESULT_ERROR_DEVICE_LOST.
Silent hang: the engine stops returning, and the kernel logs nothing until the process is killed. Meanwhile the card draws ~87–92 W (idle is ~45 W, load ~228 W), so a GPU kernel is running and never finishes. py-spy shows the host spinning in sched_yield under libze_intel_gpu.so.1.15.39122 → urEventWait (or ur_command_list_manager::appendCommandBufferExp from torch/xpu/graphs.py replay). This matches the stacks reported earlier in this thread.

Cross-card propagation happened twice, even with no inter-GPU traffic. gpu1 went device-lost at 01:53:42, and gpu2 stopped producing tokens within 12 s. On a later run, gpu3 went device-lost at 11:35:41, and gpu1 hung within 30 s. The instances are separate processes on separate cards. This matches @fbiricotti's "cascading resets on both cards", but without TP.

A caution for anyone reading dmesg: docker rm -f on a healthy instance in the middle of generating logs Engine reset ccs/bcs state=0x289 plus Faulted Address 0x0000d556a7… / -ENOENT / -EINVAL by itself. Killing an idle instance logs nothing. So bcs -ENOENT lines that appear at kill time only mean the GPU was busy, and they are not evidence of the fault. In every one of our incidents the bcs fault address was 0xd556aa4c7000, while the healthy-kill control gave 0xd556a74c0000. That is consistent with @marfrit's finding that these VAs are runtime-internal buffers (SEMAPHORE_BUFFER / COMMAND_BUFFER range) placed at the same address in every process.

2. Ablation on kernel 7.0.0-30: none of these vLLM features matters

Config (only the change from baseline is listed)	Hours	Requests done	Incidents	Per hour
Baseline	5.5	3918	6	1.1
`VLLM_XPU_ENABLE_XPU_GRAPH=0`	5	3113	3	0.6
No speculative decoding (MTP off)	5	2492	3	0.6
`--kv-cache-dtype auto` (no FP8 KV; max-model-len 80000)	5	3698	5	1.0

Every configuration produced both forms, spread across cards. Per prompt, 6 hangs fell on 5 of the 8 prompts (temp 0, ~120 repeats of each prompt per card), so the trigger is not deterministic.

3. Kernel A/B, same baseline config: 7.0.0-30 vs 6.17.0-35

	7.0.0-30 (4 runs, 20.5 h)	6.17.0-35 (8 h)
Incidents	17 (≈ 1/h)	1
`Engine memory CAT error` / `Timedout job`	present in every run	0 / 0
Page faults / `Fault response`	yes	none
Cross-card propagation	2×	none
The single 6.17 incident	—	one `Engine reset ccs` → devcoredump `Reason: LR job cleanup` → `DEVICE_LOST` on that instance only. The other 3 cards logged nothing all run; the watchdog restarted it in 4 min
Decode speed	~52 tok/s	~52.7 tok/s

With the 7.0 rate, 8 h should give about 8 incidents; seeing ≤1 by chance has p ≈ 0.004. This is consistent with @marcorigodanzo's datapoint above (6.17: resets recoverable, no Fault response; 7.0: faults with Fault response and permanent wedges).

A confound we have not ruled out: host uptime. The 6.17 run started right after a reboot (uptime 0–8 h). The 7.0 runs covered uptime 8–36 h, and the first 7.0 incident came at ~9.5 h. On this box the hang rate has grown with uptime before: in August, on 6.17.0-35, an identical workload passed at 24 min–12.9 h of uptime and hung at 22 h. An earlier matched-uptime comparison on this same VM (2026-08, UMD 26.27 on both kernels) points the same way: at 8 h uptime, 7.0.0-28 wedged on round 5 with page faults and a CAT error, while 6.17.0-35 completed 90 rounds with all counters at 0. We now run 6.17 in production with the watchdog and will follow up with the incident rate once uptime passes 24 h.

Artifacts available on request

devcoredumps: two from 7.0 (Timedout job seqno 909 / 1663), one from 6.17 (LR job cleanup)
native py-spy stacks captured before kill; full kernel logs per incident; 20 s power / temperature / token-counter samples for every run

Questions for the maintainers:

Is there a known xe change between 6.17 and 7.0 in page-fault handling or LR-queue handling that would turn a recoverable ccs reset into CAT error + -ENOENT + wedge?
Is GuC 70.60.0 on a kernel that wants 70.54.0 a supported combination, or should we pin 70.54.0 for this test?

Follow-up to my comment above: 6.17.0-35 past 24 h of uptime. The CAT / page-fault class never appeared; one silent hang did, and it looks like a separate bug

Same host and stack as above. After the kernel A/B the box went into production on 6.17.0-35-generic with the same vLLM config (4× TP1, MTP4, FP8 KV, XPU Graph on) and the same watchdog. It has now been up for 40 h.

One caveat first: production load is light. A request was running during only 3.4 % of the watchdog's 20 s samples, so none of the rates below can be compared with the fully loaded 7.0 soaks.

	kernel 7.0.0-30 (soaks, fully loaded)	kernel 6.17.0-35 (production, 40 h, light load)
`Engine memory CAT error` / `Timedout job`	in every run	0 / 0
Page faults / `Fault response` before any kill	yes	0
Cross-card propagation	2×	0
Silent hangs (GPU busy, no tokens, nothing in dmesg)	yes	1, on gpu0 at 18.3 h host uptime

The 6.17 hang:

Tokens stopped at 10:32:53. The card then sat at ~86 W (idle is ~47 W) with one request in flight, and the kernel logged nothing.

The watchdog confirmed the hang 2 min later and took a native py-spy dump before killing the container:

sched_yield (libc.so.6)
0x…c21a / 0x…ac4d (libze_intel_gpu.so.1.15.39122)
ur_command_list_manager::appendCommandBufferExp (libur_adapter_level_zero_v2.so.0)
replay (torch/xpu/graphs.py:107)
__call__ (vllm/compilation/cuda_graph.py:360)

The only xe lines are one ccs + bcs Engine reset at 10:35:05, and those are from the kill (see the kill-a-busy-instance control in my earlier comment).
The service was back at 10:39. Nothing since, now 22 h later.

Our reading: there seem to be two separate problems.

Kernel 7.0.x adds a fault-driven failure: CAT error → -ENOENT page fault → DEVICE_LOST, it propagates to other cards, and it shows up within hours under load. It has not occurred once on 6.17.
An uptime-dependent silent hang that is also present on 6.17: the host spins in libze_intel_gpu under appendCommandBufferExp / urEventWait, the GPU stays busy, and there is no dmesg signature at all. Its timing (first occurrence at 18.3 h) matches what we saw on this box in August on 6.17 (clean up to 12.9 h of uptime, hangs from ~22 h) and several of the stacks earlier in this thread. It may be the same issue some reporters here call "wedge without fault response".

If that split is right, it could explain why GuC and UMD bisects in this thread keep coming back mixed: they would be mixing two bugs that have different triggers.

A watchdog is enough for us to run on 6.17 in the meantime: the hang is detected in 90 s and service is back in ~4 min. We can run any debug build or ZE_* / NEOReadDebugKeys tracing you'd like on this box.

@SheffieldWu

Data point: 21 h of sustained TP=2 load on kernel 6.17.0-41, zero xe faults (2× Arc Pro B70)

Setup, same on both kernels: 2× Arc Pro B70 (BMG G31) on PCIe, no XeLink; EPYC 7413; Ubuntu 26.04; GuC 70.65.0.

7.0.0-27: a TP=2 vLLM run (upstream vLLM 0.26.1rc1, torch 2.13, oneCCL via xccl) failed at init with UR_RESULT_ERROR_DEVICE_LOST. Both cards faulted in the same second:

xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine memory CAT error [18]: class=ccs, logical_mask: 0x1, guc_id=13
xe 0000:c3:00.0: [drm] Tile0: GT0: Engine memory CAT error [18]: class=ccs, logical_mask: 0x1, guc_id=30
xe 0000:c7:00.0: [drm] Tile0: GT0: Timedout job: seqno=339, lrc_seqno=339, guc_id=13, flags=0x20 in python3

Single-GPU instances (TP=1) on the same kernel ran for about 69 days without any of these.

6.17.0-41 (latest 6.17 in the Ubuntu 25.10 archive): TP=2 with intel/llm-scaler-vllm:0.26.0-b2 (default oneCCL settings, --enforce-eager) on a 27B int4 model. About 21 h of load, ~2,050 requests, up to ~30 concurrent on 30–40k-token prompts, KV cache up to 93%: no CAT error, no fault response, no engine reset, no timed-out job, no DEVICE_LOST. Consistent with @SheffieldWu's report.

Caveat: not a clean kernel A/B. The 7.0 failure was on the upstream vLLM/oneCCL stack, the 6.17 run on llm-scaler; llm-scaler TP=2 on 7.0 only ran a short benchmark. One soak, one machine.

Another data point: same setup (2x Arc Pro B70, Ubuntu 26.04), kernel 7.0.0-31,
GuC 70.72.1, NEO 26.27.39122.11 / IGC 2.38.2, torch 2.13.0+xpu (ComfyUI, LTX-2.3 video).

≥40 fatal Engine memory CAT error + Engine reset ... state=0x249 since 2026-08-13,
on BOTH cards (03:00.0 and 07:00.0), devcoredumps saved for each.
Twice both cards faulted nearly simultaneously (2026-09-16 same second; 2026-09-28
64 s apart), and the follow-up bcs page fault was at the identical address
(0xd556aa4c7000) on both cards — pointing at the workload, not a single bad card.
Ruled out: iommu=pt (all faults above happened with passthrough), host memory
pressure (sar: 82 GB available, zero page scanning at fault time), PCIe.
After DEVICE_LOST, a freshly started process on the same card can hang silently
(no kernel message, no progress) for 10+ minutes; ~13 min later the card works again.
Also checked [GSD-13180] DEVICE_LOST / exec-queue-reset storms on Arc Pro B70 (BMG G31) silently corrupt host page-cache pages of ordinary files — host-wide data integrity, survives FLR/bus-reset/unbind #966: page-cache vs on-disk sha256 of our weights still match.

Happy to share devcoredumps if useful.

A narrower data point: the load-time bcs / -EINVAL variant is the copy engine reading a temporary EXTERNAL_HOST_PTR mapping, it only exists for single host<->device copies of 512 MiB or more, and splitting those copies avoids it

This covers one class only: copy-engine page faults while a model loads. It is not the under-load ccs reset others describe here, and not the teardown noise after a kill. Posting because the address turned out to be fully explainable and the workaround is cheap.

Setup

2x Arc Pro B70 (8086:e223), EPYC 8-core, 15 GiB RAM, Ubuntu 24.04, xe, GuC 70.54.0. Kernels 7.0.0-31 and 7.0.0-38. vLLM 0.29 XPU, TP=2 (oneCCL) and TP=1, in a container with compute-runtime 26.27.39122.11 / libze1 1.32.0 / torch 2.13.0+xpu; also reproduced the allocation behaviour on the host with 26.22.38646.7 and torch 2.14.0+xpu. PYTORCH_ALLOC_CONF=expandable_segments:True.

The signature (four saved incidents, both cards, both kernels)

Always during weight load, always the same GPU VA range and nearly the same page order:

xe 0000:e3:00.0: [drm] Tile0: GT0:
	ASID: 1146
	Faulted Address: 0x0000800400213000
	FaultType: 0
	AccessType: 0
	FaultLevel: 1
	EngineClass: 3 bcs
xe 0000:e3:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
... (27-45 of these, all inside 0x800400200000-0x800400228000)
xe 0000:e3:00.0: [drm] Tile0: GT0: Engine memory CAT error [18]: class=bcs, logical_mask: 0x1, guc_id=18
xe 0000:e3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=18, state=0x249
xe 0000:e3:00.0: [drm] Tile0: GT0: Timedout job: seqno=4294967260, ... in python3

Devcoredump: IPEHR 0x13000203, ACTHD 0x3c into a batch at a high heap address. (Practical note: the fault record is one multi-line kernel message, so journalctl -o short plus grep keeps only its empty first line. journalctl -k -o cat shows the address.)

What that address is

In xe_pagefault_service() (v7.0), -EINVAL is returned when the VM is not in fault mode or when xe_vm_find_vma_by_addr() finds no VMA. NEO creates its VM with LR_MODE | FAULT_MODE here, so this is "no VMA at the faulting address".
0x800400200000 is HEAP_STANDARD base plus the 2 MiB granularity, i.e. the first slot the heap allocator hands out for allocations above its 4 MiB threshold.
With NEOReadDebugKeys=1 LogAllocationType=1 LogAllocationStdout=1 PrintBOBindingResult=1 the runtime names the object:

Created Graphics Allocation of type EXTERNAL_HOST_PTR
 ... Type: EXTERNAL_HOST_PTR Pool: System4KBPages Root index: 1 Size: 1271398400 CPU VA: 0x75644437f040 - 0x75648ffff03f GPU VA: 0xffff800400200040 - 0xffff80044be8003f ...
bind BO-0 to VM 1, vmHandleId = 0, range: ffff800400200000 - ffff80044be81000, size: 1271402496, result: 0
unbind BO-0 from VM 1, vmHandleId = 0, range: ffff800400200000 - ffff80044be81000, size: 1271402496, result: 0

1,271,398,400 bytes is the model's 124,160 x 5,120 FP16 embedding / LM-head shard. A TP=2 start makes exactly eight of these, at the end of the main weight load and during the draft-model load, which is exactly where every fault landed.

So the event is: the copy engine reads the temporary userptr mapping of the host source buffer for a host-to-device copy, and there is no VMA there at that instant.

The size threshold

One-card probe, a plain cpu_tensor.to('xpu') of N MiB, counting EXTERNAL_HOST_PTR creations in the allocation log (same result on 26.27 and 26.22):

Copy size	`EXTERNAL_HOST_PTR` created
2, 8, 64, 256, 257, 300, 384, 448, 496, 500, 511 MiB	0
512, 1024, 1212 MiB	1 each, at GPU VA `0x...800400200040`

Device-to-host copies of that size make the same mapping. Smaller copies go through BUFFER_HOST_MEMORY staging and never touch this path.

Workaround that removes the path

Split any host<->device copy of 512 MiB or more into smaller pieces (we use 128 MiB slices of a flat view). With that:

allocation log: 0 EXTERNAL_HOST_PTR mappings during the whole start (control: 8 on TP=2, 3 on TP=1 where one is a 2.5 GB device-to-host move);
outputs bit-identical (weights compare equal; our 12-prompt token-identity gate passes; a video pipeline's clip hashes are unchanged);
load time unchanged, the 1.27 GB upload itself went from 0.14 s to 0.08 s.

I cannot give a before/after fault rate: with the container not swapping, the fault was about 1 in 59 starts, so counting would take hundreds of starts. The claim is only that every saved fault was a read of that mapping and the mapping is no longer created.

Things that did not help

ExperimentalH2DCpuCopyThreshold=2147483647 (with NEOReadDebugKeys=1): no effect, same mappings. preferCopyThroughLockedPtr() apparently does not classify torch's destination as device USM under expandable_segments.
ExperimentalForceCopyThroughLock=1: segfault.

Timing sensitivity

With the container allowed to swap at its own memory limit (--memory 12g --memory-swap 16g, 29 GB of weights streaming through its page cache) this exact fault hit 3 times in 4 days. With --memory-swap equal to --memory it dropped to 1 in 59 starts. Host memory pressure seems to widen whatever window this is; it is not the cause.

What I have not determined

Which side drops the mapping while the blit is still reading: the runtime's release of the temporary host-pointer allocation (the shared temporary-allocation list cleaned by cleanTemporaryAllocations, across the main and copy CSRs), or something in the kernel. I read the release logic and could not find the hole by inspection. Happy to share the four devcoredumps and the allocation logs, or to run a specific debug key if one would discriminate.

Two-card FP8 package re-accepted with the chunked-upload overlay (12/…

Data point: one Arc Pro B70, GuC 70.55.3 → 70.72.1. The visible corruption stopped, but one ccs engine reset recurred under 70.72.1.

Single Arc Pro B70 (BMG G31, xe, 0000:06:00.0), no second GPU in use, Debian sid, 7.2-series kernel. Both firmware versions were tested on the same machine on 2026-09-15.

The probe: a short completion ("The capital of France is"), then a ~26k-token prefill, then the short completion again, repeated 8 times per run. Each answer is classified as language or not.

GuC	Configuration	Probes returning language	Engine resets in the boot
70.55.3	flash attention on, q8_0 KV cache	0 of 16	3 × `engine_class=ccs ... state=0x209` in 2.6 h
70.55.3	flash attention off, f16 KV cache	none (device lost before the first answer)	(same boot)
70.72.1	flash attention on, q8_0 KV cache	16 of 16
70.72.1	the same, greedy sampling	16 of 16
70.72.1	flash attention off, f16 KV cache	16 of 16
70.72.1	flash attention on, q8_0, ~49k-token prefill	6 of 6

70.72.1 did not remove the reset. On the same 70.72.1 boot, 13 h after these runs, the kernel logged the same signature during a llama-batched-bench run:

GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2, state=0x209
GT0: Timedout job: seqno=4294967195, lrc_seqno=4294967195, guc_id=2, flags=0x0 in llama-batched-b

Since then, 15 boots and about 600 h of uptime on 70.72.1 have logged no further ccs engine reset. That is uptime, not sustained load.

One caveat: the 09-15 probes ran on llama.cpp's Vulkan backend, not Level Zero. The 70.55.3 resets above are xe kernel messages logged against that Vulkan llama-server process, so I'm posting this as a firmware data point rather than a compute-runtime one.

Follow-up to our 6.17 data point (2× Arc Pro B70, TP=2): 14 days, ~32,000 requests, zero xe events

Same setup as in #948 (comment): kernel 6.17.0-41-generic, GuC 70.65.0, intel/llm-scaler-vllm:0.26.0-b2 with TP=2 and --enforce-eager, now in production.

14 days of continuous uptime (since 2026-09-26), about 32,000 requests: production traffic plus long replays at up to ~30 concurrent requests with 30–40k-token prompts.
dmesg: no CAT error, no Fault response, no engine reset, no timed-out job, no DEVICE_LOST.
7 container restarts (model swaps, docker stop -t 60): none produced a reset either.

The other recent reports here are all on 7.x (7.0.0-31, 7.0.0-38, 7.2), with GuC 70.55 to 70.72.1. Same caveat as before: not a clean kernel A/B, since our 7.0 failure was on a different stack; one machine.

Performance: 6.17 is not a trade-off

Same stack, model and prompt (27B bf16, TP=2, llm-scaler 0.26.0-b2 eager, real 28,882-token prompt, single stream): 11.04 tok/s decode on 7.0.0-27 vs 10.49–10.91 tok/s on 6.17.0-41 (TTFT 8.8 s), within run-to-run noise.
Production on 6.17 (27B int4 AutoRound, TP=2, eager), generation throughput reported by the engine over 5 days: median ~156 tok/s (p90 ~198, peak 272) with 21–32 requests running, ~104 tok/s with 13–20, ~9 tok/s single-stream.

[GSD-13010] Sporadic permanent GPU wedge on dual Arc Pro B70 (BMG G31): ccs/bcs engine reset + "Fault response: Unsuccessful -ENOENT/-EINVAL" under sustained Level-Zero inference load #948

Description

Pre-submission Checklist

System Information

GPU Hardware

DRI Devices Information

GPU Detailed Information (lspci output)

Driver Version

Installed GPU Driver Packages

Driver Installation Details

Linux Distribution

Other Linux Distribution

Kernel Version & Boot Parameters

Issue Description

Actual Behavior

Expected Behavior

Reproduction Rate

Steps to Reproduce

Regression Information

Is this a regression?

Logs

API Call Logs

strace Logs

System Logs / dmesg Output

Backtrace (if crash or hang occurred)

Reproducer

Source Code / Reproducer

Activity

marcorigodanzo commented on Jul 6, 2026

davetha commented on Jul 19, 2026

Environment

Same signature, without any multi-GPU/oneCCL involvement

Firmware A/B — this is the main point

How to upgrade the GuC firmware (what we did)

AMD-host mitigation (both reporters here are on EPYC)

Happy to provide

marfrit commented on Sep 4, 2026

ralphsanders2001-ai commented on Sep 12, 2026

fbiricotti commented on Sep 13, 2026

Cross-report: 4-way matrix (2 card pairs × 2 vLLM XPU images × 2 GuC firmwares), all 4 combinations reproduce this exact signature — plus a controlled card/slot swap

Environment

The matrix

Notes relevant to points already raised in this thread

fbiricotti commented on Sep 16, 2026

Two fault instances (both first-fault captures, not teardown noise)

Kernel-side reading (mainline v7.0 sources)

GuC event-log terminal pattern (structural decode only)

Questions

localyouser commented on Sep 18, 2026

NateHag commented on Sep 19, 2026

NateHag commented on Sep 19, 2026

NateHag commented on Sep 19, 2026

NateHag commented on Sep 19, 2026

SheffieldWu commented on Sep 24, 2026

Environment

1. Signature (kernel 7.0.0-30)

2. Ablation on kernel 7.0.0-30: none of these vLLM features matters

3. Kernel A/B, same baseline config: 7.0.0-30 vs 6.17.0-35

Artifacts available on request

SheffieldWu commented on Sep 26, 2026

marcorigodanzo commented on Sep 27, 2026

kevinwatt commented on Sep 29, 2026

steveseguin commented on Oct 4, 2026

Setup

The signature (four saved incidents, both cards, both kernels)

What that address is

The size threshold

Workaround that removes the path

Things that did not help

Timing sensitivity

What I have not determined

RichAyotte commented on Oct 8, 2026

marcorigodanzo commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone