[BMG] Sporadic permanent compute wedge on dual Arc Pro B70: xe ccs/bcs engine reset + "Fault response: Unsuccessful -ENOENT/-EINVAL" under sustained Level-Zero inference load #946

Under sustained multi-request LLM inference on dual Arc Pro B70 (tensor-parallel=2 via vLLM + oneCCL), we hit a sporadic permanent GPU wedge every 2–6 hours: the xe driver reports ccs/bcs engine resets plus Fault response: Unsuccessful -ENOENT/-EINVAL, and the userspace Level-Zero context never recovers (only a process restart does). Below: full environment, a per-second correlation table between application failures and dmesg events on both cards, and everything we ruled out.

Related field report on identical hardware: vllm-project/vllm#41663. Same ccs-reset signature without vLLM/oneCCL involved: darktable-org/darktable#20257 — which is why we are filing here rather than with vLLM.

Environment

oneCCL "stable fallback" env (from vllm-project/vllm#41663) already applied in all runs:

CCL_ENABLE_SYCL_KERNELS=0
CCL_ATL_TRANSPORT=ofi
CCL_ZE_IPC_EXCHANGE=sockets
CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0
SYCL_UR_USE_LEVEL_ZERO_V2=0

These removed the earlier ~50-min internal_kvs/broken-pipe crash cadence, but do not eliminate the wedge below.

Symptom

Under sustained concurrent LLM inference (8–12 concurrent chat completions, mixed 4k–32k contexts), roughly once every 2–6 hours the GPU work on one card stops completing. In dmesg, at the exact same second:

xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=29, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL

After the engine reset the userspace context is permanently wedged: the in-flight SYCL kernel never completes, the vLLM TP worker never returns, and the serving engine dies with TimeoutError: RPC call to sample_tokens timed out → EngineDeadError. Only a full process/container restart recovers.

Both cards are affected (events observed on c3:00.0 and c7:00.0), so this is not a single defective unit.

Crash ↔ dmesg correlation (all times UTC, July 2–4 2026)

Key datapoint: the dying step is tiny

vLLM's crash dump for the 07-04 08:51 event shows the scheduler step that never returned was 9 requests in decode, total_num_scheduled_tokens=9 (nine tokens), KV cache usage 14%:

SchedulerOutput(..., num_scheduled_tokens={...: 1, ...: 1, ...}, total_num_scheduled_tokens=9, ...)
SchedulerStats(num_running_reqs=9, num_waiting_reqs=0, kv_cache_usage=0.1439...)

So this is not a batch-size / prefill-pressure problem: an ordinary small decode step faults. Incidence appears to scale with the number of steps executed (hours of sustained work × concurrency), not with step size.

Ruled out on our side

• Model / quant kernels — reproduces with three different models & quant paths: Llama-3.3-70B (AWQ int4), Qwen3.6-27B (FP8), Qwen3.6-27B (Intel AutoRound int4 / INC path).
• vLLM version — reproduces on intel/vllm 0.17.0-xpu prebuilt AND upstream 0.24.0 source build.
• Graph capture — --enforce-eager everywhere.
• Guided decoding backend — xgrammar vs llguidance makes no difference to the wedge.
• vLLM async scheduling — disabled (--no-async-scheduling), still reproduces.
• Scheduler oversubscription — capped with --max-num-seqs 12, still reproduces.
• Kernel version — original report (#41663) was on 6.17 HWE; we reproduce on 7.0.0-27.

Independent confirmation it is not vLLM/oneCCL-specific

• [BUG] Tumbleweed + Intel Arc B580: OpenCL LAPLACIAN_ASSEMBLE triggers Xe Engine Reset (GT0: Engine reset: engine_class=ccs) darktable-org/darktable#20257 — same Engine reset: engine_class=ccs on Arc B580 triggered by OpenCL image kernels (no vLLM, no oneCCL, single GPU).
• [Bug]: XPU TP=2 on dual Intel Arc Pro B70 (Battlemage): GP fault + xe BCS engine reset reproduces in intel/vllm:0.17.0-xpu on Ubuntu 24.04 HWE 6.17 vllm-project/vllm#41663 — same dual-B70 hardware, GP fault + bcs engine reset (we are posting a confirming comment there referencing this issue).

Reproduction

Sustained concurrent load against a vLLM server for several hours reliably triggers it:

• 2× B70, TP=2, eager, any of the models above;
• 8–12 concurrent /v1/chat/completions (structured output via response_format json_object, mixed context sizes 4k–32k);
• expect an engine reset within 2–6 hours (sooner under heavier mix).

We can provide our load-generation script, full container logs, and full dmesg on request.

Questions

1. Is this signature (state=0x289 ccs/bcs reset + Fault response: Unsuccessful -ENOENT/-EINVAL) already known for BMG?
2. Is there a newer GuC firmware / compute-runtime with a relevant fix (host is on GuC 70.58.0 / 26.05.37020.3)?
3. What additional traces would help? We can capture py-spy/gdb stacks of the hung worker, ze_debug logs, xe debugfs state at wedge time — the incident is frequent enough to instrument.