Repository navigation
[GSD-13189] Hang on exit in llama.cpp after memory overcommit on Intel B70 #967
Description
Pre-submission Checklist
- I am using the latest GPU driver version (releases)
- I have searched for similar issues and found none
GPU Hardware
Intel Arc Pro B70
DRI Devices Information
$ ls -ls /dev/dri/*
0 crw-rw----+ 1 root video 226, 1 Aug 3 17:48 /dev/dri/card1
0 crw-rw----+ 1 root video 226, 2 Aug 3 17:48 /dev/dri/card2
0 crw-rw-rw-. 1 root render 226, 128 Aug 2 21:43 /dev/dri/renderD128
0 crw-rw-rw-. 1 root render 226, 129 Aug 2 21:43 /dev/dri/renderD129
/dev/dri/by-path:
total 0
0 lrwxrwxrwx. 1 root root 8 Aug 2 21:43 pci-0000:03:00.0-card -> ../card1
0 lrwxrwxrwx. 1 root root 13 Aug 2 21:43 pci-0000:03:00.0-render -> ../renderD128
0 lrwxrwxrwx. 1 root root 8 Aug 2 21:43 pci-0000:19:00.0-card -> ../card2
0 lrwxrwxrwx. 1 root root 13 Aug 2 21:43 pci-0000:19:00.0-render -> ../renderD129
$ ls -la /dev/dri/by-path/
total 0
drwxr-xr-x. 2 root root 120 Aug 2 21:43 .
drwxr-xr-x. 3 root root 140 Aug 2 21:43 ..
lrwxrwxrwx. 1 root root 8 Aug 2 21:43 pci-0000:03:00.0-card -> ../card1
lrwxrwxrwx. 1 root root 13 Aug 2 21:43 pci-0000:03:00.0-render -> ../renderD128
lrwxrwxrwx. 1 root root 8 Aug 2 21:43 pci-0000:19:00.0-card -> ../card2
lrwxrwxrwx. 1 root root 13 Aug 2 21:43 pci-0000:19:00.0-render -> ../renderD129
GPU Detailed Information (lspci output)
$ sudo lspci -vvv -k -s 0000:03:00.0
03:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupts: unknown pin routed to IRQ 196, MSI(X) routed to IRQ 196 19:57 [37/605]
IOMMU group: 16
Region 0: Memory at f800000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at f000000000 (64-bit, prefetchable) [size=32G]
Expansion ROM at f5c00000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
DDROCCAP+ OCEn- DDRWrtVrefEn+ DDR3LEn+
CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
DDR3MaxFreq=2932MHz LPDDR3En-
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Kernel driver in use: xe
Kernel modules: xe
Driver Version
26.18.38308.4
Installed GPU Driver Packages
$ sudo rpm -qa | grep -iE "igc|gmm|opencl|level-zero|level_zero|ocloc|libze"
libsigc++30-3.6.0-6.fc43.x86_64
wine-opencl-11.0-2.fc43.x86_64
wine-opencl-11.0-2.fc43.i686
cuda-opencl-12-9-12.9.19-1.x86_64
intel-gmmlib-22.10.0-2.fc43.x86_64
intel-opencl-clang-15.0.9-2.fc43.x86_64
libsigc++20-2.12.2-1.fc43.x86_64
oneapi-level-zero-1.28.6-1.fc43.x86_64
oneapi-level-zero-devel-1.28.6-1.fc43.x86_64
intel-level-zero-gpu-raytracing-1.2.0-2.fc43.x86_64
oneapi-level-zero-zello_world-1.28.6-1.fc43.x86_64
intel-igc-2.34.4-2.fc43.x86_64
intel-igc-libs-2.34.4-2.fc43.x86_64
intel-level-zero-26.18.38308.4-2.fc43.x86_64
intel-ocloc-26.18.38308.4-2.fc43.x86_64
intel-opencl-26.18.38308.4-2.fc43.x86_64
Driver Installation Details
Distribution: Fedora 43
Regalar dnf updates.
Linux Distribution
Fedora / RHEL
Other Linux Distribution
Fedora 43
Kernel Version & Boot Parameters
$ uname -r
7.1.5-101.fc43.x86_64
$ cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt2)/vmlinuz-7.1.5-101.fc43.x86_64 root=UUID=bdad322a-efcb-4f9d-94fc-10fead02837c ro rootflags=subvol=root rhgb quiet elevator=none amdgpu.modeset=1
$ lsmod | grep -E 'i915|xe'
xe 4993024 8
intel_vsec 28672 2 pmt_telemetry,xe
drm_gpuvm 57344 1 xe
drm_gpusvm_helper 61440 1 xe
drm_buddy 12288 2 amdgpu,xe
i2c_algo_bit 20480 2 amdgpu,xe
drm_ttm_helper 20480 3 amdgpu,xe
ttm 163840 3 amdgpu,drm_ttm_helper,xe
drm_exec 12288 3 drm_gpuvm,amdgpu,xe
gpu_sched 73728 2 amdgpu,xe
drm_suballoc_helper 24576 2 amdgpu,xe
drm_display_helper 364544 2 amdgpu,xe
video 86016 2 amdgpu,xe
cec 106496 3 drm_display_helper,amdgpu,xe
Actual Behavior
Disclosure: 26.18.38308.4 is the newest driver packaged for Fedora; the 26.27.39122.11 release ships only .deb packages, so I have not been able to test on it. I did scan the 26.18.38308.4 -> 26.27.39122.11 commit range (673 commits) by subject and found nothing that matches the path described below.
With llama.cpp, when allocating too much VRAM (such as when attempting to increase context size), the process detects the over-commit and then hangs in exit().
The process exit spins forever at 100% CPU. Backtrace:
#0 NEO::DrmDirectSubmission<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::ensureRingCompletion()
#1 DirectSubmissionHw<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::stopRingBuffer(bool)
#2 CommandStreamReceiverHw<Xe2HpgCoreFamily>::stopDirectSubmission(bool, bool)
#3 NEO::Device::stopDirectSubmissionAndWaitForCompletion()
#4 L0::Device::releaseResources()
#5 L0::Device::~Device()
#6 L0::DriverHandle::~DriverHandle()
#7 _dl_fini / exit()
No timeout, no error propagation. SIGKILL is required; the GPU recovers after the kill.
Expected Behavior
The function call should return and the application should exit cleanly.
Reproduction Rate
Always reproduces - 100%
Steps to Reproduce
Compile and run the attached program.
Is this a regression?
- Yes, this is a regression - functionality that previously worked is now broken
Last Known Working Driver Version
No response
First Known Failing Driver Version
No response
API Call Logs
No response
strace Logs
No response
System Logs / dmesg Output
$ sudo dmesg --since '4 hours ago' | grep -i -E 'i915|xe|drm|gpu' | tail -n 100
[70796.532842] traps: l0-overcommit-r[508307] general protection fault ip:7f0e9f17a7ed sp:7fff2af74c50 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7f0e9f000000+7f4000]
[70813.099289] traps: l0-overcommit-r[508502] general protection fault ip:7fbc7157a7ed sp:7ffcb3efd250 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7fbc71400000+7f4000]
[71030.507820] traps: l0-overcommit-r[510098] general protection fault ip:7fc32837a7ed sp:7ffc51150e70 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7fc328200000+7f4000]
[71199.429479] traps: l0-overcommit-r[511201] general protection fault ip:7f0a8ff7a7ed sp:7ffd462b3420 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7f0a8fe00000+7f4000]
[73746.587315] traps: l0-overcommit-r[534516] general protection fault ip:7f525f77a7ed sp:7ffe18c717a0 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7f525f600000+7f4000]
[73775.132545] traps: l0-overcommit-r[534728] general protection fault ip:7fbc4257a7ed sp:7fff1ef315d0 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7fbc42400000+7f4000]
Backtrace (if crash or hang occurred)
#0 NEO::DrmDirectSubmission<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::ensureRingCompletion()
#1 DirectSubmissionHw<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::stopRingBuffer(bool)
#2 CommandStreamReceiverHw<Xe2HpgCoreFamily>::stopDirectSubmission(bool, bool)
#3 NEO::Device::stopDirectSubmissionAndWaitForCompletion()
#4 L0::Device::releaseResources()
#5 L0::Device::~Device()
#6 L0::DriverHandle::~DriverHandle()
#7 _dl_fini / exit()
Source Code / Reproducer
Compile and run the attached program.
Command Line / Application Details
No response
oneAPI Version (if applicable)
No response
Screenshots / Video
No response
Additional Notes
If this is related to #912 then this is a regression as I am on a later version of the driver.
Activity
[-]Hang on exit in llama.cpp after memory overcommit on Intel B70[/-][+][GSD-13189] Hang on exit in llama.cpp after memory overcommit on Intel B70[/+]kgibala commentedon Aug 4, 2026
Hi @rriggs,
Thank you for your contribution.
We will route this issue internally and provide an update as soon as possible.
Thanks again for bringing this to our attention.
rriggs commentedon Aug 22, 2026
@kgibala what is the status of this bug report? Has this been reproduced? Has it been fixed internally?
kgibala commentedon Aug 24, 2026
Hi @rriggs,
Thanks for the reproducer. We have not been able to reproduce this on Battlemage with your kernel (
7.1.5-101.fc43) and an identical userspace stack, so we need a bit more from your side.1. Does the reproducer crash, or hang? Your dmesg shows six general protection faults in
libze_intel_gpu, all at the same offset0x17a7ed, with the faulting process reported asl0-overcommit-r. That looks like it could bel0-overcommit-reprotruncated to the kernel's 15-charactercommlimit, but we do not want to assume that. Do those faults come from the reproducer? And were they from the same session as the llama.cpp hang, or from separate attempts?Related, and we should not guess at this either. Which binary produced the
ensureRingCompletion->_dl_finibacktrace? It appears in the report just after the description of llama.cpp hanging on exit, but the report does not actually say where it was captured from, so we do not want to read an attribution into it. Was it llama.cpp, the reproducer, or have you seen it from both?Between that and the faults above, the crash and the hang could be one failure or two, and your answers decide which. Could you also tell us what the reproducer does on your machine - crash, hang, or clean error - and paste its full stdout/stderr? If it segfaults, a backtrace would be ideal:
2. The exact command line you run, verbatim - the full shell line, including any environment-variable prefixes and arguments. We would rather not guess or steer you here, since the reproducer's behaviour changes substantially with its arguments.
3. Single vs. both devices. Your report lists two render nodes (
renderD12803:00.0,renderD12919:00.0). Is 19:00.0 a second Intel GPU or the AMD one? Which device does>> usingselect for you? And does the reproducer run against both devices visible, or do you already restrict it somehow? If you have not tried it, could you retry pinned to one device at a time - your normal invocation, just withZE_AFFINITY_MASK=0and thenZE_AFFINITY_MASK=1?4. Is your setup still as reported? The report is from 4 Aug, so before we read too much into version differences, are you still on kernel 7.1.5-101.fc43 with driver 26.18.38308.4,
intel-igc2.34.4,gmmlib22.10.0 andoneapi-level-zero1.28.6 - or has anything been updated since? The command in point 5 assumes those versions, so please tell us if it does not apply.5. One combined log. Please use your own invocation and just prefix it with the variables below. A single run keeps the driver, loader and validation-layer messages on one timeline:
set -xrecords your exact invocation at the top of the log, so point 2 is covered automatically. Expect a large file - please compress it. WhetherERROR: GPU HANG detected!appears anywhere in it is the single most informative detail.If the run hangs rather than exits, please also grab
gdb -p <pid> -batch -ex "thread apply all bt"while it is still spinning.Thank you for your collaboration.
kgibala commentedon Oct 5, 2026
Hi @rriggs,
We’d like to know if this issue is still affecting you. If so, please provide an update or any additional information. Otherwise, we’ll close this issue after 30 days of inactivity. Your feedback is appreciated.