Eval bug: SYCL multi-GPU crash with Intel Arc Pro B50 + Arc A770

[nicolataibi]

Name and Version

llama.cpp build: b10644-d7a207411

The executable was built locally from the ggml-org/llama.cpp repository.

version: 0.3.0-dev (build 10644, commit d7a2074)
built with IntelLLVM 2026.1.0 for Linux x86_64

Operating systems

Linux

GGML backends

SYCL

Hardware

Operating systems
Fedora Linux x86_64
Kernel: 7.1.10-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Aug 23 16:15:11 UTC 2026 x86_64 GNU/Linux

GGML backends
GGML SYCL
Intel Level Zero

CMake configuration:

GGML_SYCL=ON
GGML_SYCL_TARGET=INTEL
GGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON
GGML_SYCL_HOST_MEM_FALLBACK=ON
GGML_SYCL_DNN=ON
GGML_SYCL_F16=ON
GGML_SYCL_GRAPH=ON

The issue appears to affect the SYCL multi-GPU path. Single-GPU SYCL inference works correctly.
Hardware
CPU:
AMD Ryzen 9 5950X (16 cores / 32 threads)

GPU 0:
Intel Arc Pro B50
16 GB VRAM
SYCL device: SYCL0
Reported memory: 16304 MiB
Reported free memory: approximately 16220 MiB

GPU 1:
Intel Arc A770
16 GB VRAM
SYCL device: SYCL1
Reported memory: 15473 MiB
Reported free memory: approximately 14910 MiB

The two GPUs are from different Intel GPU generations:

Arc Pro B50: Battlemage
Arc A770: Alchemist

Both devices are detected correctly by llama.cpp:

Available devices:
SYCL0: Intel(R) Arc(TM) Pro B50 Graphics (16304 MiB, 16220 MiB free)
SYCL1: Intel(R) Arc(TM) A770 Graphics (15473 MiB, 14910 MiB free)

Models

Model:
gemma-4-12b-it-UD-Q6_K_XL.gguf

Quantization:
Q6_K

Parameters:
approximately 12B

The model is an Unsloth GGUF quantization.

The model loads and runs correctly when using either GPU individually.

Problem description & steps to reproduce

The problem occurs when attempting to use the Intel Arc Pro B50 and Intel Arc A770 simultaneously through the SYCL backend.

Both devices are correctly detected:

Available devices:
SYCL0: Intel(R) Arc(TM) Pro B50 Graphics (16304 MiB, 16220 MiB free)
SYCL1: Intel(R) Arc(TM) A770 Graphics (15473 MiB, 14910 MiB free)

The following environment was used for the multi-GPU tests:

env -u ONEAPI_DEVICE_SELECTOR
-u SYCL_DEVICE_FILTER
-u SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE
-u ZE_ENABLE_VALIDATION_LAYER
-u ZE_ENABLE_TRACING_LAYER
GGML_SYCL_USM_SYSTEM=1
GGML_SYCL_ENABLE_VMM=0
ZES_ENABLE_SYSMAN=1
UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1
~/dev/intel/llama.cpp/build/bin/llama-cli
--model "/run/media/nick/KINGSTON/AI/gemma-4-12b-it-UD-Q6_K_XL.gguf"
--device SYCL0,SYCL1
--split-mode layer
--ctx-size 1024
--batch-size 32
--threads 4
--threads-batch 4
-p "Hello"

This results in:

level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)

Exception caught at:
ggml/src/ggml-sycl/ggml-sycl.cpp:5513

func: operator()

SYCL error:
CHECK_TRY_ERROR((stream)->memcpy(
data,
(const char *)tensor->data + offset,
size
))

in function:
ggml_backend_sycl_get_tensor_async

Importantly, the model works successfully on each GPU individually.

B50 only:

~/dev/intel/llama.cpp/build/bin/llama-cli
--model "/run/media/nick/KINGSTON/AI/gemma-4-12b-it-UD-Q6_K_XL.gguf"
--device SYCL0
--split-mode none
--ctx-size 1024
--batch-size 32
--threads 4
--threads-batch 4
-p "Hello"

Result: works successfully.

A770 only:

~/dev/intel/llama.cpp/build/bin/llama-cli
--model "/run/media/nick/KINGSTON/AI/gemma-4-12b-it-UD-Q6_K_XL.gguf"
--device SYCL0
--split-mode none
--ctx-size 1024
--batch-size 32
--threads 4
--threads-batch 4
-p "Hello"

Result: works successfully when SYCL0 is mapped to the A770 through the corresponding device-selection environment.

Additional multi-GPU split-mode tests produced different failures:

--split-mode layer
UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY

The failure occurs inside:

ggml_backend_sycl_get_tensor_async()
ggml-sycl.cpp:5513
--split-mode row

Result:

Segmentation fault (core dumped)
--split-mode tensor

Result:

GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) failed

ggml-backend-meta.cpp:543

The following mitigations were also tested:

GGML_SYCL_ENABLE_VMM=0
GGML_SYCL_USM_SYSTEM=1
ZES_ENABLE_SYSMAN=1
UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1

None resolved the multi-GPU problem.

The following device-selection variables were also explicitly unset during testing:

ONEAPI_DEVICE_SELECTOR
SYCL_DEVICE_FILTER
SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE
ZE_ENABLE_VALIDATION_LAYER
ZE_ENABLE_TRACING_LAYER

This did not resolve the issue.

Therefore, the current evidence suggests that:

both Intel GPUs are correctly detected;
both GPUs can independently execute the same model;
the same model and quantization work independently on each device;
the failure is introduced when llama.cpp attempts multi-device execution;
different multi-GPU split modes fail in different ways;
the layer split reaches a Level Zero out-of-device-memory error during a tensor memory copy;
row results in a segmentation fault;
tensor reaches a GGML backend assertion.

I would appreciate guidance on whether this multi-GPU configuration is currently expected to work with SYCL/Level Zero when combining an Intel Arc Pro B50 (BMG) and an Intel Arc A770 (DG2), or whether there is a known limitation in the current SYCL backend.

First Bad Commit

Unknown.

The issue has not yet been bisected to an earlier commit.

The current reproducible version is:

d7a2074 (HEAD -> master, tag: b10644, origin/master, origin/HEAD)
models : support nanbeige4.2-3B (#27730)

Relevant log output

The most relevant failure is:

level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)

Exception caught at file:
/home/nick/dev/intel/llama.cpp/ggml/src/ggml-sycl/ggml-sycl.cpp,
line:5513,
func: operator()

SYCL error:
CHECK_TRY_ERROR((stream)->memcpy(
data,
(const char *)tensor->data + offset,
size
))

in function:
ggml_backend_sycl_get_tensor_async

The GDB backtrace subsequently shows the CLI waiting for the server-side completion path:

#0 __syscall_cancel_arch
#1 __internal_syscall_cancel
#2 __syscall_cancel
#3 poll
#4 cli_server::wait_ready(...)
#5 cli_context::init()
#6 llama_cli(int, char**)

The GDB libsycl.so auto-load warning appears after the crash, but I do not believe it is the cause of the failure. It is:

warning: File
"/opt/intel/oneapi/compiler/2026.1/lib/libsycl.so.9.0.0-gdb.py"
auto-loading has been declined

The actual SYCL/Level Zero error occurs before this GDB diagnostic.

The key reproducibility matrix is:

B50 only -> SUCCESS
A770 only -> SUCCESS

B50 + A770
layer -> UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY
row -> segmentation fault
tensor -> GGML_ASSERT in ggml-backend-meta.cpp

This issue can be reproduced with a minimal prompt (-p "Hello") and a very small context size (--ctx-size 1024), so the failure does not appear to be caused by a large context or inference workload.

[tv1wnd]

same issue here

2x B70

/edit

apparently there was something with the kernel driver which required a reboot ... works for me now ( both tensor & layer split mode )

[nicolataibi]

same behaviour with vulkan

[SleepinDevil]

@tv1wnd what OS are you on and is there anything special you did to get tensor split mode working on your 2x B70 setup? I am considering getting a second B70 and just doing my due diligence about my chances of success in getting it working right 😅

It would be very amazing if you could please contribute a benchmark datapoint for "llama-2-7B TheBloke Q4_0" at #23313 with Tensor Split on your 2x B70 please. SYCL 16 at least (but also 32 if you had time). The benchmark command to run is llama-bench -fa 0,1 -sm tensor -m /path/to/models/llama-2-7b.Q4_0.gguf

[arthw]

@SleepinDevil
1.
GGML_SYCL_USM_SYSTEM=1 don't set it, since USM is experience feature. It can't bring benefit in your case.

Try export GGML_SYCL_DEV2DEV_MEMCPY=2.

Try export SYCL_UR_USE_LEVEL_ZERO_V2=0.

Thank you!

[nicolataibi]

Dear Team,

I would like to report a successful resolution of a severe memory-related stability issue encountered while running the Qwen3.8-27B Q4_K_XL GGUF model with llama.cpp and the Intel SYCL backend on an Intel Arc A770.

The initial configuration consistently resulted in GPU memory allocation failures and, in some cases, system-level memory pressure severe enough to trigger process termination. The investigation showed that the problem was not caused by VMM, Level Zero, ESIMD, or fusion support, as initially suspected.

We isolated the issue by testing the SYCL features independently. The decisive finding was that:

• GGML_SYCL_ENABLE_HOST_PINNED_MEM=1 causes the crash in our environment.
• GGML_SYCL_ENABLE_HOST_PINNED_MEM=0 provides stable execution.
• GGML_SYCL_ENABLE_VMM=1 works correctly.
• GGML_SYCL_USE_LEVEL_ZERO_API=1 works correctly.
• GGML_SYCL_ENABLE_FUSION=1 works correctly.
• GGML_SYCL_ENABLE_ESIMD=1 works correctly.

An additional issue was identified with the batch configuration. Using --batch-size 1 resulted in the assertion:

GGML_ASSERT(n_tokens_all <= cparams.n_batch)

Increasing the configuration to:

--batch-size 32
--ubatch-size 32

resolved this problem.

With these findings, the stable single-GPU configuration became:

GGML_SYCL_ENABLE_VMM=1
GGML_SYCL_USE_LEVEL_ZERO_API=1
GGML_SYCL_ENABLE_FUSION=1
GGML_SYCL_ENABLE_ESIMD=1
GGML_SYCL_ENABLE_HOST_PINNED_MEM=0

Using the Intel Arc A770 alone, we were able to offload 30 layers of the 66-layer model and obtain approximately:

• Prompt processing: 15.5 tokens/s
• Generation: 5.8 tokens/s

The decisive step was then to use both Intel GPUs in layer-split mode:

--device SYCL0,SYCL1
--split-mode layer
--n-gpu-layers 66

with:

• SYCL0: Intel Arc Pro B50 16 GB
• SYCL1: Intel Arc A770 16 GB

This configuration successfully offloaded all 66/66 layers:

• B50 model buffer: approximately 7.65 GiB
• A770 model buffer: approximately 8.07 GiB
• CPU-mapped model buffer: only approximately 682 MiB

The model therefore became effectively GPU-resident across the two cards, avoiding the excessive system RAM consumption that previously caused Chrome and other desktop applications to be terminated by memory reclaim.

Most importantly, the system is now fully functional end-to-end. The Qwen3.8-27B model loads successfully, the SYCL/Level Zero execution path is stable, and inference completes correctly.

With the dual-GPU configuration we measured:

• Prompt processing: approximately 17.2 tokens/s
• Generation: approximately 11.8 tokens/s

This represents roughly a 2× improvement in generation throughput compared with the previous A770-only configuration using 30 GPU layers.

The final stable configuration is therefore:

GGML_SYCL_ENABLE_VMM=1
GGML_SYCL_USE_LEVEL_ZERO_API=1
GGML_SYCL_ENABLE_FUSION=1
GGML_SYCL_ENABLE_ESIMD=1
GGML_SYCL_ENABLE_HOST_PINNED_MEM=0

combined with:

--device SYCL0,SYCL1
--split-mode layer
--n-gpu-layers 66
--ctx-size 2048
--batch-size 32
--ubatch-size 32
--flash-attn off
--fit off
--no-warmup

In summary, the issue was resolved without disabling the SYCL features that were originally suspected to be responsible. The key requirement was to disable Host Pinned Memory explicitly and to use an appropriate batch size. Once this was done, the model could be distributed across the B50 and A770, achieving full 66-layer GPU offload with dramatically reduced system RAM usage and significantly higher inference throughput.

I believe this is a useful data point for investigating the interaction between llama.cpp, the Intel SYCL/Level Zero stack, and mixed Intel GPU architectures (BMG + DG2), particularly with Qwen35/Qwen3.8 models using Gated Delta Net and related fused operations.

Best regards,

Nick

[tv1wnd]

@tv1wnd what OS are you on and is there anything special you did to get tensor split mode working on your 2x B70 setup? I am considering getting a second B70 and just doing my due diligence about my chances of success in getting it working right 😅

It would be very amazing if you could please contribute a benchmark datapoint for "llama-2-7B TheBloke Q4_0" at #23313 with Tensor Split on your 2x B70 please. SYCL 16 at least (but also 32 if you had time). The benchmark command to run is llama-bench -fa 0,1 -sm tensor -m /path/to/models/llama-2-7b.Q4_0.gguf

current setup:

Ubuntu 26.04.1 LTS
intel drivers from: https://ppa.launchpadcontent.net/kobuk-team/intel-graphics/ubuntu/
mesa from: https://ppa.launchpadcontent.net/kisak/kisak-mesa/ubuntu/ ( only affects vulkan performance, but quite drastically )

Intel(R) Core(TM) Ultra 7 270K Plus
128GB DDR5-4800
03:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics]
08:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics]

intel compute runtime installed from:
https://github.com/intel/compute-runtime/releases

intel oneapi toolkit ( latest )
https://www.intel.com/content/www/us/en/developer/tools/oneapi/oneapi-toolkit-download.html?packages=oneapi-toolkit&oneapi-toolkit-os=linux&oneapi-lin=offline
( avoid the ubuntu repository, upgrades can and will result in messy file clashes accross packages )

cmake cli ( using latest llamacpp sources )

cmake -B build_sycl -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_SYCL_SUPPORT_VMM=OFF -DGGML_SYCL_SUPPORT_LEVEL_ZERO_API=1 -DGGML_SYCL_F16=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS:BOOL=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_SERVER:BOOL=ON -DLLAMA_BUILD_EXAMPLES:BOOL=OFF -DLLAMA_BUILD_TESTS:BOOL=OFF -DGGML_OPENMP:BOOL=OFF -DGGML_SYCL:BOOL=ON -DGGML_VULKAN:BOOL=ON -DGGML_CCACHE=true -DCMAKE_INSTALL_PREFIX=/opt/llamacpp -DCMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu

only env flag set:
ONEAPI_DEVICE_SELECTOR="level_zero:0;level_zero:1" ( to remove the igpu out of the equation )
currerntly running tensor split with Qwen3.8-27B-Q8_0.gguf and full 256k context for testing ( requires 42GB VRAM )

sycl-ls output.

[level_zero:gpu] Intel(R) oneAPI Unified Runtime over Level-Zero V2, Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.17.39395+13]
[level_zero:gpu] Intel(R) oneAPI Unified Runtime over Level-Zero V2, Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.17.39395+13]

benchmarking llama-2-7b.Q4_0.gguf with tensor split will not yield in any performance gains compared to a single B70
the dual B70 setup only makes sense for models eating up more than 32GB

but beware #27845 is unfortunately a thing

if you're going for throughput, a dual AMD R9700 pro setup is a much much better choice

hope that helps :)

[arthw]

@nicolataibi
It's great!

In the env vars list:
GGML_SYCL_ENABLE_VMM=1
GGML_SYCL_USE_LEVEL_ZERO_API=1
GGML_SYCL_ENABLE_FUSION=1
GGML_SYCL_ENABLE_ESIMD=1
GGML_SYCL_ENABLE_HOST_PINNED_MEM=0

Only need to set GGML_SYCL_ENABLE_HOST_PINNED_MEM=0, other values are default, no need to set again.

Thank you!

[arthw]

@SleepinDevil @nicolataibi
Could you upgrade the driver to 26.31.39395.13 https://github.com/intel/compute-runtime/releases?

[nicolataibi]

Thanks but i am in Fedora.

...
with this:
Level Zero API
export GGML_SYCL_DEV2DEV_MEMCPY=0
SYCL API
export GGML_SYCL_GET_MEM_API=1
ok!

but:
Level Zero API
export GGML_SYCL_DEV2DEV_MEMCPY=1
SYCL API
export GGML_SYCL_GET_MEM_API=0
Crash!!

[lslusarczyk]

@nicolataibi ,
It is very likely that your case with --split-mode layer hits #28765 which I've fixed in #28895.

In order to check if you have the same condition please check if both of your devices are on two different platforms. You may do it with the sycl-ls command:

sycl-ls --verbose | grep -E "^Platforms:|^Platform \[|^    (Name|Devices)|^        (Name|Device \[)"

Two blocks with Level-Zero platform, each having one device means your setup is the same as mine. One block with two devices means your case is different.

Also you may check if GGML_SYCL_ENABLE_HOST_PINNED_MEM=0 works around the problem. If so, then the issue is indeed same as mine.

You're welcome to cherry-pick my change and check if it fixes your problem.

[nicolataibi]

@nicolataibi , It is very likely that your case with --split-mode layer hits #28765 which I've fixed in #28895.

In order to check if you have the same condition please check if both of your devices are on two different platforms. You may do it with the sycl-ls command:

sycl-ls --verbose | grep -E "^Platforms:|^Platform \[|^    (Name|Devices)|^        (Name|Device \[)"

Two blocks with Level-Zero platform, each having one device means your setup is the same as mine. One block with two devices means your case is different.

Also you may check if GGML_SYCL_ENABLE_HOST_PINNED_MEM=0 works around the problem. If so, then the issue is indeed same as mine.

You're welcome to cherry-pick my change and check if it fixes your problem.

sudo sycl-ls --verbose | grep -E "^Platforms:|^Platform [|^ (Name|Devices)|^ (Name|Device [)"
[sudo] password di nick:
sudo: sycl-ls: comando non trovato
~/dev/intel
nick@glauke:$ sycl-ls --verbose | grep -E "^Platforms:|^Platform [|^ (Name|Devices)|^ (Name|Device [)"
[INFO]: failed to load adapter 'libur_adapter_cuda.so.0' with error: libur_adapter_cuda.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_cuda.so.0' with error: /opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_cuda.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter 'libur_adapter_hip.so.0' with error: libur_adapter_hip.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_hip.so.0' with error: /opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_hip.so.0: cannot open shared object file: No such file or directory
[INFO]: loaded adapter 0x0x5630abab0320 (libur_adapter_level_zero.so.0) from /opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_level_zero.so.0
[INFO]: loaded adapter 0x0x5630abab3e70 (libur_adapter_level_zero_v2.so.0) from /opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_level_zero_v2.so.0
[INFO]: failed to load adapter 'libur_adapter_native_cpu.so.0' with error: libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_native_cpu.so.0' with error: /opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter 'libur_adapter_offload.so.0' with error: libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_offload.so.0' with error: /opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
[INFO]: loaded adapter 0x0x5630abab76d0 (libur_adapter_opencl.so.0) from /opt/intel/oneapi/compiler/2026.1/lib/libur_adapter_opencl.so.0
Platforms: 5
Platform [#1]:
Name : Intel(R) oneAPI Unified Runtime over Level-Zero V2
Devices : 1
Device [#0]:
Name : Intel(R) Arc(TM) Pro B50 Graphics
Platform [#2]:
Name : Intel(R) oneAPI Unified Runtime over Level-Zero V2
Devices : 1
Device [#1]:
Name : Intel(R) Arc(TM) A770 Graphics
Platform [#3]:
Name : Intel(R) OpenCL
Devices : 1
Device [#0]:
Name : AMD Ryzen 9 5950X 16-Core Processor
Platform [#4]:
Name : Intel(R) OpenCL Graphics
Devices : 1
Device [#1]:
Name : Intel(R) Arc(TM) Pro B50 Graphics
Platform [#5]:
Name : Intel(R) OpenCL Graphics
Devices : 1
Device [#2]:
Name : Intel(R) Arc(TM) A770 Graphics

i cannot try your solution at the moment

[lslusarczyk]

Platform [#1]:
Name : Intel(R) oneAPI Unified Runtime over Level-Zero V2
Devices : 1
Device [#0]:
Name : Intel(R) Arc(TM) Pro B50 Graphics
Platform [#2]:
Name : Intel(R) oneAPI Unified Runtime over Level-Zero V2
Devices : 1
Device [#1]:
Name : Intel(R) Arc(TM) A770 Graphics

Two platforms. This is the same root cause. My fix should help with --split-mode layer. Please verify my fix on your end.