[am17an]

Overview

Fix a bug with state restore with partial offloading with dsv4.1 cache. Fixes #30944

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[vikcious]

Well, it's still not doing better. Right now the reasoning traces are DOA. Details below.

llama-server --version
version: 0.7.0-dev (build 11947, commit 63ea84e)
built with GNU 15.2.0 for Linux x86_64

test-llama-context --version
version: 0.7.0-dev (build 11947, commit 63ea84e)
built with GNU 15.2.0 for Linux x86_64

llama.cpp-test - initialization and execution sequence log
llama.cpp-boot-up_and_execution.log

(base) vik@SuperDomeZ4:~$ ~/llms/llama.cpp-test/build/cuda/bin/test-llama-context --model ~/models/boot/gguf/unsloth/DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL/UD-Q2_K_XL/DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL-00001-of-00007.gguf --split-mode layer --n-gpu-layers 32 --fit off -c 4096
threads: 32
0.22.361.865 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.22.361.871 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.22.364.612 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.22.364.615 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.24.838.794 I cmn  common_promp: saved session before last token to dump_state.DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL-00001-of-00007.gguf.bin, n_new = 100
baseline    OK
0.26.180.545 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.26.180.550 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.26.183.325 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.26.183.327 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
seq_rm      OK
0.29.833.195 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.29.833.201 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.29.836.021 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.29.836.024 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
state_load  OK
0.31.243.010 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.31.243.015 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.31.245.815 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.31.245.818 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
cp_h        OK
0.32.644.139 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.32.644.144 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.32.646.904 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.32.646.906 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
cp_d        OK
0.34.038.059 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.34.038.063 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.34.040.837 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.34.040.840 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
cp_h_s      OK
0.34.602.009 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.34.602.014 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.34.604.762 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.34.604.765 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
cp_d_s      OK
0.35.167.158 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.35.167.163 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.35.169.941 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.35.169.944 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
rt          OK
0.37.097.647 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.37.097.650 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.37.100.457 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.37.100.460 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
ps          OK
0.39.537.291 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.39.537.295 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.39.540.600 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.39.540.603 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.39.542.999 W resolve_fused_ops: layer 2 is assigned to device CPU but fused top-k mask is assigned to device CUDA0 (usually due to missing support)
0.39.543.002 W resolve_fused_ops: fused top-k mask not supported, set to disabled
0.40.306.208 E state_read_data: mismatched key type (1 != -1, layer 1)
0.40.309.372 E state_seq_set_data: error loading state: failed to restore kv cache
0.40.887.265 E state_read_data: mismatched key type (1 != -1, layer 1)
0.40.887.571 E llama_state_seq_load_file: error loading sequence state file: failed to restore kv cache
rf          OK
0.41.129.115 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.41.129.119 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.41.132.543 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.41.132.546 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.41.161.492 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.41.161.497 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.41.164.595 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.41.164.597 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.41.282.429 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.41.282.433 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.41.285.881 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.41.285.885 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.41.306.519 W llama_kv_cache: attention rotation force disabled (llama_model_set_attn_rot)
0.41.307.410 W llama_kv_cache: attention rotation force disabled (llama_model_set_attn_rot)
0.41.307.732 W llama_kv_cache: attention rotation force disabled (llama_model_set_attn_rot)
0.41.308.228 W llama_kv_cache: attention rotation force disabled (llama_model_set_attn_rot)
0.41.308.244 W llama_kv_cache: attention rotation force disabled (llama_model_set_attn_rot)
0.41.318.187 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.41.318.191 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.41.320.943 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.41.320.946 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.41.336.920 E state_read_data: incompatible key rotation (512 instead of 0)
set_seq_state: seq set data length 0 does not match expected length 52196
0.41.337.125 E state_seq_set_data: error loading state: failed to restore kv cache
rot         OK
0.41.360.383 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.41.360.388 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.41.363.174 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.41.363.177 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.41.429.182 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.41.429.187 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.41.431.894 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.41.431.896 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
frag        OK
0.53.032.853 W resolve_fused_ops: layer 9 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.53.032.857 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
noalloc     OK
0.53.070.016 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.53.070.020 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.53.072.842 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.53.072.846 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.53.181.264 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.53.181.273 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.53.184.131 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.53.184.134 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.53.987.871 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.53.987.875 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.53.990.960 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.53.990.963 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.54.001.267 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 894.32 MiB on device 4: cudaMalloc failed: out of memory
0.54.001.271 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA4 buffer of size 937760768
0.54.001.272 E graph_reserve: failed to allocate compute buffers
ctx: failed to create context for 'DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL-00001-of-00007.gguf'
0.54.016.872 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
rollback    FAIL
0.54.164.070 W llama_context: n_ctx is not divisible by n_seq_max - rounding down to 512
0.54.175.484 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.54.175.491 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.54.178.625 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.54.178.634 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.54.213.007 W llama_context: n_ctx is not divisible by n_seq_max - rounding down to 512
0.54.221.979 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.54.221.985 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.54.225.128 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.54.225.133 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
0.58.602.470 W ~llama_context:      CUDA0 compute buffer size of  23.0017 MiB, does not match expectation of  20.0029 MiB
0.58.602.476 W ~llama_context:      CUDA1 compute buffer size of  41.1368 MiB, does not match expectation of  27.1107 MiB
0.58.602.477 W ~llama_context:      CUDA2 compute buffer size of  40.7592 MiB, does not match expectation of  26.7749 MiB
0.58.602.478 W ~llama_context:      CUDA3 compute buffer size of  40.7795 MiB, does not match expectation of  26.7874 MiB
0.58.602.479 W ~llama_context:      CUDA4 compute buffer size of  40.7639 MiB, does not match expectation of  26.7717 MiB
0.58.602.480 W ~llama_context:      CUDA5 compute buffer size of  40.7639 MiB, does not match expectation of  26.7717 MiB
0.58.602.480 W ~llama_context:      CUDA6 compute buffer size of  40.7639 MiB, does not match expectation of  26.7717 MiB
0.58.602.481 W ~llama_context:      CUDA7 compute buffer size of  40.7347 MiB, does not match expectation of  26.7660 MiB
0.58.602.483 W ~llama_context:  CUDA_Host compute buffer size of  28.9359 MiB, does not match expectation of  25.4790 MiB
0.58.633.867 W ~llama_context:      CUDA0 compute buffer size of  23.0017 MiB, does not match expectation of  20.0029 MiB
0.58.633.872 W ~llama_context:      CUDA1 compute buffer size of  41.1368 MiB, does not match expectation of  27.1107 MiB
0.58.633.873 W ~llama_context:      CUDA2 compute buffer size of  40.7592 MiB, does not match expectation of  26.7749 MiB
0.58.633.875 W ~llama_context:      CUDA3 compute buffer size of  40.7795 MiB, does not match expectation of  26.7874 MiB
0.58.633.876 W ~llama_context:      CUDA4 compute buffer size of  40.7639 MiB, does not match expectation of  26.7717 MiB
0.58.633.877 W ~llama_context:      CUDA5 compute buffer size of  40.7639 MiB, does not match expectation of  26.7717 MiB
0.58.633.877 W ~llama_context:      CUDA6 compute buffer size of  40.7639 MiB, does not match expectation of  26.7717 MiB
0.58.633.878 W ~llama_context:      CUDA7 compute buffer size of  40.7347 MiB, does not match expectation of  26.7660 MiB
0.58.633.880 W ~llama_context:  CUDA_Host compute buffer size of  28.9359 MiB, does not match expectation of  25.4790 MiB
replay      FAIL  multi-seq split replay logits mismatch (max diff 2.662254, nmse 0.017587, first difference at index 0)
0.58.678.284 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
0.58.678.289 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
0.58.681.076 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
0.58.681.079 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
1.03.391.666 W resolve_fused_ops: layer 13 is assigned to device CUDA1 but fused DeepSeek V4 HC pre is assigned to device CUDA0 (usually due to missing support)
1.03.391.672 W resolve_fused_ops: fused DeepSeek V4 HC pre not supported, set to disabled
1.03.394.487 W resolve_fused_ops: layer 8 is assigned to device CPU but fused DeepSeek V4 HC post is assigned to device CUDA0 (usually due to missing support)
1.03.394.489 W resolve_fused_ops: fused DeepSeek V4 HC post not supported, set to disabled
shared      OK

cases    : 16
passed   : 14
skipped  : 0
failed   : 2

[am17an]

@ggerganov let's merge this as it fixes a real problem. The traces pasted above involve ZenDNN and it is likely there is a bug there too (which I won't be able to fix)