Repository navigation
Compile bug: --fit silently aborts when --cpu-moe/--n-cpu-moe sets tensor_buft_overrides, leading to OOM on compute buffer allocation #27872
Description
Git commit
version: 0.3.0-dev (build 10621, commit c1d0e7a) — tag v0.3.0
OS: Ubuntu, Linux x86_64
GPU: NVIDIA RTX 3090 (24GB), CUDA
Model: DeepSeek-V4-Flash-0731 (deepseek4 arch, 256 experts, UD-Q2_K_XL quant, unsloth)
Operating systems
Linux
GGML backends
CUDA
Problem description & steps to reproduce
When --cpu-moe is passed (which sets model_params::tensor_buft_overrides to keep MoE weights on CPU), the new -fit/--fit memory-fitting logic (moved to common/, now accounting for n_streams — #27496) refuses to run because overrides are already user-set:
common_params_fit_impl: projected to use 24840 MiB of device memory vs. 22341 MiB of free device memory
common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 3523 MiB
common_fit_params: failed to fit params to free device memory: model_params::tensor_buft_overrides already set by user, abort
This is logged as a warning (W), not a fatal error, so the server continues loading with the original, unfitted parameters instead of stopping or falling back to a safe configuration. Since the projected footprint (24840 MiB) already exceeds free VRAM (22341 MiB), this reliably ends in an OOM later, at compute-buffer allocation time — the actual failure is much less legible than the root cause:
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 14656.12 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 15368060928
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
This same class of bug was already reported for --fit on + --sleep-idle-seconds (#24684) with the same "tensor_buft_overrides already set by user, abort" message; this report is the --cpu-moe-triggered variant of what looks like the same underlying issue in the fit logic (post-#27496 refactor).
Expected behavior
Either:
-fit should fail loudly (non-zero exit) when it can't fit params to free memory, instead of silently proceeding with an oversized configuration that's guaranteed to OOM later, or
-fit should be able to work around a user-set --cpu-moe/--n-cpu-moe override (e.g. by adjusting context size / batch size instead of tensor placement) rather than aborting entirely.
Actual behavior
Server proceeds past the failed fit and crashes later during compute buffer allocation, with a much less informative error than the original fit failure.
Reproduction
llama-server
-m DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf
--cpu-moe
-c 524288
-b 8192 -ub 8192
-fa on
--port 8000
Workaround
Passing -fit off explicitly avoids the misleading log path (fit was doing nothing useful anyway with --cpu-moe set). Manually reducing -b/-ub (e.g. to 4096) shrinks the compute buffer enough to fit in the remaining VRAM.
First Bad Commit
No response
Compile command
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)Relevant log output
0.00.022.341 I cmn common_param: common_params_print_info: build 10621 (c1d0e7a00) with GNU 13.3.0 for Linux x86_64
0.00.022.344 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
0.00.022.345 I cmn common_param: device_info:
0.00.145.324 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 3090 (24117 MiB, 22374 MiB free)
0.00.145.341 I cmn common_param: - CPU : AMD Ryzen 7 3700X 8-Core Processor (128719 MiB, 128719 MiB free)
0.00.145.430 I cmn common_param: system_info: n_threads = 12 (n_threads_batch = 12) / 16 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.145.473 I srv init: running without SSL
0.00.145.547 I srv init: using 15 threads for HTTP server
0.00.145.995 W srv llama_server: -----------------
0.00.145.997 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.145.997 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.145.997 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.145.997 W srv llama_server: -----------------
0.00.146.023 I srv start: binding port with default address family
0.00.147.285 I srv load_model: loading model '/mnt/ramdisk/DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf'
0.00.147.288 I srv load_model: local path '/mnt/ramdisk/DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf'
0.00.196.576 I llama_model_loader: additional 2 GGUFs metadata loaded.
0.00.196.589 I llama_model_loader: loaded meta data with 74 key-value pairs and 1328 tensors from /mnt/ramdisk/DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf (version GGUF V3 (latest))
0.00.196.625 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.00.196.634 I llama_model_loader: - kv 0: general.architecture str = deepseek4
0.00.196.638 I llama_model_loader: - kv 1: tokenizer.chat_template str = {#- Unsloth template fixes #}\n{%- if ...
0.00.196.638 I llama_model_loader: - kv 2: general.type str = model
0.00.196.645 I llama_model_loader: - kv 3: general.sampling.top_p f32 = 1.000000
0.00.196.649 I llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
0.00.196.650 I llama_model_loader: - kv 5: general.name str = Deepseek-V4-Flash-0731
0.00.196.650 I llama_model_loader: - kv 6: general.version str = 0731
0.00.196.651 I llama_model_loader: - kv 7: general.basename str = Deepseek-V4-Flash-0731
0.00.196.651 I llama_model_loader: - kv 8: general.quantized_by str = Unsloth
0.00.196.652 I llama_model_loader: - kv 9: general.size_label str = 256x8.4B
0.00.196.652 I llama_model_loader: - kv 10: general.license str = mit
0.00.196.653 I llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth
0.00.196.655 I llama_model_loader: - kv 12: general.base_model.count u32 = 1
0.00.196.656 I llama_model_loader: - kv 13: general.base_model.0.name str = DeepSeek V4 Flash 0731
0.00.196.656 I llama_model_loader: - kv 14: general.base_model.0.version str = 0731
0.00.196.657 I llama_model_loader: - kv 15: general.base_model.0.organization str = Deepseek Ai
0.00.196.659 I llama_model_loader: - kv 16: general.base_model.0.repo_url str = https://huggingface.co/deepseek-ai/De...
0.00.196.673 I llama_model_loader: - kv 17: general.tags arr[str,1] = ["unsloth"]
0.00.196.674 I llama_model_loader: - kv 18: deepseek4.block_count u32 = 43
0.00.196.674 I llama_model_loader: - kv 19: deepseek4.context_length u32 = 1048576
0.00.196.675 I llama_model_loader: - kv 20: deepseek4.embedding_length u32 = 4096
0.00.196.675 I llama_model_loader: - kv 21: deepseek4.attention.head_count u32 = 64
0.00.196.676 I llama_model_loader: - kv 22: deepseek4.attention.head_count_kv u32 = 1
0.00.196.677 I llama_model_loader: - kv 23: deepseek4.rope.scaling.type str = yarn
0.00.196.678 I llama_model_loader: - kv 24: deepseek4.rope.scaling.factor f32 = 16.000000
0.00.196.679 I llama_model_loader: - kv 25: deepseek4.rope.scaling.original_context_length u32 = 65536
0.00.196.680 I llama_model_loader: - kv 26: deepseek4.rope.scaling.yarn_beta_fast f32 = 32.000000
0.00.196.681 I llama_model_loader: - kv 27: deepseek4.rope.scaling.yarn_beta_slow f32 = 1.000000
0.00.196.682 I llama_model_loader: - kv 28: deepseek4.rope.freq_base f32 = 10000.000000
0.00.196.683 I llama_model_loader: - kv 29: deepseek4.attention.layer_norm_rms_epsilon f32 = 0.000001
0.00.196.684 I llama_model_loader: - kv 30: deepseek4.expert_count u32 = 256
0.00.196.684 I llama_model_loader: - kv 31: deepseek4.expert_used_count u32 = 6
0.00.196.685 I llama_model_loader: - kv 32: deepseek4.expert_gating_func u32 = 4
0.00.196.686 I llama_model_loader: - kv 33: deepseek4.attention.key_length u32 = 512
0.00.196.686 I llama_model_loader: - kv 34: deepseek4.attention.value_length u32 = 512
0.00.196.687 I llama_model_loader: - kv 35: deepseek4.rope.dimension_count u32 = 64
0.00.196.687 I llama_model_loader: - kv 36: deepseek4.attention.q_lora_rank u32 = 1024
0.00.196.688 I llama_model_loader: - kv 37: deepseek4.attention.sliding_window u32 = 128
0.00.196.688 I llama_model_loader: - kv 38: deepseek4.expert_feed_forward_length u32 = 2048
0.00.196.689 I llama_model_loader: - kv 39: deepseek4.expert_shared_count u32 = 1
0.00.196.689 I llama_model_loader: - kv 40: deepseek4.expert_weights_scale f32 = 1.500000
0.00.196.690 I llama_model_loader: - kv 41: deepseek4.expert_weights_norm bool = true
0.00.196.705 I llama_model_loader: - kv 42: deepseek4.swiglu_clamp_exp arr[f32,43] = [10.000000, 10.000000, 10.000000, 10....
0.00.196.719 I llama_model_loader: - kv 43: deepseek4.swiglu_clamp_shexp arr[f32,43] = [10.000000, 10.000000, 10.000000, 10....
0.00.196.720 I llama_model_loader: - kv 44: deepseek4.attention.indexer.head_count u32 = 64
0.00.196.720 I llama_model_loader: - kv 45: deepseek4.attention.indexer.key_length u32 = 128
0.00.196.721 I llama_model_loader: - kv 46: deepseek4.attention.indexer.top_k u32 = 512
0.00.196.722 I llama_model_loader: - kv 47: deepseek4.attention.output_group_count u32 = 8
0.00.196.722 I llama_model_loader: - kv 48: deepseek4.attention.output_lora_rank u32 = 1024
0.00.196.726 I llama_model_loader: - kv 49: deepseek4.attention.compress_ratios arr[i32,46] = [0, 0, 4, 128, 4, 128, 4, 128, 4, 128...
0.00.196.728 I llama_model_loader: - kv 50: deepseek4.attention.compress_rope_freq_base f32 = 160000.000000
0.00.196.729 I llama_model_loader: - kv 51: deepseek4.hyper_connection.count u32 = 4
0.00.196.729 I llama_model_loader: - kv 52: deepseek4.hyper_connection.sinkhorn_iterations u32 = 20
0.00.196.730 I llama_model_loader: - kv 53: deepseek4.hyper_connection.epsilon f32 = 0.000001
0.00.196.731 I llama_model_loader: - kv 54: deepseek4.hash_layer_count u32 = 3
0.00.196.731 I llama_model_loader: - kv 55: tokenizer.ggml.model str = gpt2
0.00.196.732 I llama_model_loader: - kv 56: tokenizer.ggml.pre str = joyai-llm
0.00.216.876 I llama_model_loader: - kv 57: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<\EF...
0.00.221.547 I llama_model_loader: - kv 58: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
0.00.240.604 I llama_model_loader: - kv 59: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
0.00.240.609 I llama_model_loader: - kv 60: tokenizer.ggml.bos_token_id u32 = 0
0.00.240.610 I llama_model_loader: - kv 61: tokenizer.ggml.eos_token_id u32 = 1
0.00.240.611 I llama_model_loader: - kv 62: tokenizer.ggml.padding_token_id u32 = 2
0.00.240.611 I llama_model_loader: - kv 63: tokenizer.ggml.add_bos_token bool = false
0.00.240.612 I llama_model_loader: - kv 64: tokenizer.ggml.add_eos_token bool = false
0.00.240.612 I llama_model_loader: - kv 65: general.quantization_version u32 = 2
0.00.240.613 I llama_model_loader: - kv 66: general.file_type u32 = 10
0.00.240.615 I llama_model_loader: - kv 67: quantize.imatrix.file str = DeepSeek-V4-Flash-0731-GGUF/imatrix_u...
0.00.240.616 I llama_model_loader: - kv 68: quantize.imatrix.dataset str = unsloth_calibration_DeepSeek-V4-Flash...
0.00.240.616 I llama_model_loader: - kv 69: quantize.imatrix.entries_count u32 = 768
0.00.240.617 I llama_model_loader: - kv 70: quantize.imatrix.chunks_count u32 = 50
0.00.240.617 I llama_model_loader: - kv 71: split.no u16 = 0
0.00.240.618 I llama_model_loader: - kv 72: split.tensors.count i32 = 1328
0.00.240.619 I llama_model_loader: - kv 73: split.count u16 = 3
0.00.240.623 I llama_model_loader: - type f32: 662 tensors
0.00.240.624 I llama_model_loader: - type q8_0: 318 tensors
0.00.240.625 I llama_model_loader: - type q4_K: 1 tensors
0.00.240.625 I llama_model_loader: - type q5_K: 127 tensors
0.00.240.625 I llama_model_loader: - type q6_K: 45 tensors
0.00.240.626 I llama_model_loader: - type iq2_xs: 84 tensors
0.00.240.626 I llama_model_loader: - type iq3_xxs: 43 tensors
0.00.240.626 I llama_model_loader: - type i32: 3 tensors
0.00.240.627 I llama_model_loader: - type bf16: 43 tensors
0.00.240.627 I llama_model_loader: - type mxfp4: 2 tensors
0.00.240.630 I print_info: file format = GGUF V3 (latest)
0.00.240.632 I print_info: file type = Q2_K - Medium
0.00.240.635 I print_info: file size = 90.18 GiB (2.72 BPW)
0.00.247.143 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:0a:00.0) - 22374 MiB free
0.00.342.709 I load: 0 unused tokens
0.00.364.365 I load: printing all EOG tokens:
0.00.364.370 I load: - 1 ('<｜end▁of▁sentence｜>')
0.00.364.655 I load: special tokens cache size = 1283
0.00.416.701 I load: token to piece cache size = 0.8346 MB
0.00.416.724 I print_info: arch = deepseek4
0.00.416.726 I print_info: vocab_only = 0
0.00.416.727 I print_info: no_alloc = 0
0.00.416.727 I print_info: n_ctx_train = 1048576
0.00.416.728 I print_info: n_embd_inp = 4096
0.00.416.728 I print_info: n_embd = 4096
0.00.416.730 I print_info: n_embd_out = 16384
0.00.416.730 I print_info: n_layer = 43
0.00.416.731 I print_info: n_layer_all = 43
0.00.416.742 I print_info: n_head = 64
0.00.416.744 I print_info: n_head_kv = 1
0.00.416.744 I print_info: n_rot = 64
0.00.416.745 I print_info: n_swa = 128
0.00.416.747 I print_info: is_swa_any = 1
0.00.416.747 I print_info: n_embd_head_k = 512
0.00.416.748 I print_info: n_embd_head_v = 512
0.00.416.750 I print_info: n_gqa = 64
0.00.416.757 I print_info: n_embd_k_gqa = 512
0.00.416.760 I print_info: n_embd_v_gqa = 512
0.00.416.761 I print_info: f_norm_eps = 0.0e+00
0.00.416.762 I print_info: f_norm_rms_eps = 1.0e-06
0.00.416.762 I print_info: f_clamp_kqv = 0.0e+00
0.00.416.763 I print_info: f_max_alibi_bias = 0.0e+00
0.00.416.763 I print_info: f_logit_scale = 0.0e+00
0.00.416.764 I print_info: f_attn_scale = 0.0e+00
0.00.416.764 I print_info: f_attn_value_scale = 0.0000
0.00.416.767 I print_info: n_ff = 0
0.00.416.767 I print_info: n_expert = 256
0.00.416.768 I print_info: n_expert_used = 6
0.00.416.768 I print_info: n_expert_groups = 0
0.00.416.769 I print_info: n_group_used = 0
0.00.416.769 I print_info: causal attn = 1
0.00.416.769 I print_info: pooling type = -1
0.00.416.770 I print_info: rope type = 0
0.00.416.770 I print_info: rope scaling = yarn
0.00.416.772 I print_info: freq_base_train = 10000.0
0.00.416.773 I print_info: freq_scale_train = 0.0625
0.00.416.774 I print_info: freq_base_swa = 10000.0
0.00.416.775 I print_info: freq_scale_swa = 1
0.00.416.775 I print_info: n_embd_head_k_swa = 512
0.00.416.776 I print_info: n_embd_head_v_swa = 512
0.00.416.776 I print_info: n_rot_swa = 64
0.00.416.776 I print_info: n_ctx_orig_yarn = 65536
0.00.416.777 I print_info: rope_yarn_log_mul = 0.0000
0.00.416.777 I print_info: rope_finetuned = unknown
0.00.416.778 I print_info: model type = ?B
0.00.416.779 I print_info: model params = 284.33 B
0.00.416.780 I print_info: general.name = Deepseek-V4-Flash-0731
0.00.416.784 I print_info: vocab type = BPE
0.00.416.786 I print_info: n_vocab = 129280
0.00.416.787 I print_info: n_merges = 127741
0.00.416.787 I print_info: BOS token = 0 '<｜begin▁of▁sentence｜>'
0.00.416.788 I print_info: EOS token = 1 '<｜end▁of▁sentence｜>'
0.00.416.789 I print_info: EOT token = 1 '<｜end▁of▁sentence｜>'
0.00.416.789 I print_info: PAD token = 2 '<｜▁pad▁｜>'
0.00.416.790 I print_info: LF token = 201 'Ċ'
0.00.416.790 I print_info: FIM PRE token = 128801 '<｜fim▁begin｜>'
0.00.416.791 I print_info: FIM SUF token = 128800 '<｜fim▁hole｜>'
0.00.416.791 I print_info: FIM MID token = 128802 '<｜fim▁end｜>'
0.00.416.791 I print_info: EOG token = 1 '<｜end▁of▁sentence｜>'
0.00.416.792 I print_info: max token length = 256
0.00.419.342 I load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
0.00.426.889 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance
0.10.522.842 I load_tensors: offloading output layer to GPU
0.10.522.851 I load_tensors: offloading 42 repeating layers to GPU
0.10.522.852 I load_tensors: offloaded 44/44 layers to GPU
0.10.522.858 I load_tensors: CPU_Mapped model buffer size = 46738.52 MiB
0.10.522.859 I load_tensors: CPU_Mapped model buffer size = 45174.68 MiB
0.10.522.859 I load_tensors: CUDA0 model buffer size = 6378.40 MiB
0.11.048.493 I cmn common_init_: added <｜end▁of▁sentence｜> logit bias = -inf
0.11.086.229 I llama_context: constructing llama_context
0.11.086.237 I llama_context: n_seq_max = 1
0.11.086.238 I llama_context: n_ctx = 524288
0.11.086.238 I llama_context: n_ctx_seq = 524288
0.11.086.238 I llama_context: n_batch = 8192
0.11.086.238 I llama_context: n_ubatch = 8192
0.11.086.239 I llama_context: causal_attn = 1
0.11.086.240 I llama_context: flash_attn = enabled
0.11.086.240 I llama_context: kv_unified = false
0.11.086.244 I llama_context: freq_base = 10000.0
0.11.086.244 I llama_context: freq_scale = 0.0625
0.11.086.244 I llama_context: n_rs_seq = 0
0.11.086.245 I llama_context: n_outputs_max = 1
0.11.086.245 I llama_context: n_outputs_max_per_seq = 1
0.11.086.245 I llama_context: n_ctx_seq (524288) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
0.11.090.360 I llama_context: CUDA_Host output buffer size = 0.49 MiB
0.11.090.384 I llama_kv_cache_dsv4: creating DSV4 raw KV cache
0.11.090.392 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 524288 cells
0.11.101.348 I llama_kv_cache: size = 0.00 MiB (524288 cells, 0 layers, 1/1 seqs), K (f16): 0.00 MiB, V (f16): 0.00 MiB
0.11.101.353 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 0
0.11.101.354 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 0
0.11.101.355 I llama_kv_cache_iswa: creating SWA KV cache, size = 8448 cells
0.11.101.721 I llama_kv_cache: CUDA0 KV buffer size = 354.75 MiB
0.11.102.255 I llama_kv_cache: size = 354.75 MiB ( 8448 cells, 43 layers, 1/1 seqs), K (f16): 354.75 MiB, V (f16): 0.00 MiB
0.11.102.256 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.11.102.257 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 0
0.11.102.258 I llama_kv_cache_dsv4: creating DSV4 CSA compressed KV cache, size = 131072 cells
0.11.104.680 I llama_kv_cache: CUDA0 KV buffer size = 2688.00 MiB
0.11.108.314 I llama_kv_cache: size = 2688.00 MiB (131072 cells, 21 layers, 1/1 seqs), K (f16): 2688.00 MiB, V (f16): 0.00 MiB
0.11.108.316 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.11.108.316 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 0
0.11.108.317 I llama_kv_cache_dsv4: creating DSV4 HCA compressed KV cache, size = 4096 cells
0.11.108.480 I llama_kv_cache: CUDA0 KV buffer size = 80.00 MiB
0.11.108.670 I llama_kv_cache: size = 80.00 MiB ( 4096 cells, 20 layers, 1/1 seqs), K (f16): 80.00 MiB, V (f16): 0.00 MiB
0.11.108.671 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.11.108.672 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 0
0.11.108.672 I llama_kv_cache_dsv4: creating DSV4 lightning-indexer KV cache, size = 131072 cells
0.11.111.521 I llama_kv_cache: CUDA0 KV buffer size = 672.00 MiB
0.11.112.415 I llama_kv_cache: size = 672.00 MiB (131072 cells, 21 layers, 1/1 seqs), K (f16): 672.00 MiB, V (f16): 0.00 MiB
0.11.112.417 I llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 128
0.11.112.417 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 0
0.11.112.429 I llama_kv_cache_dsv4: creating DSV4 CSA compressor state
0.11.112.546 I llama_dsv4_comp_state: CUDA0 DSV4 csa state buffer size = 1.31 MiB
0.11.112.550 I llama_dsv4_comp_state: csa ratio = 4, state = 8 x 1024, streams = 1, rs_seq = 0, layers = 21, size = 1.31 MiB
0.11.112.550 I llama_kv_cache_dsv4: creating DSV4 HCA compressor state
0.11.112.658 I llama_dsv4_comp_state: CUDA0 DSV4 hca state buffer size = 10.00 MiB
0.11.112.661 I llama_dsv4_comp_state: hca ratio = 128, state = 128 x 512, streams = 1, rs_seq = 0, layers = 20, size = 10.00 MiB
0.11.112.661 I llama_kv_cache_dsv4: creating DSV4 lightning-indexer compressor state
0.11.112.692 I llama_dsv4_comp_state: CUDA0 DSV4 lid state buffer size = 0.33 MiB
0.11.112.694 I llama_dsv4_comp_state: lid ratio = 4, state = 8 x 256, streams = 1, rs_seq = 0, layers = 21, size = 0.33 MiB
0.11.118.576 I sched_reserve: reserving ...
0.11.232.902 I resolve_fused_ops: resolving fused Gated Delta Net support:
0.11.236.370 I resolve_fused_ops: fused Gated Delta Net (autoregressive) enabled
0.11.239.264 I resolve_fused_ops: fused Gated Delta Net (chunked) enabled
0.11.239.266 I resolve_fused_ops: resolving fused Lightning Indexer support:
0.11.242.054 I resolve_fused_ops: Lightning Indexer enabled
0.11.242.057 I resolve_fused_ops: resolving fused DeepSeek V4 HC support:
0.11.245.136 I resolve_fused_ops: fused DeepSeek V4 HC pre enabled
0.11.248.027 I resolve_fused_ops: fused DeepSeek V4 HC comb enabled
0.11.250.784 I resolve_fused_ops: fused DeepSeek V4 HC post enabled
0.11.257.162 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 14656.12 MiB on device 0: cudaMalloc failed: out of memory
0.11.257.168 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 15368060928
0.11.257.169 E graph_reserve: failed to allocate compute buffers
0.11.293.679 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
0.11.293.689 E cmn common_init_: failed to create context with model '/mnt/ramdisk/DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf'
0.11.293.692 E cmn common_init_: failed to create context with model '/mnt/ramdisk/DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf'
0.11.293.696 E srv load_model: failed to create_context with model '/mnt/ramdisk/DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf'
0.11.293.697 I srv operator(): operator(): cleaning up before exit...
0.11.294.351 E srv llama_server: exiting due to model loading error
Activity
github-actions commentedon Oct 12, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.