[ggerganov]

ref #20087 (comment)

Sometimes the server enters an infinite loop of empty batches. This change makes it easier to debug such cases.

[aldehir]

I managed to reproduce this reliably with gpt-oss-20b and 120b.

Windows & Nvidia RTX 3090.

repro.py
server.log 1
server.log 2

The reproducer runs an agentic scenario, and then runs it again. The second run is what triggers the loop.

Let me know if there's anything else I can provide.

[aviallon]

@ggerganov
Did you mean consecutive?

[asbelin]

https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF/discussions/5

[Bug] Fatal errors when loading unsloth/gemma-4-12B-it-qat-GGUF with MTP draft model on RTX 5070 Ti — LM Studio 0.4.16

Environment:
Software: LM Studio 0.4.16
Repository: unsloth/gemma-4-12B-it-qat-GGUF
Main Model: gemma-4-12B-it-qat-UD-Q4_K_XL.gguf
Draft/Speculative Model: gemma-4-12B-it-Q4_0-MTP.gguf (from unsloth/gemma-4-12B-it-qat-mtp-GGUF)
Operating System: Windows 11
Hardware: NVIDIA RTX 5070 Ti (Blackwell architecture)

Observed behaviour:
The developer log captures two consecutive load attempts, both ending in a fatal error.

Scenario 1 — Flash Attention enabled (AUTO mode):
Model load proceeds until the warmup stage, then crashes with an assertion failure in the CUDA Flash Attention backend:

gemma4 architecture detected, using AUTO flash attention mode
...
common_init_from_params: warming up the model with an empty run
llm-engine\llama.cpp\ggml\src\ggml-cuda\fattn-common.cuh:1110:
GGML_ASSERT(max_blocks_per_sm > 0) failed

Scenario 2 — Flash Attention disabled (fallback after Scenario 1):
LM Studio retries with FA disabled (llama_kv_cache: ... FA is not enabled - padding V cache to 2048). Both the target and draft models load, and speculative decoding is initialised:

common_speculative_impl_draft_simple: adding speculative implementation 'draft-simple'
srv load_model: speculative decoding context initialized

On the first inference request, decoding fails due to KV-cache position mismatch:

E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 53
the tokens for sequence 0 in the input batch have a starting position of Y = 0
it is required that the sequence positions remain consecutive: Y = X + 1

E decode: failed to initialize batch
E llama_decode: failed to decode, ret = -1
E process: failed to decode draft batch, ret = -1, error = 0
E srv update_slots: failed to process speculative batch

The server then terminates with:

llm-engine\llama.cpp\tools\server\server-context.cpp:3190:
fatal error - please provide logs and repro in https://github.com/ggml-org/llama.cpp/pull/20277

Request:
Could you confirm whether this behaviour is specific to the MTP tensor layout of this GGUF conversion, or whether it is a known upstream llama.cpp issue (cf. PR #20277) affecting Gemma-4 MTP on Blackwell GPUs? Any guidance on a working configuration would be appreciated.