[ggerganov]

Overview

This release adds OCP FP8 E4M3 quantization support, a new flash_attn_ext_rows op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), audio generation and transcription expand with Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices, and the Gemma 4 E2B/E4B models get major prefill optimizations. ggml is updated to v0.29.0, bringing FP8 E4M3, flash attention rows, NCCL-based multi-node tensor parallelism and performance work across the CPU, CUDA, SYCL, Metal, Vulkan, OpenCL and Hexagon backends. The server and UI receive a round of fixes and refinements.

Highlights

• Quant: OCP FP8 E4M3 tensor type (saturating mode) with conversion helpers and a new ftype (#27512)

• llama: flash_attn_ext_rows - more efficient unified KV for multi-seq, restoring non-unified KV prefill performance (#29510)

• ggml: batched tensor copy API (ggml_backend_tensor_copy_n/set_n/get_n) superseding the _2d API, implemented for Metal and CUDA (#30881)

• For a 3rd year in a row, llama.cpp is a featured repo in the Github Octoverse report:

  

API changes

• llama.h: add LLAMA_FTYPE_MOSTLY_F8_E4M3 (#27512)
• mtmd.h: add MTMD_GEN_AUDIO_TYPE_CHATTERBOX and n_voices to mtmd_gen_audio_info (#30993)
• mtmd.h: add predefined-voice APIs: mtmd_gen_audio_voice_name(), mtmd_gen_audio_get_voice(), mtmd_gen_state_set_ref() (#30993)
• mtmd.h: add mtmd_bitmap_is_voice() and mtmd_gen_audio_read_tensor() (#30993)
• mtmd.h: add MTMD_VIDEO_TEMPLATE_QWEN3VL video prompt template (#31619)
• server: GET /props reports the model's predefined TTS voices; the TTS endpoint accepts predefined voice names (#30993)
• server: /v1/responses now merges extra_params (seed, min_p, ...) into the converted chat-completions response (#30959)

New models

• Clover-35B-A3B-Preview diffusion model, AR mode (#31306)
• Limite 1B - Violetto (#29433)
• Metis-1.6 (Praxis ~2.9B and Logos ~10.2B): hybrid Mamba-2 / GQA attention with fixed-depth recurrence and LatentMoE (#31134)
• Mistral Large 4 (#31376)

Core changes

• gemma4: graph optimizations with massive prefill gains for E2B and E4B (#31356); create token_embd before the tied output in gemma3n and gemma4 (#31394)
• LFM3: enable NVFP4 GGUFs (#31608), enable tensor split (#31431), quantize the audio codec decoder (#31386)
• spec dec: save and restore MTP hidden state in context checkpoints (#30141) and record LFM3 layer inputs (#31598)
• dflash: copy hidden features via device copies (#31506); qwen4exp: write the conv state tail without an intermediate cont (#31443)
• model loading: reuse the tensor staging buffer and skip the direct-IO bounce buffer (#31321)
• kv-cache: keep Hadamard rotation constants on the GPU (#31073) and fix attention tensor rotation in layers with shared KV (#31406)
• kv-cache: never return 0 kv_rows for sequences without cells (#31570) and refuse rollbacks past the snapshots of the last ubatch (#31368)
• Gemma4 MTP assistant: follow ctx_other devices and flash-attention setting (#31635); DSV4: remove simple batch splitting (#31300)
• chat: strip the qwen3-coder think block with reasoning_format none (#31380) and end reasoning budget forcing on a prompt end tag (#31374)
• chat: allow a constraint type after the gpt-oss final channel (#31556); common: fix -hf tag matching inside longer quant names (#31620) and bound control vector layer indices (#31336)
• tools: reject mismatched shards in GGUF split merge (#31579), fix imatrix --nextn crash (#31522), fix gguf-py np_roundf dropping the sign (#31310)
• vendor: miniaudio 0.11.26 (#31459) and cpp-httplib 0.64.0 (#31400); embed an app manifest in all Windows executables (#31329)

Multi-modality changes

• add Chatterbox TTS support (#30993)
• add Nemotron 3.5 Transcribe support: offline transcription in 40 languages with prompt context and speaker attribution (#31471)
• Pocket TTS: support predefined voices without voice cloning (#31429)
• accurate support for the qwen3-vl video prompt format (#31619)
• resize Gemma 4 images without letterbox padding (#30922) and fix InternVL CLS token order (#30886)

Server changes

• cap shared decision prefixes at the first output (#30710)
• restore the speculative checkpoint for drafts longer than n_rs_seq (#31350)

UI changes

• add MCP group checkboxes in the tools settings (#31303) and request persistent storage on startup (#31299)
• keep chats at the bottom until the reader scrolls up (#31311) and show the submit button when the input has content (#31324)
• reveal the collapsed sidebar from its logo (#31307) and align alert dialog styling (#31312)
• keep queued steering messages in their own conversation (#31304) and show an interrupted state for tool calls that never returned (#31314)
• disable model deletion the server would refuse (#31305) and prefer a loading model in pre-selection (#31326)
• add a Playwright harness for recording demo videos (#31318) and run local browser tests in installed Chrome (#31596)

ggml changes

• v0.29.0: ggml v0.29.0 adds an OCP FP8 E4M3 tensor type, a new ggml_flash_attn_ext_rows op where each query slice attends to its own list of K/V rows, a batched tensor copy API, and initial support for multi-node tensor parallelism through a new multi-process communicator interface. Core improvements include bounding tensor views by the bytes their strides reach, zero-copy GGUF export of host-buffer tensors, and vm_allocate-based allocations on Apple platforms. Performance work spans all major backends: new SVE kernels in the CPU backend, DPAS flash attention for quantized KV in SYCL, fused MoE MMQ in CUDA, faster MXFP4 mat-vec in Metal, split-K tuning in Vulkan, and a large expansion of quantized GEMM/GEMV coverage in OpenCL. The Hexagon backend gains support for several new ops, including the lightning indexer, top-k mask, conv2d and upscale.