models: Add Flint model (FlintForCausalLM) by alexey-rsi · Pull Request #30928 · ggml-org/llama.cpp

alexey-rsi · 2026-10-21T14:48:51Z

Overview

Adds Flint (FlintForCausalLM) support, including Hugging Face to GGUF conversion, tokenizer handling, and inference with sliding/full attention and routed/shared experts.

Uses the adaptive expert-activation scaling from #30861 with a 1,024 threshold, leaving headroom for 32-value Q8 activation block sums to fit in F16.

The model's chat template requires for hexadecimal string escapes.

GGUF repositories: NVFP4 and INT4

Additional information

test-llama-archs passes on CPU, CUDA and Metal, including checks for Mistral4/Mellum, which share the scaling helper.
Full-model checks produce finite logits with BF16/NVFP4 on H200/B300, quantized models on RTX PRO 6000, and Q4_1 on M4 Pro.
Auto-scaling leads to slowdown of 10-30%, this will have to be addressed in follow-up PRs.

Most runtime validation used existing GGUFs.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES. Codex substantially assisted implementation, review, testing. Code has been manually reviewed and the PR description has been written manually.

ggerganov · 2026-10-21T14:53:07Z

+    // note: this can technically be:
+    // e = ggml_scale(ctx0, amax, 1.0f/32768.0f);
+    // but we leave x32 headroom for the F16 sums of 32 values in Q8 activation blocks
+    e = ggml_scale(ctx0, amax, 1.0f/1024.0f);


This is related to: #30861 (comment)

A workaround until we have a better fix.

This is super interesting. I faced this problem when testing Mellum 2.1 on CUDA with a Q5 quant. Codex patched it inside ggml-cuda.cu with additional CUDA kernels conditionally downscaling the activations before calling ggml_cuda_mul_mat_id and scaling them back afterwards. Unsure if slop or viable option, thought mentioning here wouldn't hurt.

…l-org#30928 has been added and adapted.

pozz2906 · 2026-10-22T21:08:43Z

Related Flint bug report: #31000.

With Flint NVFP4 on b11823 and OpenCode 2.1.2, some turns contain literal tool-call XML only in reasoning and end with stop, without dispatching a tool. The report links captured messages, runtime settings, the active template and customized system-prompt text. The root cause has not yet been isolated between model output, template, parser and client.

docs/models/flint covers: - README: architecture, acquisition, usage, the quantizations the kernels accept, reasoning and tool-call behaviour, and memory. - QUALITY: GPU engine vs scalar CPU oracle vs llama.cpp logit agreement, tokenizer and template parity evidence, replay guarantees (batched == serial, split prefill, snapshots), known limits, and reproduction commands. - EXPERIMENTS: retained and rejected kernel and engine decisions. - BENCHMARKS: HTTP sweeps against llama-server on the same UD-Q4_K_XL GGUF, performance power profile. Headline results vs llama.cpp: pp2048 2372.9 vs 1665.8 tok/s (+42%), tg128 67.65 vs 53.77 tok/s (+26%), 131K-depth tg 30.99 vs 27.77 (+12%), 8-user decode 270.94 vs 194.85 tok/s (+39%), load 4.1 vs 5.3 s. Peak memory at 133K context is 1.3% higher. The pinned .#llama-cpp-reference (b11069) predates Flint. The reference is therefore an ad-hoc ROCm build of upstream llama.cpp 970b51ce, which includes Flint support from a37cc461 (ggml-org/llama.cpp#30928). Its revision and build flags are recorded in model-identities.json; a Nix pin is still needed. The model is also listed in the root README, docs/BENCHMARKS.md, docs/TESTING.md, check-docs and the benchmark-model skill.

Picks up Flint architecture support (ggml-org/llama.cpp#30928). Rebases llama-opencl-adreno-cpy-constreg.patch onto the gemv_noshuffle_q4_k glu kernel change from #30421. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>

Row declared with this spec. Scope, upstream anchors, design, risks, gates and owed items are in the spec; the issue records the same first wave. The reason this row exists now: the quantized artifacts are on disk and sha256-verified, llama.cpp's only flint support is an open PR from the model author (ggml-org/llama.cpp#30928, head 1b44e30775dcb2f93952116b05f89e5bd797f b00), vLLM has no flint code, and the peer (mistral.rs) has started a Flint implementation — so the architecture has a reference implementation to gate against and a second implementation to cross-check. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]

* chore(llama-cpp): bump to a37cc461 and drop the TTS patch Move the llama.cpp pin to a37cc461, the merge of the Flint architecture (ggml-org/llama.cpp#30928). The TTS task patch is gone: upstream now carries SERVER_TASK_TYPE_TTS, server_task::tts_inp, server_task_result_tts, server_context_meta::has_cap_tts and the mtmd gen-audio helpers. The patch is detected as already applied on the new pin. The gRPC server uses the same symbols. The bonsai and turboquant forks still lack them and keep using disable-tts-task.sh, so they are unchanged. The score task patch is still needed: upstream has no SERVER_TASK_TYPE_SCORE. It is regenerated against the new pin and applies with no offset or fuzz. Upstream added SERVER_TASK_TYPE_TRANSCRIBE_RT to server_task::need_logits(), which broke one hunk, so SCORE is added next to it by hand. The C++ backend was not built for this change. Assisted-by: Claude:claude-sonnet-5-5 * feat(gallery): add flint Add Recursive Flint, a 39.8B mixture-of-experts coding model with about 2.5B active parameters, from the Unsloth GGUF repository. It runs on llama.cpp with the Flint architecture support added upstream in ggml-org/llama.cpp#30928. Three quantizations are listed, with the default declaring the others as variants: UD-Q4_K_XL (25 GB), UD-Q2_K_XL (13.6 GB) for small hosts and UD-Q6_K_XL (36 GB) for larger ones. Sampling defaults follow the model card (temperature 1, top_p 0.95, top_k 20). The library file gallery/models/index.yaml is edited and gallery/index.yaml is regenerated from it. Assisted-by: Claude:claude-sonnet-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

The b11069 .#llama-cpp-reference predates Flint support, so Flint's comparisons used an unpinned ROCm build. .#llama-cpp-flint-reference builds upstream llama.cpp 970b51ce (build 11827, which includes Flint support from a37cc461, ggml-org/llama.cpp#30928). It uses the same ROCm overrides as the other references and installs llama-server-flint and llama-bench-flint, so it can sit in one shell with the other llama.cpp pins.

…0928 Flint is a 39.8B-A2.5B MoE coder: 256 routed experts top-6 plus a shared expert, 49 layers (2 dense + 47 MoE), hybrid attention (35 sliding-window layers, window 513, with the other 14 full attention), GQA 32 q / 4 kv, head dim 128, per-head QK RMSNorm, learned attention sinks, RoPE only on the sliding-window layers. Ported from ggml-org/llama.cpp ggml-org#30928 (a37cc46): - src/models/flint.cpp + the llama_model_flint entry in models.h - LLM_ARCH_FLINT in llama-arch, loader mapping and rope type in llama-model - vocab: LLAMA_VOCAB_PRE_TYPE_FLINT pre-tokenizer and the Flint end-of-generation ids - gguf-py: MODEL_ARCH.FLINT, its tensor list, the softmax_offset sinks mapping, and the flint.attention.rope_pattern key Prerequisites our base did not have, brought in because Flint needs them: - per-layer RoPE flags: hparams.rope_pattern, llama_hparams::has_rope(), LLM_KV_ATTENTION_ROPE_PATTERN and the default fill (all layers RoPE) - llama-kv-cache: only K-shift when a layer in the cache actually has RoPE - gguf-py: GGUFWriter::add_rope_pattern - llama-model-saver: Flint joins the archs the saver skips, because it keeps a per-layer SWA pattern and the saver does not write that key back yet test-llama-archs: Flint row (MoE mandatory, half-SWA pattern, rope pattern, expert weights norm/scale, shared expert FF). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…0928 (#134) Flint is a 39.8B-A2.5B MoE coder: 256 routed experts top-6 plus a shared expert, 49 layers (2 dense + 47 MoE), hybrid attention (35 sliding-window layers, window 513, with the other 14 full attention), GQA 32 q / 4 kv, head dim 128, per-head QK RMSNorm, learned attention sinks, RoPE only on the sliding-window layers. Ported from ggml-org/llama.cpp ggml-org#30928 (a37cc46): - src/models/flint.cpp + the llama_model_flint entry in models.h - LLM_ARCH_FLINT in llama-arch, loader mapping and rope type in llama-model - vocab: LLAMA_VOCAB_PRE_TYPE_FLINT pre-tokenizer and the Flint end-of-generation ids - gguf-py: MODEL_ARCH.FLINT, its tensor list, the softmax_offset sinks mapping, and the flint.attention.rope_pattern key Prerequisites our base did not have, brought in because Flint needs them: - per-layer RoPE flags: hparams.rope_pattern, llama_hparams::has_rope(), LLM_KV_ATTENTION_ROPE_PATTERN and the default fill (all layers RoPE) - llama-kv-cache: only K-shift when a layer in the cache actually has RoPE - gguf-py: GGUFWriter::add_rope_pattern - llama-model-saver: Flint joins the archs the saver skips, because it keeps a per-layer SWA pattern and the saver does not write that key back yet test-llama-archs: Flint row (MoE mandatory, half-SWA pattern, rope pattern, expert weights norm/scale, shared expert FF). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

reference/llama.cpp predates Flint support (ggml-org/llama.cpp#30928), and moving that pin would move the calibrated parity floors of every other checkpoint. reference/llama.cpp-flint sits at the Flint merge, a37cc461, and build-llamacpp.sh takes the checkout as an optional argument. This llama.cpp revision fails to build the server and cli targets unless the UI asset gzip step is off, so the script now passes LLAMA_UI_GZIP=OFF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FQuaJnVoxF8s1wvU9Antfa

* Add Flint model (FlintForCausalLM). Assisted by Codex * cleanup * one more --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

El PR ggml-org#30928 (flint) usa cuatro cosas de upstream que este arbol no tenia (son de commits de upstream posteriores a agosto-2026): - hparams.rope_pattern: el patron de RoPE por capa. El GGUF de flint lo trae ('<arch>.attention.rope_pattern': 1 = usa RoPE, 0 = NoPE) y sin el las capas densas se desplazarian con RoPE cuando no toca. - hparams.has_rope(il): si esa capa usa RoPE. El K-shift solo se aplica si alguna capa lo usa: una capa NoPE no se puede desplazar. - LLM_KV_ATTENTION_ROPE_PATTERN: la clave (enum de llama-arch.h, su tabla de cadenas y el LLM_KV de gguf-py). - hparams.n_ff_exp por capa: aqui n_ff_exp es un escalar (los demas MoE lo cargan asi). Se anade n_ff_exp_layer(il), que usa el patron por capa si lo hay y si no el escalar, y flint lo usa para dimensionar los tensores de los expertos. Ademas, flint necesita el reescalado del src1 del MoE (el amax de upstream, PR ggml-org#30656) porque su activacion puede desbordar f16. Este fork sustituyo ese mecanismo por el guard L2 propio (que solo actua en Metal), asi que se anade una rama especifica para FLINT que conserva la via de upstream (el reescalado solo afecta a la proyeccion 'down' y se deshace con scale_s). Y el enum de los pre-types del vocab: FLINT=62 no colisiona con INKLING=57 ni con SPARK2_5=58 (los valores no se persisten: el GGUF guarda el nombre, no el numero; y no se usan como indice).

Add Flint model (FlintForCausalLM). Assisted by Codex

1b44e30

alexey-rsi requested review from CISC and ggerganov as code owners October 21, 2026 14:48

alexey-rsi changed the title ~~Add Flint model (FlintForCausalLM). Assisted by Codex~~ Add Flint model (FlintForCausalLM) Oct 21, 2026

alexey-rsi changed the title ~~Add Flint model (FlintForCausalLM)~~ models: Add Flint model (FlintForCausalLM) Oct 21, 2026

ggerganov approved these changes Oct 21, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

github-actions Bot added model Model specific testing Everything test related conversion labels Oct 21, 2026

CISC approved these changes Oct 21, 2026

View reviewed changes

Comment thread conversion/flint.py Resolved

rick-github mentioned this pull request Oct 21, 2026

Error: 500 Internal Server Error with new model Flint ollama/ollama#19186

Closed

Siddhesh2377 mentioned this pull request Oct 21, 2026

models: Flint (FlintForCausalLM) on ra/main RunanywhereAI/llama.cpp#6

Merged

dbirks mentioned this pull request Oct 21, 2026

Evaluate Recursive Flint NVFP4 on Blackwell: vLLM/llama.cpp, 262K KV sizing and reclaimed HAMi VRAM dbirks/home-k8s#158

Open

Siddhesh2377 mentioned this pull request Oct 21, 2026

swift: pin mlx-swift-lm 3.31.13 and mlx-audio-swift 0.1.10 for Recursive Flint RunanywhereAI/runanywhere-sdks#1150

Closed

jimenezcarrero mentioned this pull request Oct 21, 2026

Lunar Lake runbook: Recursive Flint as a future MoE candidate jimenezcarrero/local-agent-arena#101

Merged

CISC reviewed Oct 21, 2026

View reviewed changes

Comment thread convert_hf_to_gguf_update.py Resolved

Comment thread conversion/base.py Resolved

Comment thread conversion/flint.py Resolved

CISC added 2 commits October 21, 2026 23:05

cleanup

0ca3972

one more [no ci]

1906989

CISC merged commit a37cc46 into ggml-org:master Oct 21, 2026
1 check passed

Bl4ckyou mentioned this pull request Oct 21, 2026

Model: Add Flint MOE runtime support ikawrakow/ik_llama.cpp#2725

Merged

2 of 4 tasks

midagedev mentioned this pull request Oct 21, 2026

model: Flint (flint, 256 experts, sliding-window attention with sinks) midagedev/bloomery#632

Closed

mengshengwu mentioned this pull request Oct 22, 2026

chore(sdk): bump llama.cpp submodule to c7fc50941 qualcomm/GenieX#1582

Closed

1 of 3 tasks

brunobergher mentioned this pull request Oct 22, 2026

Offer Recursive Flint as a model on this Mac lampdotdev/Lamp#562

Closed

leehack mentioned this pull request Oct 22, 2026

Qualify post-v0.6.0 upstream correctness fixes and consumer compatibility leehack/llamadart-native#112

Open

3 of 6 tasks

Simione932 added a commit to Simione932/llama.cpp that referenced this pull request Oct 22, 2026

Support for the Flint LLM architecture (FlintForCausalLM) from PR ggm…

80dca4b

…l-org#30928 has been added and adapted.

ggerganov mentioned this pull request Oct 22, 2026

flint : add initial model support ggml-org/convert#45

Merged

pirumpi mentioned this pull request Oct 22, 2026

Add Recursive Flint support via llama.cpp Metal kyndlo/tapioca#62

Open

sawansri mentioned this pull request Oct 22, 2026

Add Flint-GGUF lemonade-sdk/lemonade#3940

Merged

lu-zero mentioned this pull request Oct 22, 2026

docs(spec): the Flint port — CPU and GGUF arms (MODEL-TEXT-FLINT) localai-org/vllm.cpp#3503

Merged

max-krasnyansky mentioned this pull request Oct 22, 2026

hexagon: add CEIL op #31006

Merged

mudler-agent mentioned this pull request Oct 23, 2026

feat(gallery): add flint and bump llama.cpp to a37cc461 mudler/LocalAI#12854

Merged

This was referenced Oct 23, 2026

Support Recursive Flint (flint MoE, 39.8B/2.5B active): known gaps, bring-up plan, per-target tuning jamesburton/dotLLM#1157

Open

Flint F1: CPU flint GGUF arm (arch, config, no routed expansion, pre-tokenizer, Jinja \x escapes) jamesburton/dotLLM#1158

Closed

jamesburton mentioned this pull request Oct 23, 2026

models: Recursive Flint (GGUF flint) on CPU; experts stay quantised, F16-safe MoE down input (#1158) jamesburton/dotLLM#1167

Merged

bong-water-water-bong mentioned this pull request Oct 23, 2026

models: add Flint (FlintForCausalLM), ported from upstream #30928 1bit-MONSTER/llama.cpp#134

Merged

Andgihat mentioned this pull request Oct 25, 2026

models: add Flint (FlintForCausalLM) support on top of v0.4.9 Anbeeld/beellama.cpp#191

Closed

ggerganov mentioned this pull request Oct 28, 2026

llama.cpp : bump version to 0.8.0 #31309

Merged

brunobergher mentioned this pull request Nov 2, 2026

Run flint on this Mac, and never choose a local model llama.cpp can't load lampdotdev/Lamp#987

Open

ChristianHohlfeld mentioned this pull request Nov 3, 2026

coding: NUMA1 → Flint Q8_0 (pinned llama.cpp b12058), NUMA0 Qwen3.8 unchanged ChristianHohlfeld/mag-portable-stack#273

Merged

Navigation Menu

Repository navigation

models: Add Flint model (FlintForCausalLM) - #30928

models: Add Flint model (FlintForCausalLM)#30928
CISC merged 3 commits into
ggml-org:masterfrom
alexey-rsi:flint-model

alexey-rsi commented Oct 21, 2026

Uh oh!

ggerganov Oct 21, 2026

Uh oh!

Danmoreng Oct 21, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pozz2906 commented Oct 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

alexey-rsi commented Oct 21, 2026

Overview

Additional information

Requirements

Uh oh!

ggerganov Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Danmoreng Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pozz2906 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Danmoreng Oct 21, 2026 •

edited

Loading

pozz2906 commented Oct 22, 2026 •

edited

Loading