model: add dots3-note by ngxson · Pull Request #27060 · ggml-org/llama.cpp

ngxson · 2026-08-14T11:42:42Z

Overview

Support https://huggingface.co/dots-studio/dots3-note-prev

Ref HF impl: huggingface/transformers#47844

So far, the biggest change is DSA + SWA, so the llama-kv-cache-dsa had to be extended

Additional information

This PR was tested with a slice (3 layers, 16 experts) from the real weight. Other configs and tokenizer stay unchanged

Logits validation (passed --> good/acceptable NMSE):

ctx	exercises	NMSE mean/max	top-1 (all pos)
1800	SWA window active, DSA dense-equivalent	5.0e-6 / 9.1e-6	99.11%
3200	+ real DSA top-k pruning (>2048)	2.6e-5 / 3.0e-4	98.78%

TODO:

decide if extending llama-kv-cache-dsa is ok
fix Metal mul_mat issue --> will be a separate PR
add MTP support
add vision + audio support --> dedicated branch, to be pushed as follow-up PR: xsn/dots3-note...xsn/dots3-note-mtmd
maybe fix the model saver/loader, not sure

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: most code generated by AI, human validated at each steps to make sure there is no invasive changes

ngxson · 2026-08-14T11:43:44Z

+    const bool has_swa = swa_type != LLAMA_SWA_TYPE_NONE;
+
+    const layer_filter_cb filter_mla_full = [&](int32_t il) {
+        if (filter_mla && !filter_mla(il)) {
+            return false;
+        }
+
+        return !(has_swa && model.hparams.is_swa(il));
+    };
+
+    const layer_filter_cb filter_mla_swa = [&](int32_t il) {
+        if (filter_mla && !filter_mla(il)) {
+            return false;
+        }
+
+        return model.hparams.is_swa(il);
+    };
+


CC @fairydreaming @ggerganov , I think adding SWA to DSA cache is not too invasive, do you think it's ok to keep this change as-is?

@ngxson From my understanding SWA in Dots3 is for non-DSA layers (they are ordinary MLA layers that don't use lightning indexer at all), so the cleanest solution that follows other memory classes would be probably to create llama_kv_cache_dsa_iswa class that would combine llama_kv_cache_dsa instance for DSA layers and llama_kv_cache instance for SWA layers.

that makes sense, I moved the logic to a new llama_kv_cache_dsa_iswa, PTAL: d9de2b2

ngxson · 2026-08-17T19:46:04Z

I think this PR is ready to be reviewed as-is

2 follow-up PRs planned: multimodal (see xsn/dots3-note...xsn/dots3-note-mtmd) and MTP support (TODO)

ngxson · 2026-08-21T17:52:25Z

tested the Q8_0 weight on a big mac 512GB RAM

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

Implement `seq_token_count` for the new upstream `llama_kv_cache_dsa_iswa` wrapper. Fixes: 5a32f7b ("model: add dots3-note (ggml-org#27060)") Signed-off-by: JenySadadia <jeny.sadadia@collabora.com>

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

Implement `seq_token_count` for the new upstream `llama_kv_cache_dsa_iswa` wrapper. Fixes: 5a32f7b ("model: add dots3-note (ggml-org#27060)") Signed-off-by: JenySadadia <jeny.sadadia@collabora.com>

Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 2d5615b) b10297 rebase: - implement seq_token_count for the new upstream llama_kv_cache_msa wrapper (delegates to kv_base, same pattern as llama_kv_cache_dsa). --- llama: add seq_token_count override for the DSA KV cache Signed-off-by: Marcus Edel <marcus.edel@collabora.com> (cherry picked from commit 0eeebe0) --- b9840 rebase: - seq_token_count() is a pure-virtual on llama_memory_i (added by "llama: add sequence token count memory API"), so every memory implementation must override it or it stays abstract. - upstream 8c146a8 ("DeepSeek V4", ggml-org#24162) introduced a second DSA cache class, llama_kv_cache_dsv4, alongside llama_kv_cache_dsa. The target commit only added the override to llama_kv_cache_dsa and missed dsv4, so on b9840 dsv4 is abstract and the tree fails to build. Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com> Squashed-with: d885b79b66bc, a6b9695be14d (cherry picked from commit 6c2e317) (cherry picked from commit 6bd56d7) b10796 squash: folded 1 commit. - b10796 rebase: fix `llama_kv_cache_dsa_iswa` --- From a466710: Implement `seq_token_count` for the new upstream `llama_kv_cache_dsa_iswa` wrapper. Fixes: 5a32f7b ("model: add dots3-note (ggml-org#27060)") Squashed-with: a466710 Co-authored-by: JenySadadia <jeny.sadadia@collabora.com> Assisted-by: Claude Opus 5 (cherry picked from commit b8ce533) (cherry picked from commit a0e0162)

Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 2d5615b) b10297 rebase: - implement seq_token_count for the new upstream llama_kv_cache_msa wrapper (delegates to kv_base, same pattern as llama_kv_cache_dsa). --- llama: add seq_token_count override for the DSA KV cache Signed-off-by: Marcus Edel <marcus.edel@collabora.com> (cherry picked from commit 0eeebe0) --- b9840 rebase: - seq_token_count() is a pure-virtual on llama_memory_i (added by "llama: add sequence token count memory API"), so every memory implementation must override it or it stays abstract. - upstream 8c146a8 ("DeepSeek V4", ggml-org#24162) introduced a second DSA cache class, llama_kv_cache_dsv4, alongside llama_kv_cache_dsa. The target commit only added the override to llama_kv_cache_dsa and missed dsv4, so on b9840 dsv4 is abstract and the tree fails to build. Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com> Squashed-with: d885b79b66bc, a6b9695be14d (cherry picked from commit 6c2e317) (cherry picked from commit 6bd56d7) b10796 squash: folded 1 commit. - b10796 rebase: fix `llama_kv_cache_dsa_iswa` --- From a466710: Implement `seq_token_count` for the new upstream `llama_kv_cache_dsa_iswa` wrapper. Fixes: 5a32f7b ("model: add dots3-note (ggml-org#27060)") Squashed-with: a466710 Co-authored-by: JenySadadia <jeny.sadadia@collabora.com> Assisted-by: Claude Opus 5 (cherry picked from commit b8ce533)

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

ngxson added 2 commits August 14, 2026 12:13

text: conversion

5784322

init impl

25b654c

github-actions Bot added model Model specific testing Everything test related conversion labels Aug 14, 2026

ngxson commented Aug 14, 2026

View reviewed changes

Comment thread src/models/dots3note.cpp Outdated

ngxson mentioned this pull request Aug 14, 2026

metal: mul_mat: fix src1 row stride edge cases with has_tensor #27064

Closed

gopinath87607 mentioned this pull request Aug 16, 2026

Feature Request: add dots3-note ikawrakow/ik_llama.cpp#2331

Open

4 tasks done

ngxson marked this pull request as ready for review August 17, 2026 19:46

ngxson requested review from CISC, JohannesGaessler and ggerganov as code owners August 17, 2026 19:46

CISC reviewed Aug 18, 2026

View reviewed changes

Comment thread conversion/dots3.py

Comment thread src/models/dots3note.cpp Outdated

Comment thread src/models/dots3note.cpp Outdated

ngxson added 2 commits August 18, 2026 12:13

Merge branch 'master' into xsn/dots3-note

b78b595

address review comments

1090236

ngxson requested a review from CISC August 19, 2026 11:56

ngxson assigned fairydreaming and unassigned fairydreaming Aug 19, 2026

ngxson requested a review from fairydreaming August 19, 2026 11:57

CISC reviewed Aug 19, 2026

View reviewed changes

Comment thread src/models/dots3note.cpp Outdated

Comment thread src/models/dots3note.cpp Outdated

ngxson added 2 commits August 19, 2026 15:26

fix rope

6f2bb72

Merge branch 'master' into xsn/dots3-note

6bb8fd0

CISC approved these changes Aug 19, 2026

View reviewed changes

move to a new llama_kv_cache_dsa_iswa

d9de2b2

fairydreaming approved these changes Aug 19, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into xsn/dots3-note

a3f308a

ngxson merged commit 5a32f7b into master Aug 21, 2026
28 of 29 checks passed

ngxson mentioned this pull request Aug 22, 2026

mtmd: support dots3-note vision+audio #27524

Merged

github-actions Bot mentioned this pull request Aug 22, 2026

Reddit News Daily 2026-08-22 gitlawr/reddit-daily-news#344

Open

cdanis mentioned this pull request Aug 23, 2026

speculative: draft-mtp for dots3note #27618

Open

therealkenc pushed a commit to therealkenc/llama.cpp that referenced this pull request Aug 24, 2026

model: add dots3-note (ggml-org#27060)

8c1e1c9

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

ggerganov mentioned this pull request Aug 25, 2026

llama.cpp : bump version to 0.3.0 #27696

Merged

gopinath87607 mentioned this pull request Aug 26, 2026

Speculative prefill #27692

Open

thecodacus pushed a commit to thecodacus/llama.cpp that referenced this pull request Sep 7, 2026

model: add dots3-note (ggml-org#27060)

37f9e11

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026

model: add dots3-note (ggml-org#27060)

a046a66

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

pl752 pushed a commit to pl752/llama.cpp that referenced this pull request Sep 15, 2026

model: add dots3-note (ggml-org#27060)

c988bfe

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026

model: add dots3-note (ggml-org#27060)

e3a7a0a

* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa

Navigation Menu

Repository navigation

model: add dots3-note - #27060

model: add dots3-note#27060
ngxson merged 8 commits into
masterfrom
xsn/dots3-note

ngxson commented Aug 14, 2026 •

edited

Loading

Uh oh!

ngxson Aug 14, 2026

Uh oh!

fairydreaming Aug 19, 2026 •

edited

Loading

Uh oh!

ngxson Aug 19, 2026

Uh oh!

Uh oh!

ngxson commented Aug 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngxson commented Aug 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ngxson commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

ngxson Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

fairydreaming Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngxson commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngxson commented Aug 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ngxson commented Aug 14, 2026 •

edited

Loading

fairydreaming Aug 19, 2026 •

edited

Loading

ngxson commented Aug 17, 2026 •

edited

Loading