delta_net_base: refactor conv state by am17an · Pull Request #30878 · ggml-org/llama.cpp

am17an · 2026-10-20T12:49:33Z

Overview

Provides a 1-2% speed-up due to less copying of the conv state.

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure:

am17an · 2026-10-20T12:53:55Z

Probably this will not work with -sm tensor again

ggerganov · 2026-10-20T12:56:45Z

Probably this will not work with -sm tensor again

The test-llama-archs passes, so I think it should work.

am17an · 2026-10-20T13:07:28Z

That's weird, it fails for me with view of permuted tensor not implemented on a real test

ggerganov · 2026-10-20T13:25:41Z

Which model did you try?

I tried:

GGML_METAL_DEVICES=2 ./bin/llama-completion -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 -p "I believe the meaning of life is" -n 32 --top-k 1 -no-cnv -lv 4 -sm tensor

and this works.

am17an · 2026-10-20T13:28:02Z

You need to try with MTP

am17an · 2026-10-21T03:10:42Z

Okay now it works with -sm tensor. @ggerganov can you check if this fixes the GLM issues for you? Note that I don't fully understand the meta backend change

JohannesGaessler · 2026-10-24T08:42:20Z

+            case GGML_OP_ARANGE: {
+                // no srcs, every device computes the same values
+                split_state = {GGML_BACKEND_SPLIT_AXIS_MIRRORED, {0}, {1}, 1};
+            } break;
            case GGML_OP_PAD_REFLECT_1D:
            case GGML_OP_ROLL:
-            case GGML_OP_ARANGE:


Please keep the ops in the same order they're declared in ggml.h.

The ARANGE case is gone, the slot index is now a host input, so the op order matches ggml.h again.

JohannesGaessler · 2026-10-24T09:25:30Z

+        // permuted view: split the view dim that spans the src split axis, other dims must be inside one element of it or outside the span
+        if (ggml_is_permuted(tensor) && !ggml_is_permuted(tensor->src[0]) && axis >= 0 && axis < GGML_MAX_DIMS-1) {
+            const size_t nb_axis = tensor->src[0]->nb[axis];
+            const size_t nb_next = tensor->src[0]->nb[axis+1];
+            for (int dim = 0; dim < GGML_MAX_DIMS; dim++) {
+                if (tensor->ne[dim] == 1 || tensor->nb[dim] < nb_axis || tensor->nb[dim]*tensor->ne[dim] != nb_next) {
+                    continue;
+                }
+                bool others_ok = true;
+                for (int i = 0; i < GGML_MAX_DIMS; i++) {
+                    if (i == dim || tensor->ne[i] == 1) {
+                        continue;
+                    }
+                    others_ok = others_ok && (tensor->nb[i]*tensor->ne[i] <= tensor->nb[dim] || tensor->nb[i] % nb_next == 0);
+                }
+                if (others_ok) {
+                    return {ggml_backend_meta_split_axis(dim), {0}, {1}, 1};
+                }
+            }
+        }


What tensor shapes does this cover and where in the modified compute graphs does this show up?

Note that I don't fully understand the meta backend change

This just fixes the permuted view of tensor is not supported in the meta backend. This is triggered by using the refactor in this PR, MTP + -sm tensor

The only permuted view in this PR is the slots view of conv_states_all in build_conv_state, shaped [d_conv - 1, n_rs_seq + 1, channels, n_seqs] with the slot stride larger than the channel stride, so it is only built with n_rs_seq > 0 (MTP or another draft that needs rollback). The source is split along channels and the new code maps that split to dim 2 of the view.

ggerganov · 2026-10-25T13:53:01Z

Okay now it works with -sm tensor. @ggerganov can you check if this fixes the GLM issues for you? Note that I don't fully understand the meta backend change

I tried combining both this PR and #30499 on top of latest master (see the gg/glm5 branch) and it still crashes like before:

[51781] 3.55.141.637 I cmn  common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)
[51781] /home/ggml/development/ggterm/llama/llama.cpp/ggml/src/ggml.c:1824: GGML_ASSERT(obj_new) failed
[51781] 3.55.190.035 W ggml_new_object: not enough space in the context's memory pool (needed 400752, available 400384)

am17an · 2026-10-25T14:22:15Z

Yes we need to port this change to glm5.3 next. Will push something soon

ServeurpersoCom · 2026-11-02T06:50:19Z

Tested in a rootless container with an RTX PRO 6000, llama-server with Qwen3.8-27B UD-Q8_K_XL and MTP only (--spec-draft-n-max 3, so 4 rollback slots), greedy, a 24.5k token prompt then 512 generated tokens, against the base of this PR.

Without this PR decode runs at 97.66 t/s.

With this PR decode runs at 97.93 t/s, so +0.27%, with the same output token for token and the same 345/495 accepted drafts. Prefill is unchanged within noise.

This PR does not touch qwen4exp, which has its own build_conv_state_at, so #31443 is independent of it.

@am17an small nit, nothing blocking: the slot index could be a host filled I32 input instead of arange + argsort, which would also drop the ARANGE case from the meta backend. Happy to push that commit on your branch if you want.

The rollback slot index of the conv windows is a constant per graph shape. It is now an I32 input of the recurrent memory filled on the host, instead of an arange and an argsort computed on the device, and the meta backend no longer needs a mirrored split for ARANGE.

am17an · 2026-11-02T07:18:21Z

@ServeurpersoCom did you check prefill at 2048 sized batches? because that's where it would matter

ServeurpersoCom · 2026-11-02T07:23:17Z

@ServeurpersoCom did you check prefill at 2048 sized batches? because that's where it would matter

Yes, the prefill numbers are with -b 4096 -ub 2048, so the 24.5k prompt goes through as 12 ubatches of 2048 with MTP active. Prefill stays within noise: 4144 t/s without this PR and 4160 t/s with it, with about 1% spread between runs.

Decode is where it shows: 97.66 t/s without this PR and 97.93 t/s with it, and the runs do not overlap (97.64 to 97.68 vs 97.91 to 97.95). On a dense 27B that is bound by memory bandwidth, there is very little left to win on the graph side, so a stable gain there is worth having. When decode is this memory bound, the per node cost is one of the few things left to shave, so fewer nodes is a free win.

ServeurpersoCom

LGTM, decode gain is small but reproducible with identical output, and the branch merges cleanly with master. The gpu-webgpu-apple failure is the deepseek4 comp_padding reallocation fixed on master by #31422, a ~~re-run~~ rebase will pick it up.

Also checked --parallel 4 with MTP (4 concurrent requests, greedy, 6 runs on each side): batching makes the output non deterministic on master too, and this PR forks at the same points with the same acceptance, while sequential runs are identical. Pure CPU gives identical output in both sequential and parallel runs, at the same speed.

Picked from ggml-org#30878 (open at the time of picking), head d9958cc, net diff against its merge base 2280f0f: 2e7a5fb delta_net: access conv state via views d9958cc fix sm tensor

ServeurpersoCom · 2026-11-03T07:28:27Z

For the record, rocm and vulkan-amd are not from this PR: they stop about 10 seconds after a successful checkout with "Can't find action.yml under .github/actions/ccache-buckets" on the new ggml-hfc-amd-r9700 runners, which run each step through Kubernetes container hooks, and the same failure shows up on other PRs. gpu-rocm stays queued until the 24h cancel, since no runner carries the self-hosted Linux AMD labels anymore. WebGPU is green locally with #31422 on top: the main ctest suite passes and test-llama-context passes the rollback cases on qwen35 and qwen35moe.

am17an · 2026-11-03T07:48:46Z

@ServeurpersoCom will you create a follow-up for doing this for qwen4exp and glm5.3-next or should I?

ggerganov · 2026-11-03T07:51:03Z

+        // window w starts at column n - (Kw - 1) + w (ascending time) and is rollback slot Kw - 1 - w
+        // slots reaching back past this ubatch are left unwritten, like the ssm snapshots in build_recurrent_attn
+        // the windows overlap, so the view is created with one window and widened after the size check
+        ggml_tensor * windows = ggml_view_4d(ctx0, conv_input,
+                conv_kernel_size - 1, 1, conv_channels, n_seqs,
+                ggml_element_size(conv_input), conv_input->nb[1], conv_input->nb[2],
+                ggml_row_size(conv_input->type, n - (Kw - 1)));
+        windows->ne[1] = Kw;


I'm taking a second look at the change and I don't get why we need to widen the view like this. What is the issue with directly using Kw in the ggml_view_4d?

because of this reason, which I guess should be fixed in ggml, i.e. it doesn't take into account the nb, and the view has to be overlapping

llama.cpp/ggml/src/ggml.c

Lines 1802 to 1809 in ac8e169

size_t data_size = ggml_row_size(type, ne[0]);

for (int i = 1; i < n_dims; i++) {

data_size *= ne[i];

}

GGML_ASSERT(view_src == NULL || data_size == 0 || data_size + view_offs <= ggml_nbytes(view_src));

void * data = view_src != NULL ? view_src->data : NULL;

ServeurpersoCom · 2026-11-03T08:21:08Z

@ServeurpersoCom will you create a follow-up for doing this for qwen4exp and glm5.3-next or should I?

Yes, I'll take it, as two follow-ups.

The first one is in ggml: the bound check of a view will be computed from its strides, like ggml_nbytes does, instead of assuming a contiguous layout. Overlapping views then become legal as they are, and the widening trick Georgi asked about goes away.

The second one ports the single set_rows write to the other archs that still copy the conv state once per rollback slot, starting with qwen4exp and glm5-next. I'll go with ggml first so the second one can use clean views from the start.

ServeurpersoCom · 2026-11-03T14:37:10Z

Follow-up 1/2: #31584

Follow-up 2/2: e72982c (depends on #31584, it creates the overlapping conv windows view directly), ports the single set_rows conv state write to qwen4exp, glm5-next, kimi-k3, bailingmoe3, lfm and mamba2.

…ore of the rollback slots (ggml-org#30878) Since ggml-org#30878 qwen35 / qwen35moe write the K rollback slots of the conv state with one set_rows over min(n_t + 1, K) overlapping windows of the conv input (index rs_s_slot_rev = {Kw - 1, ..., 0}) instead of K copies, so the fusion stopped matching on every model with rollback slots (MTP / n-gram drafts). Match that form as Kw slots: slot s gets the columns starting at n_t - s, slots >= Kw stay unwritten, as in the unfused graph. test-backend-ops: GDN_DECODE_FUSION sr variants. Assisted-by: Claude Opus 5.5 (Claude Code)

am17an requested a review from CISC as a code owner October 20, 2026 12:49

am17an closed this Oct 20, 2026

github-actions Bot added the model Model specific label Oct 20, 2026

am17an reopened this Oct 21, 2026

am17an requested a review from JohannesGaessler as a code owner October 21, 2026 03:09

github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Oct 21, 2026

ggerganov self-assigned this Oct 21, 2026

JohannesGaessler reviewed Oct 24, 2026

View reviewed changes

am17an added 2 commits October 25, 2026 15:33

delta_net: access conv state via views

2e7a5fb

fix sm tensor

d9958cc

ggerganov force-pushed the aman/conv-state-setrows branch from 3982664 to d9958cc Compare October 25, 2026 13:34

am17an mentioned this pull request Nov 2, 2026

qwen4exp: write the conv state tail without an intermediate cont #31443

Merged

ServeurpersoCom self-requested a review November 2, 2026 06:37

ServeurpersoCom requested a review from a team as a code owner November 2, 2026 07:19

ServeurpersoCom approved these changes Nov 2, 2026

View reviewed changes

ggerganov approved these changes Nov 3, 2026

View reviewed changes

ServeurpersoCom merged commit b74298c into master Nov 3, 2026
34 of 41 checks passed

ggerganov reviewed Nov 3, 2026

View reviewed changes

ServeurpersoCom mentioned this pull request Nov 3, 2026

ggml: bound views by the bytes their strides reach #31584

Merged

ggerganov mentioned this pull request Nov 4, 2026

ggml : bump version to 0.29.0 ggml-org/ggml#1680

Merged

BrewTestBot mentioned this pull request Nov 4, 2026

ggml 0.29.0 Homebrew/homebrew-core#323401

Merged

1 task

ServeurpersoCom mentioned this pull request Nov 4, 2026

graph: store the MTP conv snapshots of every recurrent model in one set_rows #31660

Merged

nbritton mentioned this pull request Nov 6, 2026

Merge llama.cpp v0.9.0 (b12110) mxxm-t/mx-llama.cpp#23

Open

	size_t data_size = ggml_row_size(type, ne[0]);
	for (int i = 1; i < n_dims; i++) {
	data_size *= ne[i];
	}

	GGML_ASSERT(view_src == NULL \|\| data_size == 0 \|\| data_size + view_offs <= ggml_nbytes(view_src));

	void * data = view_src != NULL ? view_src->data : NULL;

Conversation

am17an commented Oct 20, 2026

Overview

Additional information

Requirements

Uh oh!

am17an commented Oct 20, 2026

Uh oh!

ggerganov commented Oct 20, 2026

Uh oh!

am17an commented Oct 20, 2026

Uh oh!

ggerganov commented Oct 20, 2026

Uh oh!

am17an commented Oct 20, 2026

Uh oh!

am17an commented Oct 21, 2026

Uh oh!

JohannesGaessler Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

am17an Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Oct 25, 2026

Uh oh!

am17an commented Oct 25, 2026

Uh oh!

ServeurpersoCom commented Nov 2, 2026

Uh oh!

am17an commented Nov 2, 2026

Uh oh!

ServeurpersoCom commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented Nov 3, 2026

Uh oh!

Uh oh!

am17an commented Nov 3, 2026

Uh oh!

ggerganov Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

am17an Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented Nov 3, 2026

Uh oh!

ServeurpersoCom commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ServeurpersoCom commented Nov 2, 2026 •

edited

Loading

ServeurpersoCom left a comment •

edited

Loading