model: use ggml_rope_set_offset()

[ngxson]

Overview

Supersede #27091

May depend on #27345

Tested with deepseekv4 gguf and confirmed to output bit-exact as master

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes

[CISC]

Can't this be applied to deepseek2, deepseek32, glm-dsa and kimi-k3/linear MLA as well?

[ngxson]

Upon closer look, unfortunately some models still genuinely need concat, this is because the logic looks like this:

q_nope = ggml_permute(ctx0, q_nope, 0, 2, 1, 3);
ggml_tensor * q_nope_absorbed = ggml_mul_mat(ctx0, layer.wk_b, q_nope);
q_nope_absorbed = ggml_permute(ctx0, q_nope_absorbed, 0, 2, 1, 3);

ggml_tensor * Q = ggml_concat(ctx0, q_nope_absorbed, q_pe, 0);

While it's still [nope|rope], the nope part need to go through an extra projection. So the problem now become "can we do mul_mat inplace" which I think not worth optimizing that far.

An interesting finding though, kimi-k3 doesn't use rope at all, despite using the term "rope" in variable name, likely because that part of code is copied from deepseek MLA.

Added this to docs:

Exception: some models apply an extra op to the nope part, for example deepseek32.cpp, and may not use this optimization. While RoPE can be applied selectively to a part of the head, the extra op may not, so these models still need views plus ggml_concat.

[fairydreaming]

Looks like this change increased compute buffer sizes, added some logs with galloc debug info in #27680

[fairydreaming]

@ngxson On a different note - shouldn't ggml_rope_set_offset() be present in context shift graph for models that need it? Or perhaps they all don't support context shift?

[gopinath87607]

@ngxson On a different note - shouldn't ggml_rope_set_offset() be present in context shift graph for models that need it? Or perhaps they all don't support context shift?

hi will it solve the unnnerrry buffer size issues if i remove this pr from the main because i have 4 gbu and i cant use it properly due to this uneven buffer size issues