Skip to content

ggml: skip 0-sized ids tensor when offloading selected experts - #28739

Merged
pwilkin merged 1 commit into
masterfrom
aman/fix-offload-bug
Sep 11, 2026
Merged

pwilkin merged 1 commit into
masterfrom
aman/fix-offload-bug

Conversation

@am17an

@am17an am17an commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Overview

Fix issue when always offloading. Reproduce via:

GGML_OP_OFFLOAD_MIN_BATCH=0 ./build/bin/llama-bench -m ~/workspace/Qwen3.8-Flash-Next-GGUF/UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf -lm dio -ncmoe 999 -p 256 -ub 64

Zero-sized tensors appear in the last layer of graph for e.g. in qwen4exp:

if (il == n_layer - 1 && inp_out_ids) {
// everything below is per token, so drop the rows that produce no output
cur = ggml_get_rows(ctx0, cur, inp_out_ids);

when always offloading this loop does an out-of-bounds access:

int id = 0;
while (!ggml_bitset_get(used_ids.data(), id)) {
id++;
}

This PR exits early when the ids tensors is empty.

Additional information

Requirements

@github-actions github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Sep 11, 2026
@am17an am17an added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 11, 2026
@pwilkin
pwilkin merged commit 43f3dda into master Sep 11, 2026
37 of 41 checks passed
shikanime added a commit to shikanime-labs/machines that referenced this pull request Sep 11, 2026
qwen3.8-flash (qwen4_exp) deterministically aborts at warmup over RPC:
GGML_ASSERT(offset + size <= ggml_nbytes(tensor)) at ggml-backend.cpp:283.
Fix (skip 0-sized ids tensor when offloading selected experts) landed
upstream as ggml-org/llama.cpp#28739; the pinned unslothai fork rev has
diverged (33/43) and does not contain it. Repin to upstream master
8ea290247 (33 commits ahead of the old pin, includes #28739).

GLM-5.3-Flash (glm5next arch) is not yet in upstream master; serving it
waits on a later sync. Flash MoE and the rest of the preset are unaffected.

Design: pkgs/llama-cpp/default.nix
Related: shikanime-labs/manifests#2320
Signed-off-by: Shikanime Deva <william.phetsinorath@shikanime.studio>
shikanime added a commit to shikanime-labs/manifests that referenced this pull request Sep 11, 2026
Repin machines#1329 shipped llama.cpp upstream master 8ea290247 with the
sharded-MoE RPC warmup fix (ggml-org/llama.cpp#28739). Pin both apps to
the unified image; the dedicated llama-cpp-rpc image stopped publishing
with machines#1327 and RPC has no protocol handshake, so leader and
workers must track the same build.

Design: apps/llama-cpp/base/kustomization.yaml
Design: apps/llama-cpp-rpc/base/kustomization.yaml
Related: #2320
Signed-off-by: Shikanime Deva <william.phetsinorath@shikanime.studio>
@BrewTestBot BrewTestBot mentioned this pull request Sep 14, 2026
1 task done
liquidspikes added a commit to liquidspikes/llama.cpp that referenced this pull request Sep 16, 2026
quimmedes pushed a commit to quimmedes/cafe-llama.cpp that referenced this pull request Sep 16, 2026
x1250 pushed a commit to x1250/llama.cpp that referenced this pull request Sep 22, 2026
roverdrubber added a commit to roverdrubber/llama.cpp that referenced this pull request Oct 15, 2026
The branch is far behind upstream master and the parts that a card other than
this one needs had drifted. This ports the non-backend half of that sync,
content-matched against upstream rather than replayed by hash: the parser and
JSON schema chain (ggml-org#27764, ggml-org#28736, ggml-org#28742, which adds common/parsers/,
common/json-schema.* and tests/test-json-schema.cpp and retires the three
example scripts it replaces), the chat, grammar, jinja and server fixes
(ggml-org#28302, ggml-org#28469, ggml-org#26885, ggml-org#28279, ggml-org#28511, ggml-org#28620, ggml-org#28817, ggml-org#28749), the
qwen4exp RMSNorm and multiply fusion (ggml-org#28896, hand-ported because the norm
tensors here load with a different shape), the GDN q/k l2 norm as
flash-linear-attention defines it (ggml-org#28068), the ggml_prec_set_acc API that
replaces ggml_mul_mat_set_prec and ggml_flash_attn_ext_set_prec at every call
site, and the scheduler fixes for zero-sized expert ids (ggml-org#28739) and split
inputs (ggml-org#28387).

The GPU backend half of the same sync lands in the following two commits; this
one alone does not build the CUDA or HIP backend, because their call sites move
to the new precision API there.

test-chat, test-json-schema, test-json-schema-to-grammar,
test-grammar-integration and test-grammar-parser pass.

Assisted-by: Claude Opus 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants