Repository navigation
cuda : fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain - #30602
Merged
Merged
Conversation
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a gated template flag and computes scale_out*sigmoid(scale_in*p) from the raw weights, so the chain runs as one kernel instead of four.
github-actions
Bot
added
ggml
IMbackK
approved these changes
Oct 15, 2026
IMbackK
left a comment
IMbackK
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me from static analysis
am17an
added
the
merge ready
JohannesGaessler
approved these changes
Oct 18, 2026
3 of 6 tasks
wormley
pushed a commit
to wormley/llama.cpp
that referenced
this pull request
Oct 21, 2026
…g#30602 CUDA / ggml-org#29520 Vulkan) qwen4exp hc_combine builds its scatter weights as scale -> sigmoid -> scale feeding ggml_dsv4_hc_post; the SYCL backend ran that as 4 kernels per combine (~276 extra launches/token on a 45+1-layer model). Match the upstream hc_post_gate fusion: ggml_sycl_can_fuse() grows a branch for {SCALE, UNARY(SIGMOID), SCALE, DSV4_HC_POST} using ggml_can_fuse_subgraph + ggml_check_edges + the CUDA guards (no scale bias, F32, same shape, memory ranges), placed before the generic ggml_can_fuse() call because the gate feeds hc_post->src[2]. dsv4_hc_post_f32_sycl gains a gated template parameter applying gate_scale_out*sigmoid(gate_scale_in*p) with the same expression as the standalone kernels (numerically identical chain); ggml_sycl_op_dsv4_hc_post takes optional gate_scale_in/gate_scale_out like its CUDA counterpart. ggml_sycl_check_fusion_memory_ranges moves from topk-moe.cpp to fusion.cpp so both fusions share the CUDA-derived helper. GLM-5.3-Flash is unaffected by design (its post chain is tensor-affine MUL/ADD, not scalar SCALE). Gated by GGML_SYCL_ENABLE_FUSION (default on). Compile-validated with icpx (GGML_SYCL=ON, GGML_SYCL_DNN=ON, GGML_SYCL_F16=ON) in localhost/llama.cpp-intel:hc-post-fusion; GPU numerics/perf A/B pending — runbook in PLAN_HC_POST_SYCL_FUSION.md, image inventory in CONTAINERS.md.
wormley
pushed a commit
to wormley/llama.cpp
that referenced
this pull request
Oct 21, 2026
…_post kernel
glm5-next and deepseek4 (and dflash, via the shared builders) build their hc_post
weights as view(mixes) -> mul(scale_post) -> add(base_post) -> sigmoid -> scale(2)
feeding ggml_dsv4_hc_post: 5 dispatched kernels per hc_post, 2 per layer. The
qwen4exp scalar-gate fusion cannot match this chain (tensor affine, not scalar
SCALE), so it ran unfused. Extend the fused hc_post kernel with an affine gate
form: the kernel reads the raw mixes view, the scalar scale_post and the [hc]
base_post (all f32 device tensors, contiguous views of per-layer weights) and
applies scale*p + base[idst] followed by the existing sigmoid gate expression —
same op order as the standalone mul/add/sigmoid/scale kernels, numerically
identical. ggml_sycl_can_fuse() grows a {MUL, ADD, UNARY, SCALE, DSV4_HC_POST}
subgraph branch (before the generic ggml_can_fuse() call, like the qwen branch);
dispatch elides 4 nodes per chain (~192 launches/token at ngl 24, GLM recipe).
The op entry selects scalar vs affine by the leading chain node's op and takes
the bias operand as an optional third gate tensor. test-backend-ops grows an
affine test_dsv4_hc_post variant building GLM's exact chain (whole-graph vs CPU
reference). Fork-original extension of ggml-org#30602/ggml-org#29520 (upstream fuses only the
qwen scalar form); qwen4exp path unchanged.
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 24, 2026
…ml-org#30602) Port of the Vulkan HC_POST_GATE fusion (ggml-org#29520). hc_combine builds the hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a gated template flag and computes scale_out*sigmoid(scale_in*p) from the raw weights, so the chain runs as one kernel instead of four.
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
…ml-org#30602) Port of the Vulkan HC_POST_GATE fusion (ggml-org#29520). hc_combine builds the hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a gated template flag and computes scale_out*sigmoid(scale_in*p) from the raw weights, so the chain runs as one kernel instead of four.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the hc_post weights as 2sigmoid(inject/hc); dsv4_hc_post_f32 now takes a gated template flag and computes scale_outsigmoid(scale_in*p) from the raw weights, so the chain runs as one kernel instead of four.
Worth about 3% TG on RTX Pro 6000 on Qwen4-next
Overview
Additional information
Requirements