Skip to content

hexagon: use DMA for contiguous dim1 CONCAT - #29404

Merged
max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hexagon-concat-dim1-dma
Sep 25, 2026
Merged

max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hexagon-concat-dim1-dma

Conversation

@kurquhar

Copy link
Copy Markdown
Contributor

Overview

Use DMA for contiguous dim1 CONCAT on hexagon backend

Additional information

Motivated by latency analysis of LFM2.5-DSpark, which showed that CONCAT occupied ~25% of the profiled HTP time (excluding CPU time).

The kernel implementation was leveraged from two other kernel classes: the scalar concat-ops.c and DMA-based DDR->DDR in cpy-ops.c.

The kernel is gated to contiguous tensors of the same type (FP32, FP16, or INT32), concat along dim1, and dim0/2/3 same size.

Eval

Compared to the baseline [ec92815], this change improved LFM2.5-DSpark pipeline performance by over 23% during ABBA testing. Several portions of this pipeline run on CPU by design, so thermal throttling can impact measurements significantly:

  • Baseline: mean 30.46 tok/s
  • New: mean 37.65 tok/s
  • Gain: +7.19 tok/s, +23.6%
  • All endpoints matched: 176/692 accepted/drafted, mean length 3.23.
  • Initial 10min cooldown and additional 180s between phases

Command used for eval:

python scripts/snapdragon/run.py --install-dir <pkg-location> --devices HTP0 -- \
  llama-cli \
  -m <model dir>/LFM2.5-2.6B-Q8_0.gguf \
  -md <model dir>/LFM2.5-2.6B-DSpark-Q8_0.gguf \
  --spec-type draft-dspark --spec-draft-n-max 10 --spec-draft-n-min 0 \
  --no-spec-draft-backend-sampling \
  -c 4096 -ngl 99 -ngld 99 -fa on -n 256 \
  -p "The sky appears blue because" \
  --temp 0 --no-warmup --no-jinja --reasoning off --no-display-prompt -st -lv 4

Backend Op Tests

Compared to the baseline [ec92815], this change does not change op test results.

  • Baseline: 48 [OK]
  • New: 48 [OK]

Command used for op tests:

python scripts/snapdragon/run.py --install-dir <pkg-location> --devices HTP0 -- \
  test-backend-ops -o CONCAT -b HTP0

Requirements

  • I have read and agree with the contributing guidelines yes
  • AI usage disclosure: Yes, OpenCode+Terra was used for reasoning, coding, and A/B testing. All generated code has been reviewed by me (human).

Assisted-by: OpenCode

@kurquhar
kurquhar requested a review from a team as a code owner September 25, 2026 00:51
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Hexagon labels Sep 25, 2026
@max-krasnyansky

Copy link
Copy Markdown
Member

@lhez please review/ack when you get the chance

@max-krasnyansky max-krasnyansky added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 25, 2026
@max-krasnyansky
max-krasnyansky merged commit ed319fe into ggml-org:master Sep 25, 2026
19 checks passed
feal87 added a commit to feal87/myllama.cpp that referenced this pull request Sep 25, 2026
Merge upstream commits:
  - llama: add llama_prec_policy + model-driven W4A4 path (ggml-org#24364)
  - llama: fix tensor split for fused qkv with uneven K/V head sizes (ggml-org#29294)
  - metal: split fa kernels into per-dtype libraries (ggml-org#29329)
  - metal: FWHT kernels for block widths above 512 (ggml-org#29095)
  - CUDA: fuse RMS_NORM + SCALE into one kernel (ggml-org#29393)
  - common: extract shared unicode path/string helpers (ggml-org#29415)
  - common,rpc: simplify fs_create_directory_with_parents() (ggml-org#29432)
  - rpc: include nb in the get_alloc_size cache key (ggml-org#29283)
  - [SYCL] support sparse FA (ggml-org#28796)
  - musa: fix PH1 operator failures and build issues (ggml-org#29193)
  - HIP: bump HIP_VERSION required for fp8 (ggml-org#29231)
  - opencl: add q5_k bin kernel (ggml-org#29401)
  - hexagon: add q5_k quant type support (ggml-org#29123)
  - hexagon: use DMA for contiguous dim1 CONCAT (ggml-org#29404)
  - mtmd: fix mel preprocessor in LFM2 audio (ggml-org#29403)
  - vulkan: fix legacy GLSLC without cooperativeMatrix (ggml-org#29409)
  - gguf-py: ByteLevel processing defaults bos/eos to False (ggml-org#29422)
  - gguf-py: TemplateProcessing has final word on add_special_token (ggml-org#29417)

Assisted-by: Pi
sky-mighty pushed a commit to sky-mighty/llama.cpp that referenced this pull request Sep 26, 2026
* hexagon: use DMA for contiguous dim1 CONCAT

Assisted-by: OpenCode

* hexagon: update CONCAT DMA for DMA64

Assisted-by: OpenCode
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
* hexagon: use DMA for contiguous dim1 CONCAT

Assisted-by: OpenCode

* hexagon: update CONCAT DMA for DMA64

Assisted-by: OpenCode
edwardyoon pushed a commit to edwardyoon/focus-llama that referenced this pull request Oct 7, 2026
* hexagon: use DMA for contiguous dim1 CONCAT

Assisted-by: OpenCode

* hexagon: update CONCAT DMA for DMA64

Assisted-by: OpenCode
(cherry picked from commit ed319fe)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Hexagon merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants