Use F16 for memory_k and memory_v (as suggested in #146) by ty-everett · Pull Request #154 · ggml-org/llama.cpp

ty-everett · 2023-03-15T06:16:30Z

As suggested in #146 we are able to save lots of memory by using float16 instead of float32. I implemented the suggested changes, and tested with the 7B and 13B models, and there were no issues on my Intel-based MacBook Pro.

Merging these changes should allow more models to run more performantly on a wider range of hardware.

Green-Sky · 2023-03-15T14:15:04Z

can confirm ggml ctx size 4529.34 MB -> 4273.34 MB
speed stayed the same.

it is hard to tell if the quality changes, but the prediction does (obviously).

ggerganov · 2023-03-15T19:14:41Z

I was worried that it might degrade quality, but I have no evals as you can guess.
I think it is best to gate this through a command line argument. Have it F32 by default, and if requested by the user - set it to F16

ggerganov

.

Green-Sky · 2023-03-16T14:52:53Z

I ran some more, non scientific tests:

7B:

30B:

both where ran with -t 4 -n 2048 --repeat_penalty 1.176 --repeat_last_n 256 --temp 0.8 --top_p 0.1 -c 2048 --color -i -r "User:" -f prompts/i_example1.txt

Green-Sky · 2023-03-18T00:00:17Z

@ty-everett are you going to write the cli-param conditional version? if not, I will do it.

…154) (#294) * Use F16 for memory_k and memory_v * add command line switch to use f16 instead of f32 for memory k+v --------- Co-authored-by: Ty Everett <ty@tyweb.us>

Slim-Bullseye based docker image

…gml-org#154) (ggml-org#294) * Use F16 for memory_k and memory_v * add command line switch to use f16 instead of f32 for memory k+v --------- Co-authored-by: Ty Everett <ty@tyweb.us>

* iq2_xxs_r4: Zen4 Disapointing gain: 134.7 t/s -> 151.1 t/s for PP-512 TG-128 is better: 3.45 -> 4.61 t/s @ 1 thread * Minor * iq2_xxs_r4: NEON --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

…gml-org#154) (ggml-org#294) * Use F16 for memory_k and memory_v * add command line switch to use f16 instead of f32 for memory k+v --------- Co-authored-by: Ty Everett <ty@tyweb.us>

… 570+ (ggml-org#154) Two NVIDIA driver bugs on Blackwell GPUs (RTX 50xx series) with Linux driver versions 570+ cause illegal memory access during llama.cpp KV cache init: Bug 1: cudaMemsetAsync with cudaStreamPerThread - cudaMemsetAsync(ctx->dev_ptr, value, buffer->size, cudaStreamPerThread) -> cudaStreamSynchronize -> illegal memory access in memset32 kernel - Fix: replace with synchronous cudaMemset (avoids per-thread async stream) Bug 2: cudaEventCreateWithFlags(event, cudaEventDisableTiming) - illegal memory access on compute capability 12.0 GPUs - Fix: replace with plain cudaEventCreate (no flags) These fix crashes during model load and KV cache initialization that affect all current llama.cpp CUDA builds on RTX 50xx hardware. Reported-by: Aggroed (aggressive.lighthacker@gmail.com) Bug classification: NVIDIA driver bug in 570+ series, sm_100/sm_120 Verified: RTX 5070 Ti (sm_120) and RTX 3090 Ti (sm_86), CUDA toolkit 12.8+ Co-authored-by: blair <aggressive.lighthacker@gmail.com>

…sing, J_MAX validation, split-state init)

hip: shared perf (MMQ MoE tiles + mmvdq, prefill plumbing)

Revert "server : add --cache-disk for a persistent prompt cache" (ggml-org#81)

## Motivation Fixes required to bump [`hrx-graph-develop-v2`](https://github.com/AMD-Ecosystem/llama.cpp/tree/hrx-graph-develop-v2) up to hrx-system: ROCm/hrx-system@5f4dfc8. Decode attention writes F32 results into global memory, then Q8 packing reads values written by other lanes. The intervening barrier fenced workgroup memory only. Fence global memory before packing in both reduction paths to prevent stale values from producing nonfinite Q8 scales and NaN perplexity. This appears to be a pre-existing llama.cpp synchronization bug, not a newly introduced HRX regression. The previous green PR105 package uses the same llama.cpp revision and older HRX `4341b71`; it also reproduces the failure locally. Newer HRX may have made the race more common by changing execution timing or generated code, but the controlled local comparison did not establish that: older and newer packages both produced NaNs in all ten alternating fresh-process repetitions. A second local comparison used the September 29 package from https://github.com/ROCm/ggml-staging-automation/actions/runs/36638317205, whose saved report explicitly passed this exact model and checksum with decode PPL 10.0045. That package (HRX `32b86d4`, llama.cpp `2ba2ddc`) and the current unfixed package each failed 10/10 alternating repetitions locally. Its llama.cpp revision differs, so this is a historical package comparison rather than an HRX-only comparison. Both old reduction paths already contain the same workgroup-only pre-packing barriers. Neither local comparison provides evidence of an increased failure rate with newer HRX. No introducing HRX commit is attributed. The HRX PR referenced by the fix commit documents the barrier API, not the introduction of this bug. ### Breaking changes | Breaking hrx-system PR | Fix | | --- | --- | | None identified; pre-existing llama.cpp race | AaronStGeorge@103a47d | ## Testing Tested in `ggml-staging-automation` full all-model CI with llama.cpp `103a47dd80e7` and hrx-system `5f4dfc8731a1`. - Bump PR: ROCm/ggml-staging-automation#108 - Full all-model CI: passed on strixhalo-10, https://github.com/ROCm/ggml-staging-automation/actions/runs/39701697708, staging head `1bad5e0d6acb`. The release benchmark and security checks passed. Llama-3.2-1B-Instruct decode PPL was 10.0408 +/- 1.32270; prefill PPL was 14.4414 +/- 0.45819. Both numerical checks and Vulkan references passed. The staging workflow supplies missing GPU device groups on canary runners and launches benchmarks with refreshed membership; no model expectations were changed. - Focused numerical validation: https://github.com/ROCm/ggml-staging-automation/actions/runs/39519720454. On DT241, DT243, and strixhalo-10, the original package failed 5/5 repetitions per host and the fix passed 5/5. Fixed decode PPL was 10.0408 +/- 1.32270; prefill PPL was 14.4414 +/- 0.45819. Vulkan reference checks passed. The run was red because DT245 could not initialize either package; it produced no model measurements. - Local CPU decode reference: 10.0388 +/- 1.32349. All 37 existing attention tests passed. - Compatibility floor remains `771711d49befd397f10c3c6deb66f1c5c89e5042`. Rebuilt HRX, Loom, the backend, and embedded kernel bytecode at that floor and ran the same fixed decode command successfully: PPL 10.0408 +/- 1.32270. - Reproduction command: `ROCR_VISIBLE_DEVICES=0 llama-perplexity -m Llama-3.2-1B-Instruct-UD-Q4_K_XL.gguf -f wiki.test.raw -c 512 --chunks 2 -b 512 -ub 1 --device HRX0` on gfx1151. Model and corpus are the pinned, checksum-verified inputs in the staging manifest. Device initialization hangs are separate from this numerical fix. DT239's captured stacks block in ROCr `BlitKernel::SubmitLinearCopyCommand` during HRX device-library initialization with both packages, even with Vulkan disabled, before any model executes. The underlying cause of that runtime hang is unresolved. Historical endpoint search for a llama.cpp frequency bisect: the September 7 package cannot execute this model's Q5_K matrix multiplication; September 16 produces PPL 41875.0725; September 23 produces NaNs. None is a locally correct good endpoint. Source history places the workgroup-only pre-packing barriers in `f43c5b4d9` (August 28), the initial Llama support change. This is source-history attribution of the missing fence, not a completed runtime bisect or proof of a later frequency increase.

## Overview Fence global F32 stores before the fused Qwen MoE normalization kernel reloads them for Q8 packing. Producer and packer workitems own different channel ranges; the existing workgroup-memory barrier does not order their global-memory accesses. This follows the same fix used for decode attention in ggml-org#154. The change replaces one barrier memory domain and documents why. It preserves the existing arithmetic and 32-byte LDS allocation. ## Validation On RX 7900 XTX (gfx1100), HRX `82214df`, upstream base `930e43fe9`, flash attention and F16 KV: - The retained model-trace checker fails on original captures and passes on six fresh fence-only captures. All 80 Q8 groups match canonical packing and repeat identically. The model-specific harness is retained as investigation evidence, outside this PR. - Qwen3-Coder 30B-A3B Q4_K_M full WikiText-2: 146 chunks, 149,358 scored tokens, 49/49 GPU layers. HRX PPL **8.1441 ± 0.06075**; Vulkan **8.1470 ± 0.06078**. All 146 printed HRX cumulative rows equal the previously validated shared-memory candidate. - Same-window five-repeat median pp256: fence **3589.39**, shared-memory candidate **3589.78** tok/s (-0.011%). Median tg128: **223.994** vs **223.642** tok/s (+0.157%). - Qwen3-30B shared-consumer smoke: two-token prompt exercised the Q8 wrapper, twelve-token prompt exercised the F32 wrapper; each produced identical greedy output for two HRX runs and Vulkan. A separate one-token decode path did not select the changed wrapper and produced differing text; that result is excluded from the wrapper validation claim and remains unresolved. - Fence LDS **32 B**, versus **8,224 B** for the shared-memory candidate; SGPR24/VGPR54 for both. ## Requirements - Contributing guidelines have been read by the agent; human review and acceptance remain pending for this draft. - AI usage disclosure: YES — Codex assisted with implementation, review, validation, and this PR description. Submitted as a draft for human review.

Use F16 for memory_k and memory_v

1b73521

ggerganov requested changes Mar 15, 2023

View reviewed changes

setzer22 mentioned this pull request Mar 15, 2023

Good ideas from llama.cpp rustformers/llm#15

Closed

2 of 6 tasks

This was referenced Mar 18, 2023

Add requirements to readme #269

Merged

RISC-V (TH1520&D1) benchmark and hack for <1GB DDR device #288

Closed

Command line switch to use F16 for memory_k and memory_v (refactor of #154) #294

Merged

ggerganov closed this Mar 19, 2023

univerz mentioned this pull request Mar 21, 2023

Use F16 for memory_k and memory_v rustformers/llm#51

Closed

Deadsg pushed a commit to Deadsg/llama.cpp that referenced this pull request Dec 19, 2023

Merge pull request ggml-org#154 from th-neu/th-neu-dockerfile-slim

38b8eee

Slim-Bullseye based docker image

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

hjotha pushed a commit to hjotha/beellama.cpp that referenced this pull request Sep 18, 2026

hip: address ggml-org#154 review (HIP-gated graph skip, strict DQ par…

31a7396

…sing, J_MAX validation, split-state init)

hjotha pushed a commit to hjotha/beellama.cpp that referenced this pull request Sep 18, 2026

Merge pull request ggml-org#154 from raufaser/pr-b-shared-perf

e60ef7c

hip: shared perf (MMQ MoE tiles + mmvdq, prefill plumbing)

pwilkin pushed a commit to pwilkin/llama.cpp that referenced this pull request Oct 8, 2026

Merge pull request ggml-org#154 from halo-box/revert/prompt-cache-disk

2e7bfc2

Revert "server : add --cache-disk for a persistent prompt cache" (ggml-org#81)

Navigation Menu

Repository navigation

Use F16 for memory_k and memory_v (as suggested in #146) - #154

Use F16 for memory_k and memory_v (as suggested in #146)#154
ty-everett wants to merge 1 commit into
ggml-org:masterfrom
ty-everett:master

ty-everett commented Mar 15, 2023

Uh oh!

Green-Sky commented Mar 15, 2023

Uh oh!

ggerganov commented Mar 15, 2023

Uh oh!

ggerganov left a comment

Uh oh!

Green-Sky commented Mar 16, 2023

Uh oh!

Green-Sky commented Mar 18, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ty-everett commented Mar 15, 2023

Uh oh!

Green-Sky commented Mar 15, 2023

Uh oh!

ggerganov commented Mar 15, 2023

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

Green-Sky commented Mar 16, 2023

Uh oh!

Green-Sky commented Mar 18, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants