Repository navigation
ggml-hrx: graph inputs are never written back to host; non-replay programs wait for their input upload - #90
Merged
bong-water-water-bong merged 2 commits intoOct 5, 2026
Conversation
The Qwen attention metadata kernel (qwen_attention_metadata, an initialization command) binds positions, the K/V cache indices and the KQ mask read_write and regenerates them on the device. Those tensors are ggml graph inputs in host memory, so materialize_host_bindings() staged them with download = true: after every prompt graph the device copies were queued for a deferred memcpy back into the caller's input tensors. Since e0f637b (fork #82) every host-staging upload first publishes pending writebacks. ggml has already written the next ubatch's inputs into that same memory, so the publish overwrote them with the previous ubatch's values before they were uploaded. Every ubatch after the first then took its context base (positions[0]) from the previous ubatch: multi-ubatch prompts were wrong. Measured (strixhalo, performance mode, Qwen3-0.6B Q4_K_M, llama-server HRX0 -fa on -c 16384, one 179-token request, then a 4,798-token prompt with cache_prompt false, ubatch 512): - 3527b56 + e0f637b alone, and e57beb9 (tip) with and without the HIP add-on: the model answers with the question text and no reasoning, stable hash 8d294327 (28/28 runs on the tip, 1/1 on 3527b56 + e0f637b). - 3527b56 (before #82): the expected answer ("The secret code for the vault is 7341" in the reasoning), hash db5701f7. - this change on e57beb9: the expected answer again, hash 5fb9f8b5. Writeback logging on 3527b56 showed four deferred writebacks per prompt graph, all written by qwen_attention_metadata: positions (2048 B), K and V cache indices (4096 B each) and the mask (up to 4.98 MB). A binding now carries graph_input (GGML_TENSOR_FLAG_INPUT on the tensor or its view source); such a binding is still uploaded but never downloaded. Values produced by the graph keep the deferred writeback that #82 publishes (the gpt-oss engine#286 case). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… host memory
execute_prepared_command_program() uploads graph inputs that live in a
registered host buffer with hrx_stream_copy_buffer, a device-timeline copy
that reads the host memory when it executes, and then returns without
waiting. ggml writes the next ubatch's inputs into the same host memory as
soon as graph_compute returns (llama.cpp only synchronizes there for
pipeline parallelism), so a queued copy could read the next ubatch's
positions, cache indices and mask. The graph replay path already waits for
its launch; this path (graph replay disabled, replay fallback, debug
execution) now waits too, only when such an upload exists.
Measured (strixhalo, performance mode, Qwen3-0.6B Q4_K_M, llama-server HRX0
-fa on -c 16384, 179-token request then a 4,798-token prompt with
cache_prompt false, GGML_HRX_DISABLE_GRAPH_REPLAY=1):
- before: wrong answer ("the number 123" instead of 7341), stable hash
b00928a0 (6/6 runs on the backported-Loom build, 6/6 on hrx-system
336037562); the audit builds showed the same split (replay on db5701f7,
replay off f6f810ca).
- after: the expected answer, hash 5fb9f8b5, bit-identical to graph replay
on (3/3 runs on the backported-Loom build; 2/2 completed runs on the
pinned Loom, whose third run hit the separate prompt-FA NaN).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
merged commit Oct 5, 2026
522dab4
into
1bit/hrx-vulkan-patched
10 of 24 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two host-side fixes found while root-causing engine ggml-org#313 (the NaN itself is a Loom compiler hazard, handled separately).
command-program-bindings.h,graph-executor.cpp,command-program-executor.cpp).qwen_attention_metadatabinds the positions, the K/V cache indices and the mask as read_write, so they were queued as deferred host writebacks. Since ggml-hrx: publish deferred host writebacks before a host-staging upload (engine #286) #82 every upload first publishes pending writebacks, which overwrote the next ubatch's fresh inputs with the previous ubatch's values: on the current tip every multi-ubatch Qwen prompt gives wrong output (Qwen3-0.6B, 4,798-token prompt: 28/28 runs repeat the question, hash 8d294327, with or without the private add-on; single-ubatch prompts are unaffected, which is why the e2e gates passed). Such bindings now carry agraph_inputflag: uploaded, never downloaded. Engine move file magic/version to header, print expected version ggml-org/llama.cpp#319.command-program-executor.cpp). With graph replay off, inputs were uploaded as a device-side copy and the call returned; ggml then wrote the next ubatch's inputs into the same host memory before the copy ran (the model answered "123" instead of 7341). Explains why replay on/off gave different stable bits.Checks (Strix Halo, performance mode, Qwen3-0.6B Q4_K_M): 4,798-token needle prompt 24/24 correct (hash 5fb9f8b5, finds 7341) on a build that also carries the Loom backport; graph replay off now 3/3 identical to replay on (before: 6/6 wrong); test-backend-ops -b HRX0 1073/1073. Rebased onto f95f2db (#89), incremental build clean.
Not in this PR: the ggml-org#313 NaN is a GFX11 wave64 SGPR lane-mask hazard in Loom at hrx-system 51b1739 (missing
s_waitcnt_depctrafter an SGPR pair is overwritten while a VALU still reads it as a lane mask); upstream fixed the class in ee77c8f7f/336037562, a backport exists locally; shipping it needs an hrx-system pin change.🤖 Generated with Claude Code