Repository navigation
metal: encode window over the paged (boundary-scheduled) compute path - #8
Merged
Merged
Conversation
…ode (ggml-org#226) Add a persistent, pointer-keyed boundary-event schedule to the Metal backend so an engine can drive async event-fenced KV residency without a per-boundary GPU host drain (the cb_eval ggml_backend_synchronize that dominates paged decode). - ggml_metal_set_boundary_schedule(): register cut nodes (each optionally signalling an event after its command buffer) and wait nodes (each waiting on an event before its segment), keyed by ggml_tensor* POINTER. PERSISTENT: it applies to every graph_compute until cleared (n_cuts==0), so it covers all of ggml_backend_sched's splits within one llama_decode and the reused graph of later tokens; pointer matching is robust to split-vs-full-graph index skew. n_cuts==0 leaves the stock n_cb encode path untouched. - ggml_metal_graph_compute_paged(): sequential per-boundary-committed encode (no dispatch_apply) that, for whichever schedule nodes appear in the graph it is handed, emits encodeSignalEvent after each boundary's command buffer and encodeWaitForEvent before an admitted region's first-reader segment. Reuses ggml_metal_op per segment (fusion preserved). A graph with no schedule nodes encodes as one segment. - Explicit-value event primitives: ggml_metal_event_encode_{signal,wait}_value and host-side ggml_metal_event_host_{signal,wait,value} so an engine reclaim/prefetch thread can coordinate an MTLSharedEvent across threads. - ggml_metal_get_device(): the backend's own device (ggml_metal_device_get() returns a fresh, unrelated device each call, whose events never reach the queue) -- callers must init boundary events on this device. Verified by a standalone probe over a real ggml-metal graph: the segmented encode is bit-identical to the stock encode, every per-boundary signal fires progressively, and an admit wait fences real GPU work until a concurrent host signal from a long-lived thread releases it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PR4CoRiTG8d5YfpjumYJSS
…#226) Adds ggml_metal_event_signal_on_complete: a host reclaim thread waiting the boundary event must see coherent shared-storage KV bytes, so signal from the completion handler rather than mid-stream. Separate WIP commit on top of the seam (542d159) to preserve; squash when formalizing the fork PR. Had zero effect on the paging bit-identity bug (keep-or-drop). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PR4CoRiTG8d5YfpjumYJSS
Exposes ggml_backend_metal_set_boundary_schedule plus the ggml_backend_metal_event_* primitives so an engine can drive fusion-preserving, event-fenced KV paging instead of a whole-graph host synchronize at each cut. Claude-Session: https://claude.ai/code/session_01GWUngsbi1CQNi4cBrmNFCG Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the multi-line block comments above the newly-added forward declarations to match the terse style of the surrounding headers, and strip references to external trackers/repositories from the remaining in-body comments.
ggml_metal_buffer_init returns NULL when the buffer allocation or the residency set fails, but ggml_backend_metal_buffer_type_alloc_buffer passed the result straight to ggml_metal_buffer_is_shared, which dereferences it at ->is_shared (offset 0x10). An allocation failure that ggml already logs and handles became a SIGSEGV. ggml-alloc.c already treats a NULL from alloc_buffer as a recoverable failure, so returning NULL is the expected contract. ggml_backend_metal_device_buffer_mapped had the same unchecked pattern around ggml_metal_buffer_map. That one built a buffer with a NULL context instead of faulting immediately; llama-model.cpp already checks the result for nullptr. Assisted-by: Claude Opus 4.8 Co-authored-by: Yi Wen Wong <yiwenwong@Yis-MacBook-Air.local>
While a boundary schedule is set, an optional encode window restricts ggml_metal_graph_compute_paged to the segments intersecting an inclusive [first_node, last_node] range, matched by pointer per graph like the schedule's own cut/wait nodes (a graph containing neither endpoint encodes in full). Optional blit pairs move a boundary activation into the window's first encoded segment (before its ops, after its schedule waits) and out of its last (after its ops) as device-side blits between resident tensors -- no host round trip. Persistent until cleared; cleared state is byte-identical to the pre-window paged path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ggml_metal_graph_compute_pagedto the segments intersecting an inclusive[first_node, last_node]node range. Endpoints are pointer-matched per graph exactly like the schedule's cut/wait nodes: a graph containing neither endpoint encodes in full (the multi-split decode case); a half-matching graph asserts loudly rather than silently mis-windowing.MTLBlitCommandEncodercopies between resident tensors, no host round trip.ggml_backend_metal_clear_encode_window) is byte-identical to the pre-window paged path.ggml_backend_metal_set_encode_window/ggml_backend_metal_clear_encode_windowinggml-metal.h, backed byggml_metal_set_encode_window/ggml_metal_clear_encode_windowon the context.This is the executor mechanism for banded (layer-major) prefill in the engine (titanium-chicken#524, landed engine-side via titanium-chicken#550): one
llama_decodeper (band, ubatch) tile, full graph built as today, only the band's segments encoded, with the residual stream carried across band boundaries through a persistent stash via the blit pairs.Branch note
tc-encode-windowis stacked directly onfbb8368e0— the rev the titanium-chicken engine pins — which has diverged from this branch's tip (d53c701reverts the NULL-check commitfbb8368e0carries). Merging this PR therefore also re-applies that NULL-check line of history; if the revert ontitanium_chickenwas deliberate, prefer reconciling the branches before merging.Test plan
encode_window_restricts_paged_compute(synthetic four-node chain through the paged path) — windowed compute from the blitted-in activation, out-of-window segment untouched, blit-out capture, cleared-window full recompute, foreign-graph window restricts nothing.make gategreen against an install built from this branch; paged path byte-identical with no window set.🤖 Generated with Claude Code
https://claude.ai/code/session_014Y3vqStW3Wcs2pae6Vi4Q5