Repository navigation
metal: check for NULL from ggml_metal_buffer_init / _map - #6
Merged
wywen merged 1 commit intoJul 20, 2026
Merged
Conversation
ggml_metal_buffer_init returns NULL when the buffer allocation or the residency set fails, but ggml_backend_metal_buffer_type_alloc_buffer passed the result straight to ggml_metal_buffer_is_shared, which dereferences it at ->is_shared (offset 0x10). An allocation failure that ggml already logs and handles became a SIGSEGV. ggml-alloc.c already treats a NULL from alloc_buffer as a recoverable failure, so returning NULL is the expected contract. ggml_backend_metal_device_buffer_mapped had the same unchecked pattern around ggml_metal_buffer_map. That one built a buffer with a NULL context instead of faulting immediately; llama-model.cpp already checks the result for nullptr. Assisted-by: Claude Opus 4.8
wywen
pushed a commit
that referenced
this pull request
Jul 21, 2026
* ggml-et: Add performance logging * ggml-et: Quants helpers * ggml-et: Add MUL_MAT kernel * ggml-et: Add ROPE kernel * ggml-et: Add RMS_NORM kernel * ggml-et: Add GLU kernel * ggml-et: Add SOFT_MAX kernel * ggml-et: Add GET_ROWS kernel * ggml-et: Add CONT kernel * ggml-et: Add SET_ROWS kernel * ggml-et: Add MUL_MAT_ID kernel * ggml-et: Build et kernels as part of ggml * ggml-et: Embed kernels with fs fallback * ggml-et: Build fixes * ggml-et: Add MUL_MAT F32xF32 op * ggml_et: Add MUL_MAT_ID op * ggml-et: Disable offloading for debug * ggml-et: Refactor out block ops * ggml-et: ggml backend API changes * ggml-et: Add RESHAPE/TRANSPOSE to supported * ggml-et: Add CONT_F16 * ggml-et: Add supported ops doc * gglm-et: Initial doc * ggml-et: Remove runtime import hacks We can now import the runtime by a simple find_package(), so we can cleanup the CMakeLists.txt. * ggml-et: Fix GET_ROWS kernel Fix lost batch dimension. Also clean vibe-comments. * ggml-et: Fix SET_ROWS kernel Remove incorrect broadcasting guard. * ggml-et: Use custom instruction for fp32->fp16 * ggml-et: Vectorize set_rows fp32->fp16 * ggml-et: Fix ROPE kernel (yarn) ggml-et: fix et_logf WIP: Fix ramp WIP: fix ROPE! * ggml-et: Better sinf * ggml-et: Fix SOFT_MAX Add `max_bias` and `sink` support. * ggml-et: Fix CONT Reorder from contiguous write to read with atomic stores. * ggml-et: Fix elmap kernel Remainder handlin * ggml-et: Fix MUL_MAT MUL_MAT_ID remainders * ggml-et: Fix ET-SOC reference * ggml-et: Fix embed kernels scripts for old python This allows GGML-ET to build on pre-3.8 python. * Add sysemu support with compile time flag `-DGGML_ET_SYSEMU=ON` (#6) * Example using ET-Soc-1 emulator configuration Example usage: ```bash cmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON cmake --build build --config Release -j $(nproc) time ./build/bin/test-backend-ops ./build/bin/llama-server \ --model Qwen3-0.6B-Q8_0.gguf \ --alias Qwen3-0.6B-Q8_0 \ -fa 0 \ --ctx-size 1024 \ --no-warmup \ --host 127.0.0.1 \ --port 8080 ``` * build: proper dep tracking for kernels * support host using MOLD linker * initial multi core GET_ROW F32 implementation * vectorized q8 dequant * wip: cland warning clenaups and initial logging refactor * wip: message default message cleanup * chore: message cleanups * cmake cleanup * migrate to use platform provided functions * cmake back into subdir * support et_print() in kernels * fix: repair kernel building * perf: operations run async by default * debug: proper kernel dep tracking and error detection on kenrel launch * fix: kernel binary dep tracking and fixing get_rows_f32 erroring * perf: back to doing async kernel runs by default * perf: vectorize and parallel device memset * merge matmul work * misc: align allocation and enable all offload * misc: delete deadcode and respect memory limits * fix: repair tensor debug print * fix: loosen RMS_NORM op percision * feat: Q4_0 GET_ROWS * perf: FP32 MUL_MAT using TensorFMA * update limitations * perf: redue L1 load in compute_block_dot_product_q8_0 * feat: save kernel mapping (name to id) when profiling is enabled * chore: memops cleanup * perf: parallelize softmax by rows * perf: vectorize 2nd phase of softmax * perf: ban GET_ROWS from offloaded * perf: vectorize and non-atomic for eltwise ops and sub support * perf: vectorize normal rope * perf: glu runs in parallel * merge: manually merge saqib's work on kernel fixes * perf: more vectorized RoPE * perf: parallelize mul_mat_id * perf: parallelize set_rows_f32 * perf: vectorize softmax * feat: support kernel fusion and fuse RMS_NORM + MUL * fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE * fix: bump max rope dims for gemma * feat: GeGLU and SCALE support to fully offload Gemma * perf: faster device memset * feat: get_rows supporting Q4_K and avoid cont cache coherent issues * better F32 MM * feat: NORM for ET backend * feat: SQR for ET backend * feat: UNARY on ET * feat: el_map support broadcasting for ET * feat: SUM_ROWS in ET backend * feat: more ops in ET backend * feat: WKV* operators in ET backend * perf: parallelize operators across cacheline instead of row * perf: parallelize get_rows on cacheline * wip: baseline FlashAttention for ET backend * wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on * feat: f16 x f16 -> f32 MM using matrix engine * wip: f16 FlashAttention using matrix engine * wip: clean up * feat: barriers * perf: optimize FA_F16 in ET * perf: vectorize pack_k_for_transpose16 * perf: prefetch next loop matrix tile * perf: FlashAttention 2nd MM uses TensorFMA and optimizations * cleanup: flashattention reorg * perf: optimizations and fixes * feat: L2SCP API and make FlashAttention support DV = 256 for gemma * perf: parallelize norms beyond single row * feat: GATED_DELTA_NET support and relaxed L2_NORM requirment * feat: loosen RMS_NORM, NORM, ROPE contingous req too * feat: repeat supports brocasting on dim 0 and loosen cont check * feat: FILL and DIAG operator * feat: loosen UNARY support chcek * feat: TRI support * feat: SOLVE_TRI support * feat: basic SET support * feat: loosen CONT req * perf: fp16_to_fp32 use ASM * feat: IMROPE support * feat: PAD support * feat: global barrier * fix: view must live on the same backend as backing tensor * feat: relax CONCAT in ET backend * feat: dead simple CUMSUM implementation * feat: basic SSM_CONV support * feat: loosen CONCAT req * feat: relax GATED_DELTA_NET and add SET support proper * cleanup: cleanup LCM math * feat: SWIGLU single input * feat: SSM_SCAN support * feat: el_map supports non aligned tensors in best effort * feat: basic GROUP_NORM support * feat: loosen MUL_MAT capablities slightly * feat: loosen MUL_MAT and GET_ROWS and add IM2COL * feat: special case for softmax 1x1x1x1 * feat: loosen SOFT_MAX req in ET backend * fix: el_map unaligned acse fixes * perf: optimize zero_acc_vec in flash_attn_ext_f16_me * perf: use hart 1 for packing in MM and FA for FP16 * feat: kernel semaphore * perf: better instruction sequence in FlashAttention * fix: gated_delta_net with proper masking * perf: better parallelization for GATED_DELTA_NET * perf: parallelize SSM_CONV over nr * perf: vectorize SSM_CONV * perf: optimize MUL_MAT for q8 * feat: support Gemma 4 * fix: support multi-device * feat: broader GLU support * feat: unary ops supports view * fix: repair fp16 MM using matrix engine * perf: handle large N GEMV better * perf: better q8_0 MM * perf: better set_rows * add back deleted files * fix: repair after merge * feat: POC version of uberkernel * feat: RMS_NORM in uberkernel * feat: add more kernels into usage * chore: clean up uberkernel compilation * perf: faster flash attention * perf: opt flash attention for large seq length * feat: loosen op bounds. clamp and mean support * perf: vectorize ssm_scan * perf: slightly faster FA * perf: FlashAttention parallel MM and load * perf: fuse Q8 MM and ADD * feat: basic conv kernel for ET * softMAx_test * set_rows_f32 * get_rows and cont * testing * set_rows_exp * Junk addition * Narrowing the issue * Update flash_attn_ext_f16_me.c Focusing FA_ext_f16_me * test * Eviction updated * Detailed cache eviction debug * mulmat * removeal of `BUILD_FOR_UBERKERNEL` flag * cleaning... * fix: balance FCC0 count * feat: implement mul_mat and mul_mat_id for Q4_0 type * optimize uberkernel plan upload * add mul_mat q4 into uberkernel * enable gating flush to just uberkernel * update docs for ET * update op support for ET * et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations * et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0 * et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs * test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage * et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32 * et-backend: vectorize Q4_0 matrix-engine dequantization * et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail * et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat * et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency * Update uberkernel.c * Update unary_f32.c * gemma 4 * bisect gemma4: enable scale_f32 only * bisect gemma4: +rms_norm_f32 * bisect gemma4: +rms_norm_mul_f32 * bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT * bisect gemma4: +rope_f32 (skip rms_norm_mul) * bisect gemma4: +el_map_f32 * bisect gemma4: +softmax_f32 * bisect gemma4: +get_rows_f32 * bisect gemma4: +glu_f32 * bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me * bisect gemma4: +mul_mat_id_f32 * bisect gemma4: +sum_rows_f32 * bisect gemma4: +cont_f16 * bisect gemma4: +fill_f32 * bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul) * Update rms_norm_mul_f32.c * bisect2 gemma4 n64: +scale_f32 only * bisect2 gemma4 n64: +rms_norm_f32 +rope_f32 * bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix) * bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul) * bisect2 gemma4 n64: all ops enabled except rms_norm_mul * bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn) * bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0) * bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks) * bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0) * run-3: matmul + rms_norm_mul * run-4 * Revert "run-4" * run5 * changes after cleanup * cleanup before upstream * restrict changes into ET backend * move kernel embedding from Python to CMake * move uberkernel gen into CMake * apply clang format * update CMake style * update to match C and C++ style * use source ggml and quant headers instead of ET's * MROPE support * absorb view ops into same branch as none * fix bad rebase * add marty1885 to codeowners * oops * remove redundant newline * fix CI editor warnings --------- Co-authored-by: Vidas <vidas@nuolat.lt> Co-authored-by: Gianluca Guida <glguida@tlbflush.org> Co-authored-by: Gianluca Guida <gianluca@nekko.ai> Co-authored-by: ubergarm <leimgrub@gmail.com> Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai> Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>
wywen
added a commit
that referenced
this pull request
Jul 21, 2026
ggml_metal_buffer_init returns NULL when the buffer allocation or the residency set fails, but ggml_backend_metal_buffer_type_alloc_buffer passed the result straight to ggml_metal_buffer_is_shared, which dereferences it at ->is_shared (offset 0x10). An allocation failure that ggml already logs and handles became a SIGSEGV. ggml-alloc.c already treats a NULL from alloc_buffer as a recoverable failure, so returning NULL is the expected contract. ggml_backend_metal_device_buffer_mapped had the same unchecked pattern around ggml_metal_buffer_map. That one built a buffer with a NULL context instead of faulting immediately; llama-model.cpp already checks the result for nullptr. Assisted-by: Claude Opus 4.8 Co-authored-by: Yi Wen Wong <yiwenwong@Yis-MacBook-Air.local>
wywen
added a commit
that referenced
this pull request
Jul 22, 2026
…#8) * metal: persistent pointer-keyed boundary-event schedule for paged decode (ggml-org#226) Add a persistent, pointer-keyed boundary-event schedule to the Metal backend so an engine can drive async event-fenced KV residency without a per-boundary GPU host drain (the cb_eval ggml_backend_synchronize that dominates paged decode). - ggml_metal_set_boundary_schedule(): register cut nodes (each optionally signalling an event after its command buffer) and wait nodes (each waiting on an event before its segment), keyed by ggml_tensor* POINTER. PERSISTENT: it applies to every graph_compute until cleared (n_cuts==0), so it covers all of ggml_backend_sched's splits within one llama_decode and the reused graph of later tokens; pointer matching is robust to split-vs-full-graph index skew. n_cuts==0 leaves the stock n_cb encode path untouched. - ggml_metal_graph_compute_paged(): sequential per-boundary-committed encode (no dispatch_apply) that, for whichever schedule nodes appear in the graph it is handed, emits encodeSignalEvent after each boundary's command buffer and encodeWaitForEvent before an admitted region's first-reader segment. Reuses ggml_metal_op per segment (fusion preserved). A graph with no schedule nodes encodes as one segment. - Explicit-value event primitives: ggml_metal_event_encode_{signal,wait}_value and host-side ggml_metal_event_host_{signal,wait,value} so an engine reclaim/prefetch thread can coordinate an MTLSharedEvent across threads. - ggml_metal_get_device(): the backend's own device (ggml_metal_device_get() returns a fresh, unrelated device each call, whose events never reach the queue) -- callers must init boundary events on this device. Verified by a standalone probe over a real ggml-metal graph: the segmented encode is bit-identical to the stock encode, every per-boundary signal fires progressively, and an admit wait fences real GPU work until a concurrent host signal from a long-lived thread releases it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PR4CoRiTG8d5YfpjumYJSS * wip(metal): signal boundary events on cmd-buffer completion (ggml-org#226) Adds ggml_metal_event_signal_on_complete: a host reclaim thread waiting the boundary event must see coherent shared-storage KV bytes, so signal from the completion handler rather than mid-stream. Separate WIP commit on top of the seam (542d159) to preserve; squash when formalizing the fork PR. Had zero effect on the paging bit-identity bug (keep-or-drop). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PR4CoRiTG8d5YfpjumYJSS * feat(metal): public boundary-event schedule + event API for paged decode Exposes ggml_backend_metal_set_boundary_schedule plus the ggml_backend_metal_event_* primitives so an engine can drive fusion-preserving, event-fenced KV paging instead of a whole-graph host synchronize at each cut. Claude-Session: https://claude.ai/code/session_01GWUngsbi1CQNi4cBrmNFCG Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update llama.h ti remove comment * metal: trim declaration comments, drop external references Remove the multi-line block comments above the newly-added forward declarations to match the terse style of the surrounding headers, and strip references to external trackers/repositories from the remaining in-body comments. * metal: check for NULL from ggml_metal_buffer_init / _map (#6) ggml_metal_buffer_init returns NULL when the buffer allocation or the residency set fails, but ggml_backend_metal_buffer_type_alloc_buffer passed the result straight to ggml_metal_buffer_is_shared, which dereferences it at ->is_shared (offset 0x10). An allocation failure that ggml already logs and handles became a SIGSEGV. ggml-alloc.c already treats a NULL from alloc_buffer as a recoverable failure, so returning NULL is the expected contract. ggml_backend_metal_device_buffer_mapped had the same unchecked pattern around ggml_metal_buffer_map. That one built a buffer with a NULL context instead of faulting immediately; llama-model.cpp already checks the result for nullptr. Assisted-by: Claude Opus 4.8 Co-authored-by: Yi Wen Wong <yiwenwong@Yis-MacBook-Air.local> * metal: encode window over the paged (boundary-scheduled) compute path While a boundary schedule is set, an optional encode window restricts ggml_metal_graph_compute_paged to the segments intersecting an inclusive [first_node, last_node] range, matched by pointer per graph like the schedule's own cut/wait nodes (a graph containing neither endpoint encodes in full). Optional blit pairs move a boundary activation into the window's first encoded segment (before its ops, after its schedule waits) and out of its last (after its ops) as device-side blits between resident tensors -- no host round trip. Persistent until cleared; cleared state is byte-identical to the pre-window paged path. --------- Co-authored-by: Yi Wen Wong <yiwenwong@Yis-MacBook-Air.local> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ggml_metal_buffer_initreturnsNULL(and logs the reason) when the buffer allocation or the residency set fails, butggml_backend_metal_buffer_type_alloc_bufferpassed the result straight intoggml_metal_buffer_is_shared, which dereferences it at->is_shared. That field sits at offset0x10ofstruct ggml_metal_buffer(all_data0x0,all_size0x8), so a handled allocation failure turned intoEXC_BAD_ACCESS (KERN_INVALID_ADDRESS at 0x10).ggml-alloc.calready treats aNULLfromalloc_bufferas a recoverable failure (ggml_gallocr_reserve_nat :432, and :1132), so returningNULLis the contract the callers expect -- the Metal backend was the outlier.ggml_backend_metal_device_buffer_mappedhad the same unchecked pattern aroundggml_metal_buffer_map. That one did not fault immediately: it built aggml_backend_bufferwith aNULLcontext that would fail later on first use.llama-model.cpp:1538already checks the result fornullptrand throws "unable to allocate ... buffer".Verification
A probe that forces the failure (request 512 GiB against a
max_buffer_sizeof ~9.5 GB), built against this tree before and after the change, same source rev otherwise:ggml_metal_buffer_init: error: failed to allocate buffer, size = 524288.00 MiBalloc_buffer returned NULLBefore, the process dies immediately after ggml's own error log; after, the failure propagates as
NULL.Found while investigating a SIGSEGV in a downstream test suite whose stack was
ggml_backend_metal_buffer_type_shared_alloc_buffer <- ggml_gallocr_reserve_n_impl <- ggml_backend_sched_reserve <- llama_context::graph_reserve. The trigger for that allocation failure is still being investigated separately; this change only stops it being a crash.Assisted-by: Claude Opus 4.8