[glennneuber]

Stacked on #180 — review that first; this branch targets it, not main.

The finding

MLX's allocator keeps freed blocks for reuse, and its default cache limit is derived from total device memory — 90.22 GiB on this card — the same wrong denominator the allocator ceiling had. Measured on gemma4:31b-nvfp4, one 3072×1728 image, CUDA:

memory.active 18.29 GiB     live tensors, ~= the weights
memory.cache  13.16 GiB     retained
nvidia-smi    33,134 MiB

llama.cpp does the identical work — same image, same num_ctx, same prompt_eval_count 1118 — in 23,586 MiB.

active + cache accounts for 31.45 of the 32.36 GiB observed, so the excess is MLX's cache and not CUDA's async memory pool, which was my first assumption and would have needed cudaMemPoolTrimTo — out of our reach.

The lever was never bound

mlx_set_cache_limit ships in MLX and had no Go binding. SetMemoryLimit did, and the two are easy to confuse:

	caps	effect
SetMemoryLimit	total allocation	decides when an allocation fails; releases nothing
SetCacheLimit	retained buffers	decides how much MLX keeps — governs steady-state footprint

Measured

One 3072×1728 image, same binary:

cache limit	peak footprint	MLX cache
unset (MLX default, 90.22 GiB)	33,134 MiB	13.16 GiB
4 GiB explicit	28,526 MiB	4.02 GiB
default (4 GiB, env unset)	28,718 / 28,748 MiB	—

The default reproduces the explicit setting across two separate runs.

Why 4 GiB and not 0

Peak 25.37 GiB against active 18.29 GiB puts the transient working set near 7 GiB, so 4 GiB covers the bulk of the reuse while bounding retention. Returning every buffer costs allocator churn on a path that runs per forward — a footprint win that doesn't pay for an unmeasured latency risk. OLLAMA_MLX_CACHE_LIMIT overrides, including 0.

A parser bug this surfaced

parseCacheLimit exists because parseMemoryBudget rejects zero — correctly, since a zero memory budget forbids every allocation — while 0 is the most useful cache limit there is.

Reusing it silently discarded OLLAMA_MLX_CACHE_LIMIT=0 and left the backend default in place, so my measurement of that setting reported the default's footprint under the cap-0 label: 33,134 MiB, byte-identical to the unset arm. That identity is what gave it away, and it's now pinned by a test.

What this does NOT fix

Cache fell 13.16 → 4.02 GiB (9.1 GiB) but the process fell only 4.6 GB — roughly half the freed cache is not returned to the driver. That residue is the cudaMallocAsync pool. Secondary, not zero, out of scope here.

MLX at a bounded cache is still ~4.9 GB above llama.cpp for the same work.

Tests

TestParseCacheLimitAdmitsZero            0 is accepted and means retain nothing
TestParseCacheLimitRejectsAbsentAndJunk  "" / abc / -1 / overflow rejected
TestZeroIsStillInvalidAsAMemoryBudget    the two parsers differ ONLY on zero
TestDefaultCacheLimitIsBounded           unset applies a bounded default, not MLX's

go test ./x/mlxrunner -run 'CacheLimit|MemoryBudget|BudgetOverride'   ok
go vet ./x/mlxrunner/...                                             clean

Applied after the weights are resident, so the logged previous and any trim reflect a loaded model rather than an empty allocator.

🤖 Generated with Claude Code

[glennneuber]

Reviewing as consolidator. This is the right lever and the evidence is the right evidence — SetMemoryLimit vs SetCacheLimit is exactly the confusion worth documenting in that table, stacking on #180 rather than duplicating it is the correct call, and the parseCacheLimit split is a real catch: parseMemoryBudget rejecting zero is correct for a memory budget and wrong for a cache one, where 0 is the most meaningful value.

The numbers match what I measured independently on the same image (active 18.29 / cache 13.16 / nvidia-smi 33,134, GGUF 23,586 at prompt_eval 1118), so the finding is reproduced rather than merely reported.

One claim the arithmetic doesn't support

The PR says the excess is MLX's cache and not CUDA's async pool. Your own before/after numbers say it is partly both:

cache released inside MLX :  13.16 -> 4.02 GiB  =  9.14 GiB
process footprint dropped :  33,134 -> 28,526 MiB =  4.50 GiB
unaccounted               :                          4.64 GiB

MLX gave back 9.14 GiB and the process only shrank by 4.50. The missing 4.64 GiB was freed by MLX and not returned to the driver — which is precisely cudaMallocAsync's pool retaining it. The active + cache ≈ nvidia-smi identity holds at a single point in time; it doesn't establish where memory goes after a free, and that is the question the pool hypothesis was about.

So I'd soften that section rather than drop it: the cache is the dominant term and the one we can control from Go, which is enough to justify the change. But the pool still holds ~4.6 GiB, and it's the likely reason the capped footprint is 28,526 MiB against llama.cpp's 23,586 — still 4.82 GiB higher for identical work, which the PR doesn't remark on.

Worth stating explicitly in the PR, because "the excess is cache, not the pool" is the kind of line that stops the next person looking.

On the 4 GiB default

The reasoning — peak 25.37 minus active 18.29 ≈ 7 GiB transient, so 4 GiB covers the bulk — is sound as far as it goes, but memory.peak is a high-water mark over the whole run, not the per-forward transient, so 7 GiB is an upper bound on what reuse needs rather than an estimate of it. That argues for 4 GiB, not against; I'd just not present it as tightly derived.

Two things I'd want before this is treated as tuned rather than reasonable:

• A latency check at the default. The footprint case is made, but "returning every buffer costs allocator churn" is asserted, not measured, and 0 is one env var away. A single scene_single n=3 at OLLAMA_MLX_CACHE_LIMIT unset / 4 GiB / 0 would settle whether the churn is real and price the tail.
• The 3072×1728 geometry is the OOM case. qwen3.6:35b-a3b-nvfp4 aborts there with cudaMallocAsync ... out of memory while 85 GiB reads free. If a bounded cache fixes that, it is a much stronger argument for this PR than footprint alone — and if it doesn't, we've learned the OOM is the pool, not the cache.

Neither blocks merge. The change is correct, bounded, env-overridable and defaults conservatively.

[glennneuber]

The throughput arm contradicts the default — do not merge yet

I said in the PR body that a default costing tok/s should be reconsidered before merge. It appears to.

Same binary, same image, same num_ctx, identical work (prompt_eval 1121, eval 120 in both):

default   (4 GiB)    PEAK 28,748 MiB    gen_tps 32.73
unbounded (90 GiB)   PEAK 33,320 MiB    gen_tps 35.57

−4.6 GB footprint, −8% decode.

Two reasons not to act on that yet, and one reason not to ignore it:

• n=1 per arm. An 8% gap is close to the per-rep spread measured earlier on this cell (28.27–30.16 tok/s on scene_single, ~7%), so it may be noise.
• The cache is doing its job. Buffer reuse is exactly what avoids cudaMalloc/cudaFree round-trips; a cost here is the mechanism working as designed, not a surprise.
• But 4 GiB was chosen from a footprint argument alone, and the transient working set is ~7 GiB. If the cost is real, 4 GiB is simply too tight and the right default is nearer 8 GiB — enough to hold the working set, still an order of magnitude below MLX's 90 GiB.

Running default-vs-unbounded at n=3, alternating so neither arm is systematically first. I'll post the result. If the gap holds, I'll either raise the default to cover the working set or drop the default entirely and leave the knob opt-in — the binding and the parser fix stand on their own either way.

[glennneuber]

n=3: the gap is real, and 4 GiB is the wrong default

Ranges do not overlap. default's best run is below unbounded's worst.

default   (4 GiB)    gen_tps [31.02, 27.19, 30.10]  mean 29.44   peak 28,749 MiB
unbounded (90 GiB)   gen_tps [36.30, 32.96, 35.67]  mean 34.98   peak 33,276 MiB

−15.8% decode for −13.6% footprint. Identical work in all six runs (prompt_eval 1121, eval 120).

My earlier "8% may be noise" was too generous: at n=1 it looked marginal, at n=3 it is separated cleanly.

Why 4 GiB was the wrong pick, in hindsight from the same numbers I already had: peak 25.37 GiB against active 18.29 GiB puts the transient working set near 7 GiB. A 4 GiB cache cannot hold it, so the allocator round-trips to the driver inside every forward. I quoted that 7 GiB figure in the commit message as justification for 4 GiB — it was an argument for something larger.

Sweeping 8 and 16 GiB at n=3 to find the smallest cache that keeps full throughput. If one exists well below MLX's 90 GiB, that is the right default. If throughput only recovers at a value close to unbounded, then there should be no default and the knob stays opt-in for operators who need the footprint on a shared card.

Either way the binding and parseCacheLimit stand — only DefaultCacheLimit is in question. Holding this PR until that lands.

[glennneuber]

Resolved: no default. The trade has no middle.

Swept 8 and 16 GiB at n=3 to find the smallest cache that keeps full throughput:

cache limit	gen_tps (n=3)	mean	peak footprint
4 GiB	31.02, 27.19, 30.10	29.44	28,749 MiB
8 GiB	33.89, 34.46, 35.83	34.73	32,779 MiB
16 GiB	36.50, 33.87, 33.93	34.77	33,295 MiB
90 GiB (MLX default)	36.30, 32.96, 35.67	34.98	33,276 MiB

Throughput recovers fully by 8 GiB — but 8 GiB saves 497 MiB. The whole 4.5 GB saving is at 4 GiB, which costs 15.8% decode.

So neither candidate is worth shipping: at 8 GiB a default buys nothing while adding a surprise; at 4 GiB it silently costs every user a sixth of their throughput to save memory most are not short of. DefaultCacheLimit is withdrawn and the knob is opt-in, for the cases that genuinely want it — a shared card, or a model that OOMs at large geometries (the open qwen3.6:35b-a3b-nvfp4 abort at 3072×1728).

The cause was visible before I picked 4 GiB: peak 25.37 GiB against active 18.29 GiB puts the transient working set near 7 GiB, so a 4 GiB cache cannot hold it and the allocator round-trips to the driver inside every forward. I quoted that 7 GiB figure in the commit message as justification for 4 GiB. It was an argument for something larger.

The measured curve now sits in the code beside the knob, so the next person choosing a value starts from data rather than from the same footprint-only argument.

Unchanged and still the substance of this PR: mlx_set_cache_limit was never bound, and parseMemoryBudget silently ate OLLAMA_MLX_CACHE_LIMIT=0. Ready for review.

[glennneuber]

The full-suite sweep reached 67 GB, and the penalty is concentrated

Every number in the PR's cache table is one 3072×1728 image. Driving the allocator from the other knob — OLLAMA_MLX_MEMORY_LIMIT, full 12-test suite, n=2 — shows where the cost actually lands:

cell	24 GiB	82 GiB	delta
scene_single	30.46 t/s	30.87 t/s	+1%
multi_3img	28.46 t/s	29.19 t/s	+3%
bbox_contract_multi	17.71 t/s	27.21 t/s	+54%

Peak footprint 33,536 MiB against 67,518 MiB — the 67 GB regime finally reproduced. Stable across both reps on each side (17.73/17.71 vs 25.0/27.21), so the 54% is not noise.

This strengthens the no-default decision. A bounded cache would hit hardest precisely on multi-image prompts, which is the shape of real vision traffic and the reason the suite has bbox_contract_multi at all.

The methodological note matters more than the number. Three separate sweeps over the allocator ceiling came back flat today and were each reported as "no throughput effect". All three ran scene_single — a single-image cell that cannot reach the constraint. Correctly scoped every time, and the scope was the defect. Any future sweep over anything allocator-related needs a multi-image cell or it is measuring nothing.

Recorded in the code beside the knob.