[eapache]

Overview

Turn the simple LRU cache for MoEs into a 50/50 split LRU cache. Think similar to generational garbage collection, where there is an older stable set and a younger more volatile set. This performs about 1-5% better than simple LRU in my various tests.

Additional information

I had Astra manage a fairly extensive set of experiments across two GPUs (a 3060 on a slow PCI bus and a 4500 on a fast one), two models (Gemma4-26B-A4B and Qwen3.8-Flash-Next) and a wide variety of proposed mechanisms (LFU, leaving rare experts to execute on CPU, and many others and hybrid tunings). This approach gave the best improvement without any regressions, and was also conveniently one of the simplest.

Depending on the constraints of the system there are some approaches that can squeeze out another 5% or so, but those approaches tend to perform worse than plain LRU on other versions of the setup.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, Astra ran the experiments and after I reviewed the findings, Astra wrote the (relatively simple) implementation.

[eapache]

@am17an since you did the original LRU version

[am17an]

I'll run some tests

[am17an]

I see a small but consistent gain with this PR

[kudraw]

I seem to have got a regression on GLM 5.3 Flash, from about 15 tg/s to about 12.5.

This is my configuration:

readonly MODEL=/models/GLM-5.3-Flash/IQ3_XS/GLM-5.3-Flash-BF16-IQ3_XS-00001-of-00004.gguf
readonly PARAMETERS=(
--alias GLM-5.3-Flash
--fit off
--cache-ram 0
--parallel 1
--ctx-size $((1*65536))
--load-mode none
--n-cpu-moe 40
--moe-cache-mib 4500
--temp 1.0
--top-p 0.95
--jinja
)

Running on Arch Linux with latest updates, on the following hardware:

0.00.000.799 I srv llama_server: initializing ...
0.00.169.559 I ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32218 MiB):
0.00.169.566 I Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32218 MiB
0.00.170.193 I cmn common_param: common_params_print_info: build 11741 (1b29621) with GNU 16.2.1 for Linux x86_64
0.00.170.195 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the -lv N CLI arg)
0.00.170.195 I cmn common_param: device_info:
0.00.242.422 I cmn common_param: - CUDA0 : NVIDIA RTX PRO 4500 Blackwell (32218 MiB, 31941 MiB free)
0.00.242.430 I cmn common_param: - CPU : AMD Ryzen 9 9950X3D 16-Core Processor (125907 MiB, 125907 MiB free)
0.00.242.493 I cmn common_param: system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | FA_QUANTS = q4_0-q4_0,q8_0-q8_0,f16-f16,bf16-bf16 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

[eapache]

I seem to have got a regression on GLM 5.3 Flash, from about 15 tg/s to about 12.5.

Did you narrow it down to this PR specifically, or was it a regression just after pulling the latest master? There have been a lot of changes recently.

Eyeballing the config, I think it's plausible that this PR would cause a regression in your use case because your cache (4500MiB) is so tiny relative to the size of the model. In particular (thanks to Astra for doing some of this math):

• GLM-5.3-Flash at IQ3_XS takes about 10MiB per complete expert depending on the exact quant recipe
• a 4500MiB cache thus can store roughly 450 experts
• each token in GLM-5.3-Flash (with the --n-cpu-moe 40 in your config) accesses about 300 experts from system RAM

So prior to this PR, an entire token's worth of experts fit in the cache, with room to spare for some prior experts. This PR splits the cache space in half, so you end up with an old set and a new set but neither set has room for a single token's worth of experts, and I can easily imagine that causing a bunch of bad cache churn behaviour.

Ultimately this is a very theoretical explanation since I don't have your system or full config, but... it's a definite maybe. If you have ~3GiB of system ram to spare I would suggest offloading one more layer of experts and using the space savings for a much bigger cache (i.e. just --n-cpu-moe 41 --moe-cache-mib 7000 or something like that). But plausibly this PR did increase the "minimum useful cache size" for some configurations. I hadn't thought of that case before.

[eapache]

If my analysis is correct, then it is up to the maintainers whether this case is sufficiently important to warrant a revert or not. One alternative would be to print a warning when the configured cache is small enough to cause significant churn, i.e. something like (AI-written PoC, I haven't actually tried this):

diff --git a/src/llama-moe-cache.cpp b/src/llama-moe-cache.cpp
index 4d3596b24..efe41194c 100644
--- a/src/llama-moe-cache.cpp
+++ b/src/llama-moe-cache.cpp
@@ -357,6 +357,13 @@ struct llama_moe_cache::impl {
                 g.n_slots = 0;
                 continue;
             }
+            int64_t n_expert_token = 0;
+            for (int32_t il : g.layers) {
+                n_expert_token += model.hparams.n_expert_used(il);
+            }
+            if (g.n_slots < max_slots && g.n_slots < 2*n_expert_token) {
+                LLAMA_LOG_WARN("%s: MoE cache for %zu layers on %s holds less than twice the experts used by one token; this may cause excessive cache churn\n", __func__, g.layers.size(), ggml_backend_name(d.backend));
+            }
             g.slru.init(model.layers.size(), n_expert, g.n_slots);
             n_tensors[g.id] += g.ref.size()*(1 + g.layers.size());
             n_tensors_host  += g.layers.size();

Though even in this case, having a small cache may be better than no cache at all, it's hard to say.

Edit: Or even falling back to a non-split LRU cache for small cache sizes might not be too hard. Though we may still want a warning for caches that are particularly tiny.

[eapache]

Terminology: "1x" means the cache is large enough to hold exactly one complete token's worth of experts.

Via Astra I ran a variety of further experiments. Some condensed findings:

• Below 1x (i.e. the cache cannot hold even a single token's experts), it thrashes and decode speed is limited more-or-less by PCI and RAM bandwidth since effectively every expert has to be streamed from RAM for every calculation. There is a sharp performance improvement as soon as the cache is slightly larger than 1x.
• While the cache is >1x but still "small", plain LRU substantially beats SLRU, presumably because there is too much churn and the multi-generation setup frequently preserves the wrong experts, etc.
• The crossover point where SLRU starts to dominate LRU is heavily dependent on the amount of expert reuse, which in practice means it is dependent on both the model and the workload. I have seen it anywhere from ~1.4x to ~1.8x, which is lower than the naive 2x threshold in my previous comment.

These results very roughly match/explain the regression seen by @kudraw, though the reported regression is still on the larger side of what I'd expect.

---

Implications / proposals:

• There is a capacity guard in the code already which confused me for a while, but it is limited to a single layer's experts for a single token - below which the model can't even execute correctly because not enough of the weights can be cached at a time. I would propose increasing this guard to fire for cache sizes below 1x (a much higher threshhold), since those sizes cannot realistically provide any benefit over simply executing the experts on CPU. Or at least adding a strong warning that your config is suboptimal.
• It would be relatively straightforward for the implementation in this PR to fall back to plain LRU, either when requested by a flag, or based on the cache size expert ratio. I am torn on how to do this, given that it's such a rare case, but the precise cross-over point is not super-well-defined for a given workload. An automatic 1.8x threshold would probably be good enough, at the risk of leaving a few % of performance on the table for certain rare combinations of cache size, model, and workload.

[am17an]

Perhaps there is an adjustment needed in the --moe-cache-mib, currently we don't guide anything for it but in my tests anything below a 75% cache hit rate is not helpful on a slow PCIe4 box (PCIe5 can probably take a lesser hit rate). @eapache since you are running experiments it would be helpful to the community to create a discussion post with benchmarks. There is clearly not one right answer and it involves a lot of tuning, perhaps a script to find the optimal setting may be useful too.

[eapache]

@kudraw #30848 should fix the regression in your config by falling back to LRU for small caches. Please give it a try. I have left it in draft because it is AI written and I haven't given it a thorough look yet, but it is only +14/-10.

@am17an unfortunately I only have the one weird box to do my tests on. It is asymmetric multi-GPU which has let me try a few different options, but it is unusual (my RAM is only about as fast as my PCI bus) and without more combinations of speeds I am hesitant to draw too many broad recommendations, thus why I was fairly conservative in proposing this PR (and I still managed to miss a case and cause a regression). I have thought a bit about how we would choose --moe-cache-mib automatically using --fit, but the number of additional parameters and combinations seems like too much complexity to be able to handle gracefully and get it even approximately right in all cases.

I'm at the end of my day here (EST), and I will probably be busy with my day job for a bit. When I get a chance to come back to this, I will see what I can cook up.

P.S. You are welcome to take #30848 out of draft and ship it if you like the shape.

[kudraw]

Hi @eapache , thank you for your patience and your work.

I tested your patch (nothing formal, I use a couple of fixed prompts via llama-cli to test generation speed), here the results:

configuration	b11741	pr-30848
39 moe / 0 MiB	14.6 t/s	14.65 t/s
40 moe / 4500 MiB	13.35 t/s (hit rate 4.95%)	15.55 t/s (hit rate 30.60%)
41 moe / 7000 MiB	17.10 t/s (hit rate 44.06%)	17.30 t/s (hit rate 45.27%)

It seems that the patch restored the original performance :D, I see the hit rate was very low on b11741 for this particular moe cache configuration.

Just to clarify my hardware, I'm on Ryzen 9 9950x3d 16 cores, 128 GiB RAM DDR5 6400 CL40 and GPU NVIDIA RTX PRO 4500 Blackwell Workstation Ed. 32 GB. Motherboard is an ASUS ProArt B850-CREATOR WIFI NEO.

@am17an about the bus, I'm on PCI-e 5.0, your patches were great for Qwen 3.8 Flash Next (I got Q8 from about 22 t/s to above 30 in the latest builds).

At work I've a server with an RTX A4500 20GB on PCI-e 3.0 and effectively the cache doesn't help much.

Anyway, tell me if I can help with more tests (also on other models) or make more rigorous ones. Thank you!

PS: I'm on CEST +0200, so I can do something only when I go back from work, in the evening.

[eapache]

Thanks! I had a clever idea and pushed another commit to #30848 which I think will work even better for these middle-small caches and have taken it out of draft. Please give it a try!

[kudraw]

Hi @eapache , I've tested again with the newer commit, here the results (I've added a column to the previous table):

configuration	b11741	pr-30848	pr-30848-da05b4628
39 moe / 0 MiB	14.6 t/s	14.65 t/s	not tested
40 moe / 4500 MiB	13.35 t/s (hit rate 4.95%)	15.55 t/s (hit rate 30.60%)	16.15 t/s (hit rate 34.06%)
41 moe / 7000 MiB	17.10 t/s (hit rate 44.06%)	17.30 t/s (hit rate 45.27%)	17.45 t/s (hit rate 44.82%)

Not bad :).