[mirror #259] Show the MoE cache hit rate in perf output and llama-server /metrics by LeoBorcherding · Pull Request #13 · LeoBorcherding/llama.cpp

LeoBorcherding · 2026-11-02T22:00:48Z

Mirror of unslothai#259 at bac1669e8 on its merge base, for Codex review. Do not merge.

- add MoE cache hits, misses and uploaded bytes to llama_perf_context_data, reset by llama_perf_context_reset - print the hit rate in llama_perf_context_print and common_perf_print - server: add llamacpp:moe_cache_* counters to /metrics and moe_cache_hits/misses to the request timings - cli: show the hit rate next to the prompt and generation speed

LeoBorcherding · 2026-11-02T22:00:49Z

@codex review

chatgpt-codex-connector · 2026-11-02T22:01:03Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T07:16:23.435493Z	`66ba7fd`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-11-02T22:01:04Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Setup repeatedly failed. Update the setup script or commit, then try again later.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

LeoBorcherding · 2026-11-02T22:04:06Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bac1669e8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-02T22:06:28Z

+            res.n_moe_cache_hits   = perf.n_moe_cache_hits   - perf_start.n_moe_cache_hits;
+            res.n_moe_cache_misses = perf.n_moe_cache_misses - perf_start.n_moe_cache_misses;


Keep MoE timing counts request-local

When multiple server slots process requests concurrently, these deltas include cache hits and misses from every slot since this slot started. Each response can therefore report the same shared cache activity as if it belonged to that request, making its timing data and the CLI's displayed hit rate misleading. Keep these counts at the context-wide /metrics level or avoid presenting them as per-request timings.

Useful? React with 👍 / 👎.

…red a decode The cache counters belong to the context, so with concurrent slots each request's delta included the other slots' lookups. /metrics keeps the totals.

LeoBorcherding · 2026-11-02T23:44:32Z

Pushed 049eee313. The per-request moe_cache_hits / moe_cache_misses were deltas of context-wide counters, so with concurrent slots each request reported the other slots' lookups too (on a 2-slot server, two concurrent requests both reported 538 hits / 20470 misses, the shared total). A slot now records when another slot was processing during one of its decodes and leaves the counts out of its timings in that case; a request that ran alone still reports them, and /metrics keeps the totals. Checked on Apple M5 (Metal, granite-3.1-1b-a400m, --cpu-moe -nr --moe-cache-mib 200 -np 2): solo request 5143 / 7525 before and after, concurrent requests 538 / 20470 each before, omitted after.

@codex review

chatgpt-codex-connector · 2026-11-02T23:44:44Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Setup repeatedly failed. Update the setup script or commit, then try again later.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

LeoBorcherding · 2026-11-02T23:54:51Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 049eee3134

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-02T23:57:25Z

                    res->n_processing_slots  = n_processing_slots;
                    res->n_tasks_deferred    = queue_tasks.queue_tasks_deferred_size();
                    res->metrics             = metrics;
+                    res->perf                = llama_perf_context(ctx_tgt);


Avoid reading performance counters during decode

When /metrics is served during a decode, server_queue::yield_to_queue() runs task handling on its worker thread while the caller runs llama_process() on the main thread. This snapshot reaches llama_moe_cache::get_stats() and reads the plain stats_small/stats_large counters while the cache callback updates them, creating a C++ data race that can also return inconsistent totals. Take the snapshot after decode synchronization or protect the counters.

Useful? React with 👍 / 👎.

…each decode /metrics can run on the queue worker during llama_process(), which raced with the cache updating its counters.

LeoBorcherding · 2026-11-03T07:13:29Z

Pushed 66ba7fd2c. /metrics read the MoE cache counters through llama_perf_context() on the queue worker, which runs during llama_process() on the main thread, so it raced with the cache updating them. The main thread now snapshots the perf data after each decode returns and /metrics serves that snapshot; the yield's mutex handoff orders the write before the worker's next read. Checked on Apple M5 (Metal, granite-3.1-1b-a400m, --cpu-moe -nr --moe-cache-mib 200 --metrics): totals advance while a 256-token request decodes (5805 then 8338 hits), and after a following solo request the total grows by exactly that request's 112 hits / 272 misses.

@codex review

chatgpt-codex-connector · 2026-11-03T07:16:22Z

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 66ba7fd2c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector Bot reviewed Nov 2, 2026

View reviewed changes

server : leave out per-request MoE cache counts when another slot sha…

049eee3

…red a decode The cache counters belong to the context, so with concurrent slots each request's delta included the other slots' lookups. /metrics keeps the totals.

chatgpt-codex-connector Bot reviewed Nov 2, 2026

View reviewed changes

server : serve /metrics MoE cache totals from a snapshot taken after …

66ba7fd

…each decode /metrics can run on the queue worker during llama_process(), which raced with the cache updating its counters.

LeoBorcherding mentioned this pull request Nov 3, 2026

[stage CI] 259 at 66ba7fd #23

Closed

		res.n_moe_cache_hits = perf.n_moe_cache_hits - perf_start.n_moe_cache_hits;
		res.n_moe_cache_misses = perf.n_moe_cache_misses - perf_start.n_moe_cache_misses;

Conversation

LeoBorcherding commented Nov 2, 2026

Uh oh!

LeoBorcherding commented Nov 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Nov 2, 2026

Uh oh!

LeoBorcherding commented Nov 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

LeoBorcherding commented Nov 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 2, 2026

Uh oh!

LeoBorcherding commented Nov 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

LeoBorcherding commented Nov 3, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chatgpt-codex-connector Bot commented Nov 2, 2026 •

edited

Loading