[code-martin]

Summary

Refines MoE cache telemetry to accurately report token generation hit rate (excluding prompt prefill skew), adds multi-turn prefix/system prompt KV cache retention, and provides real-time cache metrics in the web UI.

Key Changes

• Decode-Only Hit Rate Baseline:
  • Captures baseline cache lookups and hits at the exact boundary where prompt processing ends and autoregressive decode begins.
  • Excludes static prompt evaluation from token generation hit rate statistics, reflecting true decode performance.
• Prefix & System Prompt Checkpoint Retention:
  • Creates prompt checkpoints at every turn boundary (<|im_start|>) rather than solely at the very last one.
  • Preserves the root checkpoint (checks[0], holding system prompt and initial instructions) during FIFO cache eviction, allowing branching conversations and parallel subagents to immediately reuse the shared prefix.
• Engine Protocol Enhancements:
  • Extends the DONE line to output
    eq_hits,
    eq_look, otal_swaps, and drift.
  • Adds the STATS command for queryable cumulative telemetry.
• Web UI & REST API:
  • Exposes
    ecent_hit_pct,
    ecent_hits,
    ecent_lookups, and cumulative counters in /metrics and /api/stats.
  • Adds an Expert Cache Hit Rate progress bar and swap counter to the Monitor tab.
  • Adds a dedicated Hit rate column to the Recent requests table in the web dashboard.

Testing

• Verified live on web dashboard: displays accurate per-request decode hit rates (e.g. 51.2%, 45.6%) and aggregate moving window percentages.
• Verified prompt cache hit behavior with multi-turn chat: turn prefixes and system prompt are retained and reused across turns.

[Niko1221]

Thanks for the work and the write-ups. I went through all three carefully, and I'm not merging them as they are - here is why, so the useful parts can come back in a shape that fits:

#37 (VMM cache + on-demand vision):

• The cache can only shrink and grow back to its startup size, and the slot gain in the description comes from loading vision after the engine rather than from VMM.
• RESIZE/GROW copy the cache's startup residency table over the live one (xcache.residency_table()), while adaptation (on by default) has already swapped slots - after a resize those slots are read as the wrong expert.
• Since 0.1.13 the prompt path borrows the top of the cache for its buffers; shrinking unmaps exactly that region, and a partial GROW leaves those pointers on unmapped memory.
• The 5.4 GB / 10 GB targets are hard-coded, and every image pays a full encoder load (~7.5 s).

#38 (adaptive eviction): it mostly retunes the adaptation the engine already has (0.7 was already exponential decay), the new ADAPT command blocks after every request, and it reads usage even when adaptation is off (--adapt-swaps 0), which crashes. A per-swap stderr line floods the log.

#39 (telemetry): it changes adapt_every 4 -> 1 and adapt_swaps 96 -> 1 without saying so, checkpoints at every turn boundary (for a 40-turn agent prompt that is ~40 borrow/refill cycles and ~40 x 118 MB saves, which undoes much of 0.1.13's prompt speedup), and /api/stats skips the API-key check.

What I'd gladly take as small separate PRs against current main: the per-request decode hit rate (the DONE fields, the Monitor column), keeping the root checkpoint (only the first turn boundary), the Vision.close() temp-dir cleanup, and flags for the adaptation thresholds - each with a before/after measurement. Also please drop tools/test_vision_client.py and the CMakeLists BOM change from them.