Skip to content

server: enhance FIFO prompt cache eviction with second-chance algorithm - #23666

Open
nonml wants to merge 1 commit into
ggml-org:masterfrom
nonml:clock-eviction
Open

server: enhance FIFO prompt cache eviction with second-chance algorithm#23666
nonml wants to merge 1 commit into
ggml-org:masterfrom
nonml:clock-eviction

Conversation

@nonml

@nonml nonml commented May 25, 2026

Copy link
Copy Markdown
  • Pure FIFO eviction always removes the oldest cache entry
  • A small side session request evicts a large session's cached KV state immediately, forcing full reprocessing and preventing users on small machines from switching sessions back and forth.

What the fix does:

  • Assign score = 1 to new cache entries
  • Only on a cache hit (best match) score += 1 (max score=4) -> 4 is the sweet spot
  • During checkpoint limit reached, instead of always evicting first out,
    • If front score <= 1, evict it
    • Otherwise, score -= 1 rotate it to the back (second chance), repeat
  • If all entries have max score, cap max iter at states.size() * 5 and falls back to evicting the front to prevents infinite loops

With this algorithm, I can switch between sessions without sacrificing compute.

Additional information

related issue (#23030, #20510)

Requirements

@nonml
nonml requested a review from a team as a code owner May 25, 2026 13:18
@jacekpoplawski

jacekpoplawski commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

I initially tried this #22826

@nonml
nonml force-pushed the clock-eviction branch 2 times, most recently from e207139 to 2d4c4ca Compare June 15, 2026 21:04
@nonml
server: enhance FIFO prompt cache eviction with second-chance algorithm
21a8f12 
Switches to a second-chance policy with basic hit tracking and decay to prevent one-off requests from evicting heavily used system prompts. Uses a simple score-decay approach to track hits without adding O(N) scan overhead.
@ggerganov

ggerganov commented Jun 18, 2026

Copy link
Copy Markdown
Member

I initially tried this #22826

This PR has a scoring logic that is not in #22826.

Slopych pushed a commit to Slopych/qllama-sm70 that referenced this pull request Sep 18, 2026
Slopych Slopych
server: replace FIFO prompt cache eviction with second-chance (#23666)
7920ee5 
FIFO eviction drops the oldest entry as soon as the cache exceeds its
size/token limit, even if that entry is still actively used while a
younger, larger entry is hot. This punishes the alternating
multi-conversation pattern: the long-lived conversation is the first to
be evicted.

Add a second-chance score to each cached prompt: every cache hit bumps
the score (capped at 4), eviction decays the front entry's score and
rotates it to the back, only removing it once the score has decayed to
1 (with a hard iteration cap so an all-maxed cache still makes progress).
Both the size and token limits now use the same evict_one() path.

Note: adapted to this fork's server_prompt_cache_state layout, where the
prompt lives in state.prompt (upstream keeps states as list<server_prompt>),
so the score is carried on the nested server_prompt.

backport of ggml-org/llama.cpp#23666

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

examples server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nonml @jacekpoplawski @ggerganov