[Feature request] Hybrid model cache: add --checkpoint-every-nb

Mainline llama.cpp added ggml-org#20087

I suggest adding a UI feature for hybrid models (such as Nemotron-H, Qwen 3.5, Jamba2-Mini) in the Context section to account for this feature (notably, context shifting is not supposed to work with RNN models, so these should likely be mutually exclusive).

Essentially, this creates cache checkpoints after processing every n batches during prompt processing. Why: it is really helpful when using apps such as SillyTavern that can insert an "anchoring" prompt N messages above the user's prompt (this is common practice to make the model adhere to system prompt guidelines). Without this feature, RNN models invalidate the whole cache, and the entire prompt must be reprocessed. With this feature, however, only the n batches of prompt caching are invalidated and must be reprocessed, which speeds up the usage significantly.

Yes we have that, its enabled by default in RNN models now if you select context shift, it will attempt to do this instead,

you can control the cache slots with --smartcache N. it will intelligently set and clear the checkpoints.

No, this is something else. The PR I mentioned was only merged 5 days ago, please check it out: ggml-org#20087

What you are referring to is SmartCache, which (as far as I understand it) creates a checkpoint in RAM after processing the whole prompt. 20087 provides a better/more complete fix for the issue.

Basically, PR ggml-org#20087 adds intermediate prefix checkpoints calculated automatically during prefill, at every n batches, so hybrid/recurrent models are more likely to successfully resume from an earlier saved prefix. While it stores several snapshots at different prefix lengths, which is similar to the option you mentioned, the idea here is more advanced, as these checkpoints are created automatically during the prefill and not after it. So when SillyTavern invalidates e.g. the last 1000 tokens by inserting a system message at level 4, llama.cpp can reuse one of the automatically created checkpoints and recompute only the suffix after that point.

I don't know how to better explain it, but in practice I find that, with ggml-org#20087, llama.cpp will not reprocess the prompt more often than, currently, koboldcpp. Which is why I suggested koboldcpp to also include this feature.

KoboldCpp uses its own caching algorithm, it caches during prefill too. It uses a 3 part cache, one done slightly before the end of prefill, one done after prefill is complete, and one done after generation is complete.

https://github.com/LostRuins/koboldcpp/blob/concedo_experimental/gpttype_adapter.cpp#L4464-L4482

Have you tried it?
It is incorrect to assume that our implement always matches upstream. Rather than spamming snapshots every N tokens, koboldcpp intelligently processes the prompt until X (currently set to 32) tokens before the end of the input. This is usually enough to remove boundary tokens as well as short inputs or trimmed portions of a user's previous input

This. 32 tokens is obviously too little for when SillyTavern inserts anchoring message at depth 4 (default); that's 4 messages higher than the current prompt, which obviously breaks caching with koboldcpp in this scenario. How can we change this value in koboldcpp? Is there a command line option? A reasonable value for me personally would be 2048 or even 4096 tokens.
(I'm talking about things like Character Notes and Author Notes; they are widely used to ground the model into desired character voice, for example. They reinforce system messages, which have diminishing effect as the context grows.)

Hmm... The 3 part caching is triggered every turn. However, by the time 4 turns have passed its likely the checkpoint has been rotated out.

For shorter contexts you can use a higher number of checkpoints e.g. --smartcache 15, however for longer stories this becomes a little too RAM intensive. I will have to figure out a better snapshotting strategy.

I improved the snapshotting behavior, running it with about 8 slots should now be enough for this case in the next release

I'm currently using OpenClaw with KoboldCpp, Qwen 3.5 122B with 128 GB of VRAM on Strix Halo, and SmartCache somehow most of the time doesn't keep KoboldCPP from reprocessing around 50000 tokens around every turn. RAM is maxed out pretty quickly with those smart cache slots, too. Those snapshots are a few hundred megabytes each and the NVMe SSD is capable of around 3 Gbit/s. So having around 100 slots ready to load into and out of the VRAM which shouldn't take more than a second would be very nice solution to make up for the slow prompt processing of the Strix Halo. Could that be done?

Ahh openclaw. Openclaw changes the context contents very frequently. Caching won't really work for it because it's constantly swapping different stuff

With LMStudio the context caching worked really well with openclaw using checkpoints and rarely needed full reprocessing, but then unfortunately it crashed after 100.000-300.000 processed tokens every time which KoboldCPP doesn't.

I'm adjusting the snapshot logic, hopefully the next version should require less reprocessing

@LostRuins

I guess I'm late to the party on this, as I'm only now discovering that my previous stable strategies for dynamic data insertion, author's notes etc, completely break down with RNN/Hybrid models, triggering reprocessing of the entire context. What solution did you go for in the end, @LostRuins ?

Thinking about how I would attempt to resolve it within my own means, I considered adding an async safety margin between stable data and volatile/dynamic fields. Something like:

"...[Turn_5][Turn_6][X]~[Author's_Note][Turn_7]"

If enabled, a user hits "Send" and the model processes the input up to the [X], generates 1 token that gets discarded and triggers the save-state, then resumes from that save-state as if this chunk of data was recently added, carrying on as normal, and at the end of their output, no save-state needs to be triggered. [Edit: In fact, now that I think about it, the model should theoretically prep the snapshot in the background while the user is typing, making the eventual response feel instantaneous, similar to how snapshots are already saved at the end of output. "Pre-compute" until [X] after every output. Well, it's an option.]

For setting the [X] I considered either,

Token slider: Set the number of tokens to count from the most recent ones, to truncate or "cut" the prompt in half, and we determine the exact window for dynamic data injections, and the minimum number of tokens we can expect to be reprocessed on each turn. Works fine if insertion at depth likewise counts tokens, just set it to +1 of whatever that value is.

Regex marker: Pick any obscure symbol that isn't being used for anything, like ⛆, or a specific string, make it inert and invisible to the model so that there's no chance of it being generated, and place it at the very front of whatever dynamic data, author's note etc. That's the stopping point for the first leg of the prompt, the intermediary save-state creation loop, and the resumption point for the second leg. A more flexible and dynamic approach, in case the dynamic injection field needs to grow -- maybe we're doing variable image analysis at the very bottom, who knows? Won't ever trigger full reprocessing, but less control over the expected number of reprocessed tokens per turn.

I think that might be feasible to implement as part of a front-end, though maybe you had a more elegant approach in mind.

I did a very simple approach of snapshotting the context near the end (~30 tokens) of each processing run for every turn. It allows rewinding up to that point which includes changing the latest instruction, and adjusting the boundary tokens near the end of the AI's last input.

@LostRuins

Hi @LostRuins ! Jumping in on this thread. SmartCache handles boundary tokenisation brilliantly, but I want to raise a workload that currently falls through the cracks and propose a complementary mode.

SillyTavern Dynamic Tails -
The background is that, when using ST character cards, the context typically has a massive stable prefix (20k~ tokens of lore/persona/past messages) and a highly dynamic tail. Every turn, the last 500 to 3,000 tokens change constantly due to rotating summaries, dynamic World Info injection, and injected author's notes (e.g., fixed at @depth 4+).

Because SmartCache snapshots roughly at end - 32 and relies on an exact match, a tail that changes further back than 32 tokens causes a total cache miss. We end up doing a full prefill every time. (Aoleg's earlier point about the @depth 4 anchor is exactly the same issue). For pure transformer models they can build a workaround with context shift, but for hybrid models there isn't currently such a good solution afaik.

Could we add an additional periodic-checkpoint mode alongside SmartCache?

The Idea is that, similar to what llama.cpp does (ggml-org#19264, ggml-org#16382, ggml-org#20087), we take periodic prefill checkpoints (e.g., every N tokens) and use longest-common-prefix matching on restore.
For hybrid/recurrent models, these partial snapshots are tiny and cheap, making dense checkpointing practical.
This can be added separately or built into SmartCache. For the latter case, we could maybe add a flag (like --smartcache-interval N) to feed the cache slot pool with these periodic hooks too. Restore just picks whichever matches the longest prefix.

Ollama has a similar method (https://github.com/ollama/ollama/blob/main/kvcache/recurrent.go#L13-L17) that it builds 24 segments * 1664 tokens from the bottom of the prompt specifically for recurrent models. I've tried that and the solution was verified for my case - instead of processing the whole prompt (150 seconds with KCPP+current SmartCache) only the changed tail needs to be processed (12 seconds with Ollama's checkpoints). I had a quick scan of llama.cpp's work on this and if I'm not mistaken it seems that most of the required infrastructure/API is already production ready, which might be good news.

It solves the problem in the event that the dynamic tail falls within the set interval, but if out of alignment, worst case we go back by an extra segment. I suspect that in the majority of (maybe all?) cases, we have at most three logical segments, of a [Static Data] block (world info, instructions etc), [Stable History] (interaction) and the [Dynamic Tail], and breaking them up into further sub-segments seems like extra trouble for no additional benefit. I guess the benefit is "brute force" of setting any random interval and forgetting, and having it "work" even if sub-optimally.

Static & Stable blocks do work as a single segment, until and unless we reset the interaction, which means that there is some benefit to keeping the two apart. I still think a "smart" system would look for delineators in the context, to figure out with perfect precision where to introduce a "cut", though I'm making the assumption here that there's no reason why segments would need to be uniform, correct?

KoboldCpp actually does this, as you play the session and it gets longer you accumulate snapshots. Eventually once the limit is reached, the least use snapshot is dropped and replaced.

So since you say RNN is cheap and can afford more snapshots, simply increase the --smartcache amount, maybe --smartcache 30 or something. That way you approximately have the last 30 turns, and if say you need to rewind to turn 18 for some reason, it will be able to do so.

So long as the "mega memory prefix" is accessed somewhat often it shouldn't ever be evicted. And if you've left it alone for long enough that even 30 turns evicts the base memory, then you weren't referencing it that much anyway and a full reprocess shouldn't be an issue.

Increasing smartcache slots doesn't solve the problem I raised for ST workloads.

In ST, the typical dynamic parts in tail (World Info, Author’s Note, Summary) is injected at the same position every turn, one example is the said @depth=4. The injection happens every turn at a fixed position in the prompt, not a fixed neighbourhood of context. Since smartcache only snapshots at the very end, every saved snapshot is by its design poisoned by stale tail data from previous turns. A byte-match will always fail at where the first injection happened, no matter how many slots you add.

edit: I have started working on my fork on this since I have a strong need.

but if its injected at depth=4, won't it simply use the checkpoint at depth=5?

Anyway yeah feel free to explore, the snapshot function is called smartcache_quick_snapshot(); and its quite easy to use

Thanks for the information! I've got a prototype running and it seems good so far as I tested on my typical use cases (tho not sure if it's robust).

Maybe let me visualise this, assume the context length is not full (no ST trim) -

say arbitrary turn N =

[CharDescription]
[CharMsg1]
[UserMsg1]
[CharMsg2]
[UserMsg2]
[CharMsg3]
[UserMsg3]
[CharMsg4]
[UserMsg4]
[CharMsg5]
[UserMsg5]
[CharMsgPrefill]
<generation head>

This is a clean case where smartcache can work. I summarise it as typical append- and rewind-only workload.

if there's an injectioin at depth=2, turn N will be

[CharDescription]
[CharMsg1]
[UserMsg1]
[CharMsg2]
[UserMsg2]
[CharMsg3]
[UserMsg3]
[CharMsg4]
[UserMsg4]
[@depth=2 Author's Note]
[CharMsg5]
[UserMsg5]
[CharMsgPrefill]
<generation head>

It can't resume from checkpoints like Msg4 because in Msg4's checkpoint it was like

[CharDescription]
[CharMsg1]
[UserMsg1]
[CharMsg2]
[UserMsg2]
[CharMsg3]
[UserMsg3]
[@depth=2 Author's Note]
[CharMsg4]
[UserMsg4]
[CharMsgPrefill]
<generation head>

so does all earlier checkpoints.

I see, that makes sense.

Okay, went back to polish on the dirty prototype, and this time I'm grabbing the return of def generate, and replacing it with:

# --- Specialist Override ---            
        # 1. Capture the original text and return this at the very end to keep the user experience clean.
        final_output_package = {
            "text": outstr,
            "status": ret.status,
            "stopreason": ret.stopreason,
            "prompt_tokens": ret.prompt_tokens,
            "completion_tokens": ret.completion_tokens
        }

        # 2. Define Marker & Logic
        marker = "⛯"
        # We only care about what was SENT, not what was just generated.
        if marker in prompt:
                # We split ONLY the prompt, leaving memory as its own 'clean' block
                stable_prompt = prompt.rsplit(marker, 1)[0] + marker
            
                orig_prompt = inputs.prompt
                orig_memory = inputs.memory # Save the original encoded memory
                orig_max = inputs.max_length
            
                # Point to the stable prompt, but keep original memory
                inputs.prompt = stable_prompt.encode("UTF-8")
                inputs.memory = orig_memory # Use the exact bytes the engine already likes
                inputs.max_length = 1 
            
                # The 'final_output_package' ensures the user never sees the extra token generated here.
                handle.generate(inputs)

                # Restore
                inputs.prompt = orig_prompt
                inputs.memory = orig_memory
                inputs.max_length = orig_max

        return final_output_package

Now the 1-token turn triggers after model output as a kind of post-processing, so the writing passes don't feel like unloading a ton of bricks every time you hit "Send." Combining prompt and memory caused me a lot of headache here, so I moved away from that. Did a few quick tests with Gen, Regen, Continue and ST, and everything seems to work just fine. Cycle looks like:

Input sent, Input posts:
RNN Match Slot N3, Slot switch
Slot 1 create, write_data, read_data
Restored KV, Processing Prompt 1
Slot 4 create, write trash data (presumably)
Slot 23 create (presumably trash regen data, but we never find out), write data
Gen normal. Output starts streaming. On final token:
RNN Match N3 (same match again), Slot switch
Slot 2 create, write_data, read_data
Restored KV, Processing Split Prompt
Slot N3 create (usable prompt-resume data without volatile layer), write_data, read_data
Slot 23 (overwrite) create (usable regen data), write data
Gen 1 token (Always EOS for me)
Output posts:

I've done my best but still not sure it's accurate, a lot of confusing jumps to follow. I assumed that the 23 regen/continue slot was preserved in the first pass, but doesn't seem like it. So, it looks like all slots could be written on the second "split" pass only. Doesn't seem correct, I'm probably being tricked by the log somehow -- looking at it gives me Mandela Effect. Want to say that responses are instantaneous after match, slot switch and processing, and that it's the logs are getting messed up, but no longer remember.

Over on ggml-org#20225 the "dirty fix" is a similar approach to mine, but instead,

adding an empty user message after each tool_call or message

This should work even better if the data in the volatile layer is static, unchanging between turns, like say an interaction border, since it creates a comprehensive snapshot of the future turn for an exact match to the final token even in the outstr. If it's a dynamic injection, where data changes between turns, however, it'll always fail.

An ideal scenario would be one where we take branching path snapshots of future turns, one assuming only the positioning of the volatile layer will shift, while the other assumes the data in the volatile layer will shift, and then on the actual turn we make the relevant match, though it'll probably need to happen in the reverse order. Compute up to border marker, snapshot, resume from marker, compute up to end of last outstr, make relevant match. Again, this sounds like it would benefit from dedicated slots or dedicated slot pool based on function.

@EmanuelOverride

I put my local patch online.

@EmanuelOverride

If you (&anyone, though I doubt) have a hard need like me for such a ST use case maybe try out my fork at https://github.com/Elypha/hybrid-cpp

It implements a checkpoint pool for hybrid models, to help with partial rewind (dynamic World Info injection, Author's Note, etc).

The interval/slots can be configured. For instance,

an interval of 256 allows you to recover from the nearest checkpoint with no more than 256 tokens re-computed (mathematical exp. is half that =128).
a slot number of 64 will cover 64*256=16k context

Like I said for hybrid models checkpoints are cheap - e.g., 150MB each for Qwen3.5 and is O(1). Performance overhead is practically minimal. More in my local benchmark https://github.com/Elypha/hybrid-cpp#benchmark.

(At least) my problem is solved.

@Elypha

@Elypha Cheers, I'll give it a try. It's certainly a more straight-forward strategy for checkpoint implementation that what I'm envisioning, but I'm also quite invested in my own headcanon. I grin.

on gen, if RNN model AND smartcache enabled = get identical / bestslot / in-class-order, from file etc (same basic logic as before, with a few tweaks);
in-class-order, if no RNN match, do initial population/ re-population / pre-processing turn;
enable slot logic class regen & resume,
if field_01 flag & marker_01 set, enable slot logic class static_mem (instructions/world info/static data, read only unless no RNN match, final fallback for full interaction reset),
compute index 0 to marker_01, stop, save slot_01, load_slot01,
if field_02 flag & marker_02 set, enable slot logic class stable_mem (interaction history up to volatile border, i.e author's note, inserts etc, primary pool for n-slot rollback),
compute to marker_02, stop, save slot_02, load slot02,
Copy prompt
//end of pre-processing step
// compute as normal, generate, finish,
//start of post-processing loop
if field_02 = disabled,
save slot_03 to class regen (rolling regen),
if no RNN match for slot_04, find deviation_marker via std::mismatch, compute to deviation_marker, save slot_04, load slot_04,
else
load slot_04, add user input 0, compute, save to class resume pool from n (interaction history with presumed static volatile layer, interaction border etc, secondary conditional n-slot pool), remove user input 0, return,

else,
load slot_02, compute up to marker_02, save to stable_mem class pool from n,
load stable_mem slot, compute as normal, save slot_03 class regen,
load slot_04, add user input 0, compute, save to class resume, remove user input 0, return,

in-class-order matching priority: 04 -> 03 -> 02 -> 01

class logic:
static_mem 1 slot, never updates while matched;
stable_mem 2+n slots, adds updates, overwrites oldest;
regen 1 slot, rolling updates, overwrites oldest/self;
resume slots conditional, if stable_mem=0 then 2+n slots, else 1 slot, if stable_mem=0 then overwrites oldest/self, else adds updates, overwrites oldest;

Plainly speaking, such a system would give us a dedicated sys-instruction/world info slot, static_mem, to resume interactions on a fresh context, and two branching paths options for static volatile layers and dynamic volatile layers, i.e "if author's note doesn't change from turn A->B, resume compute at last token, else resume on border to volatile layer", regen from start of AI turn, continue from end of AI turn, and the option to circumvent stable_mem logic if dynamic volatile layers aren't expected in workload, and n-slot utilization based on those expectations, with the heavy lifting, past initial population, occurring as a post-processing step. I think it's a feasible design, especially for multi-user setups, where you might want to get more functionality out of every slot than just segmentation.

EDIT: There's no avoiding setting a marker at least once, to get the ball rolling without breaking the flow, because we get the regen/continue snapshot as part of normal gen, which would be a shame not to utilize, but then have to roll back to a little before that if we want to ensure a clean resume checkpoint with insertion at any depth.

Navigation Menu

Repository navigation

[Feature request] Hybrid model cache: add --checkpoint-every-nb #2034

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Feature request] Hybrid model cache: add --checkpoint-every-nb #2034

Description

Activity

LostRuins commented on Mar 14, 2026

aoleg commented on Mar 14, 2026

LostRuins commented on Mar 14, 2026

aoleg commented on Mar 14, 2026

LostRuins commented on Mar 14, 2026

LostRuins commented on Mar 15, 2026

wypiki commented on Mar 15, 2026

LostRuins commented on Mar 16, 2026

wypiki commented on Mar 16, 2026

LostRuins commented on Mar 16, 2026

EmanuelOverride commented on Apr 13, 2026

LostRuins commented on Apr 13, 2026

Elypha commented on Apr 13, 2026

EmanuelOverride commented on Apr 14, 2026

LostRuins commented on Apr 14, 2026

Elypha commented on Apr 14, 2026

LostRuins commented on Apr 14, 2026

Elypha commented on Apr 14, 2026

LostRuins commented on Apr 14, 2026

EmanuelOverride commented on Apr 14, 2026

EmanuelOverride commented on Apr 15, 2026

EmanuelOverride commented on Apr 16, 2026

Elypha commented on Apr 16, 2026

EmanuelOverride commented on Apr 17, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions