[Reithan]

This adds a memoized data cache to llama_grammar_reject_candidates_for_stack

1. The cache first checks the stack as this is reasonable small check since stacks are usually between 20-60 bytes.
2. If the stack is a hit, it checks the current candidate list size.

• Candidate list sizes vary greatly. 50% are ~600b or below, 75% below 1280b, but they max out around 72k
• Most 'problematic' recursion happens with lists of ~24b.
  a. If the candidate list is <1280b, it's also checked for cache hit to return early
  b. If this wasn't a hit - the result of the function is cached

This change completely eliminates 'hangs' from branching explosions in complex grammar in all my tests so far, as well as speeding up 'heavy' grammar processing to run at nearly the same speed as no grammar at all.

[Reithan]

Example of grammar on/off with this change. This is with a fairly large and complex grammar block.

On

Off

[LostRuins]

I think for a 2% (35 vs 36) t/s change, this is probably not worth it for the additional complexity.

The original speedup in the previous PR was very good, but this one seems marginal at best and probably makes debugging harder.

[Reithan]

I think for a 2% (35 vs 36) t/s change, this is probably not worth it for the additional complexity.

The original speedup in the previous PR was very good, but this one seems marginal at best and probably makes debugging harder.

Sorry, you misunderstand, the picture is comparing no grammar, vs grammar - not no change vs change.
Both pictures have the change.

[Reithan]

The key problem being solved with this patch is "complex slow grammars that cause combinatorial explosion of stacks"

You can see this reliably with a test grammar like

# ── character classes ─────────────────────────────────────────────
U ::= [A-Z]        # one required upper-case letter
o ::= "" | [a-z]   # 18 independent “present or not” lower-case letters

# ── a single high-fan-out segment ─────────────────────────────────
segment ::= U o o o o o o o o o o o o o o o o o

# ── entry point: unlimited repetition of that segment ─────────────
root ::= segment+

On un-patched branch, this runs at ~5T/s on my setup

With this patch, even setting the batch cutoff much lower, like 680b, we see this:

Almost as fast as no grammar at all (35-37T/s).

While this is more complex than no check at all, I think a simple 'hash and check' memoization scheme is pretty standard dynamic programming. This should be familiar to most engineers, so I don't worry too much about maintenance.

The key tuning bit would be the cutoff value. A high value like the current 1280 catches many many branches, but also runs the has very often, which may not be worth it, since most of those resolve quickly anyway.

A lower values like 640 or even lower will catch many less branches early - but still stop the degenerate 'stuck' cases.

I could also reverse the checks and 'early-out' on candidate size before we even do the stack hash to save even more time.

[Reithan]

Tested with doing size check first and lowering byte cutoff, both result is roughly the same speed.

tl;dr this change makes complex grammar that normally gets 'stuck' or slows down considerably run at 'normal' speed.
This does nothing for grammar that already runs smoothly.

So, you get:

• ~0% speed up/down for normal case
• 700%+ speed up in 'hard' cases. (or more!)

[Reithan]

Actually doing an aggressive size-check first speeds up some 'easy' cases - like the tool call example:

40T/s up from 35T/s

I'll push that change.

[LostRuins]

Do we never clear the cache? Will that be a problem? Seems like we might want to clear the cache upon each new gen.

[Reithan]

Do we never clear the cache? Will that be a problem? Seems like we might want to clear the cache upon each new gen.

I was debating that. In local use, not clearing that cache lets it use that speedup over every new gen, which helps quite a bit.

But if we're running in horde mode, or changing grammar, or things like that, clearing cache would be useful.

• Testing memo cache reset now.

[Reithan]

• Added cache reset tied to current grammar-reload logic.

[LostRuins]

You accidentally committed version.txt

[LostRuins]

Btw it's not compiling on my own pc

src/llama-grammar.cpp:915:58: error: cannot bind non-const lvalue reference of type 'std::__detail::_Node_iterator...

had to change line 915
if (auto & cache_hit2 = candidates_memos.find(candidates_hash); cache_hit2 != candidates_memos.end()) {
to
if (auto cache_hit2 = candidates_memos.find(candidates_hash); cache_hit2 != candidates_memos.end()) {

and line 911 as well.

Other than that, seems to be fine.

[Reithan]

Interesting. I guess just compiler differences. I've always made a habit of specifying & or * for auto types, JIC. But if your compiles won't do it, that's fine. I'll remove those.

[LostRuins]

Merging. Do let me know if any issues arise (to anyone else too)