[Reithan]

Overview

This PR is a collection of changes from PRs that merged to koboldcpp months ago. The purpose of these changes was to remove possible edge cases from schema or grammar constrained inference that caused slowdowns or hangs. The core mechanism here was a hand-tuned memoization cache to avoid runaway branch explosion while parsing the grammar, along with some other basic optimizations around how grammar interacts with other samplers.

This PR only migrates the memoization changes, 2 more PRs branches are prepared to migrate the compaction and pre-cull changes after this is reviewed.

Additional information

The prior related PRs in the koboldcpp repo are:

• 1615: memoization, lifted degenerate use cases from ~5 T/s on an RTX 5090 to ~35-37 T/s. No change in speed on simpler/normal grammar use cases.
• 1606: compaction, 20-40% speed improvement on heavy grammars, light grammar no slower than prior baseline
• 1597+1600: pre-culling tokens for grammar, with all 3 sets of changes, a small but significant speed increase was seen in simple grammars, roughly ~3x increase in 'moderate' cases, and up to 30x increase in previously degenerate cases, plus the elimination of most hangs.

If fresh benchmark numbers are needed, I will be able to provide them soon.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: The original PRs were written by hand, AI was used to fetch the old work, compare it to the current state of llama.cpp and aggregate the prior changes into new PRs. This is the first of 3 that will cover all the prior work migration to upstream.

[ggml-gh-bot]

Hi @Reithan, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[Reithan]

Hi @Reithan, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.
• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.
• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Summary revised with hand-written description following proper template. Sorry for the mix-up.

[Reithan]

CI run locally with result:

All tests passed.

real    0m2.423s
user    0m7.932s
sys     0m0.190s
```
### ctest_with_model_debug

Runs ctest with model files in debug mode
- status: 127
```
+ LLAMACPP_TEST_MODELFILE=/home/reithan/projects/llama.cpp/tmp/mnt/models/qwen3/0.6B/ggml-model-q4_0.gguf
+ time ctest -C Debug --output-on-failure -L model
./ci/run.sh: line 379: time: command not found
```

This should cover all tests relevant to this PR. The time: command not found seems to be an issue with the CI script? I'm running WSL2 on windows.

[Reithan]

Grammar specific results from CI run were:

(.venv)~/projects/llama.cpp$ grep -i "test-llama-grammar" ./tmp/results/*-ctest.log
./tmp/results/ctest_debug-ctest.log:      Start 22: test-llama-grammar
./tmp/results/ctest_debug-ctest.log:22/50 Test #22: test-llama-grammar .........................   Passed    0.00 sec
./tmp/results/ctest_release-ctest.log:      Start 22: test-llama-grammar
./tmp/results/ctest_release-ctest.log:22/53 Test #22: test-llama-grammar .........................   Passed    0.00 sec

[Reithan]

Baseline bench results for regression check:

| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen3 0.6B Q4_0                | 358.21 MiB |   596.05 M | CPU        |       4 |           pp512 |       404.22 ± 11.98 |
| qwen3 0.6B Q4_0                | 358.21 MiB |   596.05 M | CPU        |       4 |           tg128 |         93.44 ± 1.31 |

build: 4cbe8b070 (10803)
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen3 0.6B Q4_0                | 358.21 MiB |   596.05 M | CPU        |       4 |           pp512 |       404.49 ± 19.03 |
| qwen3 0.6B Q4_0                | 358.21 MiB |   596.05 M | CPU        |       4 |           tg128 |         93.03 ± 1.61 |

build: c68fd6ea6 (10805)

[Reithan]

Simple standard grammar test shows minor improvement over baseline (baseline shown first, then PR):

Generate a JSON object with fields for name, age, and address:{"name": "John", "age": 25, "address": "123 Main Street"}
                     [end of text]


0.02.177.730 I common_perf_print:    sampling time =      98.94 ms
0.02.177.731 I common_perf_print:    samplers time =       4.34 ms /    57 tokens
0.02.177.734 I common_perf_print:        load time =    1353.43 ms
0.02.177.747 I common_perf_print: prompt eval time =      41.06 ms /    14 tokens (    2.93 ms per token,   340.98 tokens per second)
0.02.177.748 I common_perf_print:        eval time =     466.56 ms /    42 runs   (   11.11 ms per token,    90.02 tokens per second)
0.02.177.748 I common_perf_print:       total time =     608.83 ms /    56 tokens
0.02.177.749 I common_perf_print: unaccounted time =       2.28 ms /   0.4 %      (total - sampling - prompt eval - eval) / (total)
0.02.177.749 I common_perf_print:    graphs reused =         41
...
0.01.495.789 I common_perf_print:    sampling time =      93.05 ms
0.01.495.790 I common_perf_print:    samplers time =       3.86 ms /    57 tokens
0.01.495.794 I common_perf_print:        load time =     692.78 ms
0.01.495.795 I common_perf_print: prompt eval time =      40.78 ms /    14 tokens (    2.91 ms per token,   343.28 tokens per second)
0.01.495.796 I common_perf_print:        eval time =     459.76 ms /    42 runs   (   10.95 ms per token,    91.35 tokens per second)
0.01.495.796 I common_perf_print:       total time =     596.09 ms /    56 tokens
0.01.495.797 I common_perf_print: unaccounted time =       2.50 ms /   0.4 %      (total - sampling - prompt eval - eval) / (total)
0.01.495.797 I common_perf_print:    graphs reused =         41

[Reithan]

Testing a purely degenerate case shows that the current dedupe strategy doesn't fully cover cases of ambiguous grammar. In cases where multiple options create duplicate alternatives (identical character classes or literals defined separately), this can still cause exponential branch explosion due to distinct rule/body pointers that the existing dedupe can't collapse.

Tested with:

# Stress test for grammar ambiguity: `sp` has three identical alternatives, so every
# leading space in a candidate token can be matched three ways. Without memoization of
# llama_grammar_reject_candidates_for_stack, tokens that consist of N spaces followed by a
# character the grammar cannot accept (e.g. the "<N spaces>\r\n" tokens in the Qwen vocab,
# N up to 24) cost 3^N work per rejection pass.
root   ::= line+
line   ::= indent ([a-zA-Z] [^\r\n]*)? "\n"
indent ::= sp*
sp     ::= " " | " " | " " | "\t"

Result is baseline times out, while PR build gives:

0.02.323.246 I common_perf_print:    sampling time =     267.22 ms
0.02.323.248 I common_perf_print:    samplers time =       1.94 ms /    31 tokens
0.02.323.251 I common_perf_print:        load time =    1543.32 ms
0.02.323.252 I common_perf_print: prompt eval time =      34.06 ms /    11 tokens (    3.10 ms per token,   322.93 tokens per second)
0.02.323.253 I common_perf_print:        eval time =     217.31 ms /    19 runs   (   11.44 ms per token,    87.43 tokens per second)
0.02.323.253 I common_perf_print:       total time =     519.93 ms /    30 tokens
0.02.323.254 I common_perf_print: unaccounted time =       1.35 ms /   0.3 %      (total - sampling - prompt eval - eval) / (total)
0.02.323.254 I common_perf_print:    graphs reused =         18