grammar : compact candidate array after grammar rejection

[Reithan]

llama_grammar_apply_impl marks rejected tokens as logit = -INFINITY in-place, leaving cur_p->size at the full vocabulary size. Downstream samplers (top_k, softmax, dist) then operate on the full n-entry array even when grammar has reduced the valid set to M entries (M << n). For a 128k-token vocabulary, top_k performs an O(n) pass over all entries every sampling step (partial_sort for k ≤ 128, bucket sort above — src/llama-sampler.cpp:135–215); with compaction it operates on only M entries.

This PR switches rejection bookkeeping to a byte bitmask and compacts the array with std::remove_if before returning from llama_grammar_apply_impl. All downstream samplers benefit automatically. std::remove_if is stable, so a pre-sorted array remains sorted after compaction (the sorted flag is preserved).

Also fixes common/sampling.cpp: the is_valid check for single-token grammar validation previously tested data[0].logit != -INFINITY. After compaction, remove_if on a 1-element rejected range leaves the element in place with its original logit, so that check always passes, accepting grammar-invalid tokens and corrupting grammar state on the next llama_grammar_accept_impl call. Changed to single_token_data_array.size > 0. Third-party code wrapping llama_sampler_init_grammar that checks logit for validity should be updated to check size instead.

New failure mode (not a regression): When grammar rejects all tokens, cur_p->size reaches 0 and the next sampler hits its size assertion immediately. Upstream currently also aborts in this case, one step later (NaN-sampled token → !stacks.empty() assert in llama_grammar_accept_impl), so this is a fast-fail, not a regression.

Estimated improvement: 20–40% on heavy grammars (complex JSON schema, tool-call grammars). No regression on simple grammars or the no-grammar path.

See also: #28371 (grammar recursion memoization), #28373 (pre-cull top-k before grammar scan)

Adapted from LostRuins/koboldcpp PR #1606 by Reithan.

[ggml-gh-bot]

Hi @Reithan, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[Reithan]

Closing to comply with the 1-open-PR limit for new contributors. Will reopen after PR #28371 is processed. Branch remains at Reithan:grammar-compact-apply-impl.