🛡️ Fail closed on un-compilable grammar constraints; re-rail whitespace stalls

[sdougbrown]

Note

I was very pleased to see the structured output support but encountered problems with it using gemma4. This addresses the issues I found.

Grammar-constrained requests (json_schema, regex_pattern, grammar_string) fail in two ways:

1. An un-compilable schema, regex, or grammar is silently dropped. Generation runs without the constraint and returns HTTP 200. The compile error goes to server stdout only.
2. A grammar can forbid tokens, but it cannot force the model to continue or stop. When the legal continuation set includes unlimited whitespace, a model that "feels done" emits whitespace until max_tokens. The finish reason is length, and the output never satisfies the grammar. ExLlamaV3 loop detection does not catch this case: its 800-token window targets longer repeating sequences.

This branch fixes both. It carries two commits, and each commit is self-contained. I can split them into separate PRs if you prefer.

Commit 1: refuse to generate unconstrained on compile failure

• LLGuidanceFilter construction failures now raise GrammarParseError (new, in common/errors.py) instead of logging and skipping the filter.
• A new except GrammarParseError clause in each consumer maps the error. Three non-streaming handlers return HTTP 400. These are OpenAI-compatible (OAI) chat/completions, completions, and Kobold. Streaming handlers yield an error event that carries the compile failure.
• If the llguidance installation is missing or broken, the code re-raises the original exception. That failure still maps to a server error (503), not a client error.
• The failing schema, regex, or grammar is logged once with its compile exception.

Commit 2: re-rail whitespace stalls

• WhitespaceStallGuardFilter wraps each grammar filter. After stall_guard_tokens (default 8) consecutive pure-whitespace tokens, while the grammar is incomplete, the next logit mask excludes whitespace. The sampler then picks a legal content token, or an end-of-sequence (EOS) token. Each re-rail advances the grammar state, so repeated stalls terminate.
• If excluding whitespace empties the legal set, the grammar mask passes through unchanged.
• The wrapper delegates the feed, rewind, rollback, reset, and attach methods. It also mirrors is_active, trigger_token, and eos_after_completed. Job-level activation and deactivation keep working.
• Whitespace token ids come from one packed bitmask per handler, built from the tokenizer vocab. The code prefers inner get_vocab() and falls back to extended_id_to_piece. If the bitmask cannot be built, the guard disables itself.
• No exllamav3 changes are required.

stall_guard_tokens is a sampling param (0 or null disables; default 8) and is logged with the request settings.

Measured

Hardware: NVIDIA RTX 3090. Model: Gemma4-26B-A4B EXL3 3.1bpw. Identical prompts at temp 0 and at temp 1.0, top_p 0.95, top_k 40:

case	before	after
malformed json_schema	HTTP 200, unconstrained output	HTTP 400 with compile error
nested schema; trailing property skipped	5/5 length, ~300 tokens, unparseable	5/5 stop, ~160 tokens, valid JSON
schemas that never stall	3/3 valid	unchanged

Validation

• python -m pytest tests/ : 182 passed (157 pre-existing, 25 new).
• New tests cover GrammarParseError propagation for schema, regex, and llama.cpp GBNF (grammar format) inputs.
• Other new tests cover the 400 mapping on all three consumers.
• Further tests cover the stream error event. More cover guard activation and masking for packed and dense masks. More cover rewind, rollback, and is_active mirroring, plus the disable and threshold plumbing.

Limitations and follow-ups

• Behavior change: a request with an un-compilable constraint now fails instead of returning unconstrained text.
• The guard can mask 8+ consecutive whitespace tokens inside legitimate string content. Example: text art in a JSON string value. Set stall_guard_tokens: 0 per request to opt out.
• stall_guard_tokens has no sampler-override preset entry yet. Add one if you want per-preset control.

[turboderp]

I've merged the first commit (fail on unparsable constraint), but for the whitespace, LLGuidance has a whitespace_flexible option, which I defaulted to False instead, so default behavior allows compact JSON format only. (Send {"x-guidance": {"whitespace_flexible": true}} to override.)

Is this sufficient?

[sdougbrown]

probably! i'll check it out, thanks!

[sdougbrown]

@turboderp can finally confirm that technically yes the whitespace stall is solved with that config yes.

the interesting thing going on here though is that now the problem moved. with whitespace forbidden a model that thinks it's done before the schema is satisfied keeps repeating characters inside a string instead haha. i think based on my read of this is it's because the schema is applied as a mask but never exposed to the model unless manually put into the prompt. putting the schema in the system prompt (iirc vllm and sglang do this) would probably fix it, unless it's expected that the schema should be manually included each time?

i think this can be closed and i'd be happy to look into the schema system prompt injection if you want