chat : fix gemma4 required tool grammar

[aldehir]

Overview

Fixes #29089. The parsing for awkward edge cases negatively impacts the grammar when tool_choice = required.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, for running test scenarios.

[pwilkin]

He's right, though. An interaction of two untils is incorrect. His analysis basically shows that if you have two untils, you can have an incorrect parse where the second untils partial parse completes the string that's the delimiter for the first until, i.e.

(<|tool_call)(>something)

[pwilkin]

For required tool calls this is the right workaround, but we should think about a way to fix the "double until" behavior.

[aldehir]

Well, that's why we remove them! It's only a problem in a CFG grammar. The PEG parser, being greedy, doesn't have the same issue. For grammars, ac() gives better guarantees at the cost of ergonomics since the rule consumes the delimiter.

[aldehir]

but we should think about a way to fix the "double until" behavior.

This problem will mostly go away once we add token-aware parsing and we can construct grammars that operate on tokens.