Overview

Gemma 4 12B (Q4_K_M) running on the corrected agent loop emits tool calls as Python-style function calls in markdown code blocks rather than structured `[TOOL_CALLS]` tokens. The parser never fires, so every actionable turn produces zero real tool calls and the anti-fabrication guard from #1332 suppresses the response.

Observed in the #1329 M4 Pro / 48 GB A/B trial (post-#1332 run, with MLflow tracing):

```

Scenario: "Create an invoice tracking database"

raw_response: To create an invoice tracking system, I will first define a new schema...

```python
create_schema(name="Invoice", fields=[...])
```

thought
```

`[TOOL_CALLS]` never appears. The model knows the right action but renders it as prose code rather than a structured invocation.

Root Cause

The `OpenAIChatTemplateParams` passed by llama.cpp to the Gemma 4 12B chat template does not elicit the `[TOOL_CALLS]` sentinel format. Gemma 4 12B has a different internal chat template from E4B — it appears to default to a Python function-call representation when the tool-injection format is not exactly matched to what its template expects.

E4B works because its template (and the llama.cpp integration) produces `[TOOL_CALLS]{"tool_name":...,"tool_args":...}` which the existing parser handles. 12B's template either requires a different injection format or a different sentinel to trigger structured output.

Verified via MLflow traces (issue #1341): `raw_response` span attribute shows Python code blocks, `tool_calls_parsed` is always `[]`.

Proposed Solution

Investigate and fix the chat template / tool injection format for Gemma 4 12B:

1. Inspect the embedded chat template — use `llama_cpp_2` to dump the Jinja template from the 12B GGUF and compare it against E4B's template. Identify where tool-call format diverges.
2. Adjust `OpenAIChatTemplateParams` — if llama.cpp supports a template override or tool-format hint for 12B, wire it through `ChatConfig` or a per-model `chat_template_override` field in `CatalogEntry`.
3. Alternatively: add a Python-prose tool-call parser — detect the `function_name(...)` pattern in the response and parse it as a tool call. Less correct but unblocks 12B without template surgery.

Option 1 is preferred — it fixes the root cause and is the same approach needed for any future Gemma 4 variant.

Acceptance Criteria

• Gemma 4 12B correctly emits `[TOOL_CALLS]` (or equivalent structured format) on tool-requiring turns
• The `aichat-matrix.ts` scenario matrix on `gemma-4-12b-q4km` shows real tool calls on scenarios 3–8 (schema creation, instance creation, search, update)
• E4B behavior is unchanged
• MLflow traces show non-empty `tool_calls_parsed` for actionable 12B turns

Technical Specifications

Reference Files

• Chat template invocation: `packages/nlp-engine/src/chat/mod.rs` — `OpenAIChatTemplateParams` construction, where tool specs are injected into the prompt
• Tool-call parser: `packages/nlp-engine/src/chat/parser.rs` — `parse_tool_calls()` — currently handles `[TOOL_CALLS]` sentinel formats for Mistral and Gemma E4B
• Model catalog: `packages/agent/src/local_agent/model_manager.rs` — `GEMMA_4_12B` catalog entry — may need a `chat_template_variant` or `tool_call_format` field
• Trial branch: `issue-1329-12b-trial` — has the full matrix harness and MLflow tracing wired up for immediate re-verification

Observed Raw Output (12B, post-#1332)

```

Scenario 3: "Create an invoice tracking database"

create_schema(name="Invoice", fields=[{"name": "amount", "type": "number"}, ...])thought

Scenario 5: "List all my invoices"

search_semantic(query="List my invoices", node_types=["invoice"])thought

Scenario 6: "Mark the $500 invoice as paid"

search_semantic(query="Find my $500 invoice", node_types=["invoice"])
update_node(id="[target_invoice_id]", properties={"status": "paid"})thought
```

Hardware Context

• M4 Pro / 48 GB: 12B loads fine (7 GB weights + ~5 GB KV@32K with Q8_0), 17–90s/turn — viable if tool routing is fixed
• 16 GB machines: still not viable (KV cache bloat even with Q8_0)

Related Issues

• A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent #1329 (A/B trial that produced this data)
• Model-capability guardrails + larger-model prerequisites (anti-fabrication, KV-cache quant, per-model n_ctx, Gemma-12B tool parsing) #1332 (anti-fabrication guard, KV-cache quant, per-model n_ctx — prerequisites now done)
• Dev observability: emit agent turns as MLflow traces via OTLP (assembled prompt, tools, iterations) #1341 (MLflow tracing — used to observe raw_response)