Gemma 4 12B runs away (4096-token cap every turn) + OAI parser ffi error -3 through NodeSpace's inference path, despite clean output via raw llama-server #1365

Overview

Re-running the #1329 scenario matrix against Gemma 4 12B (Q4_K_M) on the current stack (upgraded llama.cpp #1361 + skill-mediated routing #1352 + instruction-bearing skills #1356) shows 12B still misbehaves through NodeSpace's inference path — even though the same model produces clean, valid tool calls via raw llama-server --jinja. The binding upgrade was necessary but not sufficient.

This is the trustworthy re-measurement #1329 was meant to produce: 12B is not yet shippable through NodeSpace, but for different, fixable reasons than #1329's original (now-superseded) verdict.

Evidence (clean DB, fresh seed, prompt-dump on, current main)

Captured via NODESPACE_PROMPT_DUMP + daemon log:

1. Runaway generation — every turn hits the 4096-token ceiling. 3 of the first 3 substantive turns generated exactly 4096 completion tokens:

   • "What can you do?" (conversational, should be 1–2 sentences) → 4096 tokens, ~3 min.
   • "Create an invoice tracking database" → 4096 tokens.
     Final replies are capped at MAX_RESPONSE_TOKENS = 2048, but tool-calling iterations use max_tokens: None and hit the hard 4096 cap in agent_loop.rs:951. The model never emits a stop.

2. OAI parser update error: ffi error -3 during streaming (packages/nlp-engine/src/chat/mod.rs:529). The daemon died mid-run.

3. Wrong tool / wrong outcome on schema creation. The dump showed the model was offered only 5 tools (search_nodes, search_semantic, get_node, create_node, update_node) — no create_schema, no search_skills — and an old 957-char system prompt, not the skill-mediated prompt. The model fell back to create_node with empty args (missing field node_type), retried as a text node, and produced a generic text node instead of an Invoice schema.

Contrast: raw llama-server is clean

The identical 12B GGUF via llama-server --jinja (build 8660, native chat template + sampler) produced one clean, valid create_schema call with correct nested fields. So the failure is in NodeSpace's inference plumbing, not the model or the engine version:

• NodeSpace uses its own OpenAIChatTemplateParams + custom OAI delta parser (emit_oai_delta) + sampler chain, rather than the model's native jinja path.
• 12B runs away and throws ffi error -3 specifically through that path.

Suspected causes (to investigate)

• Stop/EOG handling for Gemma 4 12B in the OAI delta parser path — the model isn't being stopped on its turn-end token, causing runaway to the cap. (Gemma 4's <turn|> EOG handling was a known sharp edge.)
• Tool offering: the 5-tool subset + old prompt in the dump suggests either a stale-build artifact or that the schema/skill tools aren't reaching 12B's turn — needs confirming against a definitively-current daemon.
• ffi error -3 is the same class of error documented at chat/mod.rs:745,1212 (template/decode failure) — may be a malformed message shape or a Metal decode issue under 12B's config.

Acceptance Criteria

• Root-cause the runaway generation (stop/EOG token not honored for 12B through the OAI parser path).
• Root-cause and fix OAI parser update error: ffi error -3 for 12B.
• Confirm 12B is offered the full tool set (incl. create_schema, search_skills) and the skill-mediated prompt under the current build.
• Re-run the scenario matrix against 12B end-to-end without runaway, parser errors, or daemon death.
• Document the verdict: keep E4B vs. ship 12B, on a trustworthy in-NodeSpace measurement.

Technical Specifications

Reference Files

• packages/nlp-engine/src/chat/mod.rs — OAI delta parser (emit_oai_delta, line ~529 error site; ffi error -3 notes at 745, 1212), sampler chain (~433).
• packages/agent/src/local_agent/agent_loop.rs — token caps (MAX_RESPONSE_TOKENS=2048 line 47; uncapped tool iterations line 363; 4096 cap line 951).
• packages/agent/src/local_agent/model_manager.rs — GEMMA_4_12B catalog (context_window 32_768, Q8_0 KV).
• scripts/aichat-matrix.ts / NODESPACE_PROMPT_DUMP — reproduction harness.

Reproduce

# build current main; wipe + fresh-seed test DB; start daemon with NODESPACE_PROMPT_DUMP set
target/release/nodespace --socket /tmp/nodespaced-test/daemon.sock model load gemma-4-12b-q4km
NS_BIN=target/release/nodespace NODESPACED_SOCKET=/tmp/nodespaced-test/daemon.sock \
  NS_LOG=/tmp/nodespaced-test/daemon.log NS_MODEL=gemma-4-12b-q4km NS_TIMEOUT_MS=180000 \
  bun run scripts/aichat-matrix.ts gemma-12b out.json
# inspect: bun run prompt:dump <dump.jsonl> --last

Related Issues

• A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent #1329 (original A/B trial — verdict superseded; this is the trustworthy re-measurement)
• Upgrade vendored llama.cpp so local models route tools reliably (gates the skill-routing redesign; supersedes #1329 verdict) #1361 (llama.cpp upgrade — fixed the API/build level, not this plumbing-level issue)
• Model-capability guardrails + larger-model prerequisites (anti-fabrication, KV-cache quant, per-model n_ctx, Gemma-12B tool parsing) #1332 (model-capability guardrails / larger-model prerequisites — Gemma-12B tool parsing)
• ADR-036 (skill-mediated routing — what this matrix exercises)

Non-Goals

• Reverting the binding upgrade (it is correct and required).
• Promoting 12B to default before these are fixed.