A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent

Overview

The local ai-chat agent currently runs Gemma 4 E4B (Q4_K_M), a ~4B-effective-parameter model. Recent agent-loop fixes (response token cap, turn-structure normalization, repetition-penalty sampling, leaked tool-call-prose stripping) removed the infrastructure-level failure modes — hangs, runaway generation, and template-token leakage — that were independent of model size.

What remains are capability-bound rough edges that a larger model is expected to improve:

Calling tools (e.g. search_semantic) on purely conversational turns despite explicit "no tools on conversational turns" guidance.
Verbose, multi-paragraph answers where 1–2 sentences were requested.
Trailing "thinking aloud" after the answer is complete.
Over-eager schema/relationship creation beyond what was asked.

This issue establishes a controlled A/B comparison of Gemma 4 12B (Q4_K_M) against E4B on the corrected agent loop, so we measure what model capability buys us on top of the fixes rather than conflating the two.

Problem Statement

We do not yet know the real quality ceiling of the local agent because, until the loop was corrected, model behavior was masked by infrastructure bugs. Before deciding whether to change the default model or build model-capability guardrails (detecting too-complex requests and asking the user to simplify), we need empirical data on how a larger model behaves on the same scenarios.

The trial must also confirm that 12B is operationally viable on the target hardware class (Apple Silicon, 16 GB unified memory), where the Metal GPU allocation ceiling is ~10.6 GB and a 12B Q4_K_M footprint (weights + KV cache + compute buffers at the 32K context budget) lands around 9–10 GB — fitting, but with little headroom.

Proposed Solution

Run both models through an identical scenario matrix on the corrected agent loop using the existing headless CLI harness (scripts/aichat.ts), and record per-scenario tool-routing decisions, response quality, and latency. Use the results to recommend whether to keep E4B, switch to 12B, or make the model configurable, and to inform a follow-up model-capability-guardrails issue.

Scenario Matrix

Run each scenario against both models on a freshly seeded test database:

Greeting ("Hi there") — expect no tool calls, short reply
Capability question ("What can you do?") — expect no tool calls, concise reply, no trailing reasoning
Schema creation ("Create an invoice tracking database") — expect exactly one create_schema, no proactive related-type creation
Instance creation against the new schema ("Add an invoice for $500 due next Friday")
List/query ("List all my invoices") — expect search_nodes with type filter
Update ("Mark invoice #X as paid") — expect search-then-update
Empty-result query (search for something that does not exist) — expect single statement, no retry
Multi-custom-type CRUD: create 2–3 custom types via natural language, then create and query instances across them

Acceptance Criteria

Confirm 12B Q4_K_M loads and runs on a 16 GB Apple Silicon machine without falling back to CPU or exhausting the Metal allocation ceiling
Both models run the full scenario matrix on the corrected agent loop
Per-scenario results captured for both models: tool calls made, whether routing was correct, response text, and wall-clock latency
Side-by-side comparison documented in a comment on this issue (table or per-scenario notes)
A recommendation recorded: keep E4B, switch default to 12B, or make the model user-configurable — with the rationale (quality delta vs. latency/memory cost)
Findings feed a follow-up model-capability-guardrails issue (link it)

Technical Specifications

Reference Files

scripts/aichat.ts — headless harness driving an ai-chat node via the CLI (new / send <id> <msg> / ask <msg> / show <id>); honors NS_MODEL, NODESPACED_SOCKET, NS_LOG, NS_TIMEOUT_MS
packages/cli model {list|load|recommended} subcommand — loads a model into the test daemon
packages/agent/src/local_agent/agent_loop.rs — corrected ReAct loop (MAX_RESPONSE_TOKENS cap, turn-structure handling)
packages/nlp-engine/src/chat/mod.rs — sampler chain (repetition penalty) and generation loop
packages/agent/src/agent_guidance.rs, prompt_assembler.rs, skill_pipeline.rs — seeded prompt/skill guidance under test

Acquisition

The 12B Q4_K_M GGUF is not yet present on the machine and must be downloaded (~7 GB) and registered with the model loader before the trial. Record the source and exact filename used so the comparison is reproducible.

Test Procedure

Build release daemon + CLI, start a fresh headless test daemon on a throwaway DB/socket (re-seeds prompts/skills), load the model under test via model load <id>, then drive bun run chat ask "..." per scenario.
Run E4B and 12B against the same corrected build so the only variable is the model.

Non-Goals

Implementing model-capability guardrails (separate follow-up issue informed by these results)
Changing the default model in code (gated on this trial's recommendation)
Evaluating non-Gemma model families

Related Issues

Wire the agent's embedding service by construction instead of re-injecting it per engine swap #1328 (embedding service wired by construction)

Note: special-token coverage check (E4B → 12B)

Gemma 4 E4B and 12B share the same tokenizer and embedded chat template, so the special-token strippers landing in the corrected loop — strip_gemma_special_tokens (nlp-engine) plus the channel-block / tool-call-prose strippers (agent response_processing.rs) — and the ffi error -3 template fix (null content + normalized tool-call arguments) all apply to 12B unchanged; they are family/template-level, not size-specific.

The leaks these strippers clean up (<|channel>thought…, call:foo{…} narrated as prose) are capability failures — a weaker model losing track of which channel it is emitting into — so the 12B run should hit them far less often, turning the strippers into a safety net rather than a load-bearing fix.

Acceptance check to add during the A/B: after running the matrix on 12B, grep the raw daemon log for any un-stripped <|…> / <…|> markers in the final assistant replies. If the downloaded 12B GGUF was converted with a slightly different template variant (e.g. a <|channel>final marker not covered verbatim), add it to the strip lists. Record whichever GGUF source/quantization was used so this is reproducible.

Capability signal observed on E4B (baseline for the A/B)

On the corrected loop, E4B still exhibits a cross-turn "thinking aloud" leak that is capability-bound, not infrastructural:

It produces a correct, complete answer, then — because it keeps taking ReAct iterations — emits an internal reasoning block as content on a later turn (e.g. …What would you like to do today?thought\nThe user responded with "Hi there"…).
Crucially this leak is unmarked: the model emits a bare thought\n… with NO <|channel> / <channel|> delimiter tokens (verified: zero channel markers in the raw daemon log for these turns). So the marker-based strippers cannot catch it without risking eating legitimate prose containing the word "thought".

Two interacting capability misses drive it:

E4B calls tools (e.g. search_semantic) on purely conversational turns despite an explicit "conversational turns use NO tools" rule — which forces extra ReAct iterations.
Once it has answered, it does not stop; the extra iterations are where the bare-thought rambling appears.

This is a prime thing for the A/B to quantify: does 12B (a) skip tools on conversational turns, and (b) stop after a complete answer? If so, the rambling disappears without needing fragile loop-level surgery or bare-"thought" regex stripping. If 12B still rambles, that argues for a loop-level "stop re-prompting after the model emits a complete text answer" change (deeper agent_loop work, must not break multi-step tool chains).

Fix ai-chat runaway/hang + tool-call template leaks; tune agent prompts

Address review: protect tool-turn pairing in summarization; capture s…

ai-chat agent quality: fix runaway/hang + tool-call template leaks, t…

Frontend baseline (worktree issue-1329-ab-test-12b): Test Files 133 passed, Tests 3841 passed. Duration 37.43s.

A/B trial results: Gemma 4 12B (Q4_K_M) vs E4B on the corrected agent loop

Hardware: Mac mini, Apple M2 Pro, 16 GB unified memory (the target class). Metal reported ~12.1 GB usable GPU allocation.
Build: corrected agent loop on main (post #1331). Both models run the same daemon/CLI build; the only variable is the model.
Harness: scripts/aichat.ts driving a fresh ai-chat node per scenario group, throwaway test socket/DB (real ~/.nodespace untouched), prompts/skills freshly seeded.

Acquisition (reproducible)

Source: https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF → gemma-4-12B-it-Q4_K_M.gguf
On-disk size: 7,381,382,048 bytes (6.9 GiB) — Q4_K_M weights (284 q4_K + 45 q6_K + 338 f32 tensors).
This is the quantized build; the bf16 original is ~24 GB. Weight quantization was never the constraint (see below).

Headline: 12B is not viable on 16 GB for this agent. Recommendation: keep E4B.

The trial uncovered two independent blockers, either of which alone disqualifies 12B here.

Blocker 1 — Memory: the f16 KV cache, not the weights, busts the budget

ChatConfig::default() hardcodes n_ctx: 32_768 for every chat model (packages/nlp-engine/src/chat/types.rs). At 32K, Gemma 4 12B allocates a 10.24 GB KV cache (it uses key_length=512, value_length=512 across 40 KV layers, f16):

	weights (GPU)	KV cache @32k	total Metal	fits in ~12 GB?
E4B	~5.0 GB	~1.8 GB	~7 GB	✅ comfortable
12B	~7.0 GB	~10.24 GB	~17.7 GB	❌ overshoots by ~6 GB

At 32K the first real turn drove the machine deep into swap (4.7 GB used) and never completed in 180 s — the daemon stayed pinned in status:processing. The model loads fine (all 49/49 layers offload to Metal in ~7 s, no CPU fallback); it's the runtime KV cache that doesn't fit. KV-cache quantization is not wired in the engine (ChatConfig has no type_k/type_v), so Q4_K_M weights don't help the KV cache.

Reducing context to 8K (probe: n_ctx = 8_192) shrinks the KV cache to ~2.69 GB, total ~10.2 GB — which does fit with no swap thrash. So 12B is loadable and runnable on 16 GB only at a reduced context. But then Blocker 2 dominates.

Blocker 2 — Behavior at 8K: 12B stops calling tools and fabricates results

Running the full 8-scenario matrix on 12B @ 8K, the model emitted a thought preamble and prose describing actions it never performed, with tool_calls=0 on 9 of 12 turns:

"I have created the new invoice for you… ID: 104" — no create_node was ever called.
"I have updated the status of your $500 invoice… Invoice ID: 124 → Paid" — no update_node; the invoice doesn't exist.
"I searched your records… the only record is the $500 payment from Project Alpha" — invented; nothing was searched or created.

This is worse than E4B's rough edges: E4B at least invokes tools (sometimes wrongly), whereas 12B confidently hallucinates tool successes — actively misleading. When it did emit tool calls (the multi-type group), it returned no user-facing reply and double-fired create_schema, or dumped invented JSON as if it were the database. Every reply also leaks the bare thought token (and sometimes a trailing one) — the unmarked-narration leak from #1331, pervasive here.

Root cause is consistent with the prior loop work: the agent loop's tool-call parsing/turn-structure handling was tuned to E4B's output. Gemma-4-12B's chat template (distinct <|tool_response>/<turn|> EOG tokens and a leading thought channel) isn't parsed as tool calls, so the loop sees prose and the model free-associates.

Blocker 3 — Latency

Even at 8K and fitting in memory, 12B ran median 126 s/turn (range 8–241 s) — running near the memory ceiling on an M2 Pro. E4B ran median ~34 s (range 8–79 s). 2-minute replies are not interactive.

Scenario matrix (side by side)

tools = tools actually invoked (from daemon log). ✅/⚠️/❌ = routing vs. expectation.

#	Scenario	E4B tools / latency	E4B verdict	12B@8K tools / latency	12B verdict
1	Greeting "Hi there"	`search_semantic`×3, `update_task_status` / 64s	❌ tools on chat turn; verbose	`[]` / 26s	✅ no tools, concise
2	"What can you do?"	`search_skills` / 30s	⚠️ 1 spurious tool; verbose	`[]` / 122s	✅ no tools (but verbose, slow)
3	Create invoice DB	`create_schema`×3 (+2 sem) / 14s	⚠️ created, but 3× + extras	`[]` / 122s	❌ claims created, no tool call
4	Add $500 invoice	`create_node` (+sem) / 79s	✅ created instance	`[]` / 126s	❌ fabricated "ID 104"
5	List invoices	`search_nodes`×4 (+sem) / 57s	❌ couldn't filter "all"	`[]` / 126s	❌ fabricated a table
6	Mark invoice paid	`update_node` / 25s	❌ tool errored	`[]` / 145s	❌ fabricated "ID 124 → Paid"
7	Empty-result query	`search_nodes` / 8s	✅ single statement	`[]` / 143s	❌ invented "Project Alpha $500"
8a	Create type: book	`create_schema`×3 (+sem) / 32s	⚠️ created, 3×	`create_schema` / 240s	⚠️ created, no reply
8b	Create type: contact	`create_schema`×2, `update_schema`×2, `update_node` / 35s	⚠️ created, thrashy	`create_schema`×2 / 241s	⚠️ double-fired, no reply
8c	Add a book	`search_nodes`,`create_node`,`update_node` (+sem) / 44s	✅ created	`create_schema`×2 / 75s	❌ dumped invented JSON
8d	Add a contact	`create_node`,`get_related_nodes`,`update_node` (+sem) / 56s	✅ created	`[]` / 52s	❌ dumped invented JSON
8e	Query across types	`search_nodes`×2 / 23s	⚠️ "no books" (lost the create)	`[]` / 8s	❌ fabricated result

Totals: E4B — every actionable scenario attempted tools (with the known over-eager/duplicate/verbose rough edges). 12B@8K — 5 tool calls across the whole matrix; 9/12 turns called nothing and narrated fictional outcomes.

Acceptance criteria

12B loads/runs on 16 GB without CPU fallback / Metal exhaustion — only at reduced context. At the shipping 32K budget it overshoots (~17.7 GB) and swap-thrashes; at 8K it fits (~10.2 GB). All layers offload to GPU either way.
Both models run the full scenario matrix on the corrected loop (12B at 8K; at 32K it cannot complete a turn).
Per-scenario tool calls / routing / response / latency captured (table above).
Side-by-side comparison documented (this comment).
Recommendation: keep E4B as the default (rationale below).
Follow-up guardrails issue linked (below).

Recommendation — keep E4B; do not add 12B to the catalog yet

Memory: 12B only fits on 16 GB by cutting context to 8K, which halves the agent's working window.
Correctness: at 8K, 12B regresses tool use to near-zero and fabricates results — a worse failure mode than E4B's over-eager-but-real tool calls.
Latency: 126 s median is not interactive.

The 12B GGUF and the (uncommitted, worktree-local) catalog entry are kept for follow-up once the prerequisites below land. No production code change ships from this trial.

What would make a larger model worth re-testing (prerequisites, not part of this issue)

KV-cache quantization (type_k/type_v q8_0/q4_0 in ChatConfig + engine wiring): would let 12B keep a larger context inside the budget — the real unlock, since weights are already minimal.
Per-model n_ctx sourced from the catalog context_window instead of the hardcoded 32K default.
Tool-call parsing for Gemma-4-12B's template (thought channel + <|tool_response>/<turn|> EOG): without this, no Gemma model larger than E4B will route tools correctly regardless of memory.

Reproduce

cargo build --release -p nodespace-daemon -p nodespace-cli
NODESPACED_HEADLESS=1 NODESPACED_SOCKET=/tmp/nodespaced-test/daemon.sock \
  NODESPACED_DB_PATH=/tmp/nodespaced-test/nodespace.db target/release/nodespaced &
target/release/nodespace --socket /tmp/nodespaced-test/daemon.sock model load <gemma-4-e4b-q4km|gemma-4-12b-q4km>
NS_BIN=target/release/nodespace NODESPACED_SOCKET=/tmp/nodespaced-test/daemon.sock \
  NS_LOG=/tmp/nodespaced-test/daemon.log NS_MODEL=<id> bun run scripts/aichat-matrix.ts <label> out.json

(32K is the default; the 8K probe was a one-line n_ctx change in chat/types.rs.)

Follow-up issue created: #1332 — Model-capability guardrails + larger-model prerequisites (anti-fabrication guard, KV-cache quantization, per-model n_ctx, Gemma-4-12B tool-call parsing). It carries forward the unblockers this trial surfaced before any larger model is worth re-testing.

Trial setup pushed to branch issue-1329-12b-trial (commit 1c2617b) to continue on more capable hardware and extend the comparison to Gemma 4 31B. It adds the 12B catalog entry, the n_ctx=8K probe (revert to 32K on a machine with headroom), scripts/aichat-matrix.ts, and scripts/aichat-1329-trial.md with reproduce steps. Note: blocker 2 (12B fabricates tool results / tool-call template not parsed) is hardware-independent and must be fixed (#1332) before any larger Gemma routes tools correctly.

Model-capability guardrails + larger-model prerequisites (anti-fabric…

WIP: #1329 12B-vs-E4B A/B trial setup + first-run findings (keep E4B)

Revert n_ctx trial probe: restore 32_768 (was 8_192 for 16GB fit test)

Fix Ollama streaming tool calls, remove tool-iteration token cap, fix…

Fix Ollama streaming tool calls + remove tool-iteration token cap (cl…

Upgrade vendored llama.cpp so local models route tools reliably (gate…

ai-chat agent quality: fix runaway/hang + tool-call template leaks, t…

Model-capability guardrails + larger-model prerequisites (anti-fabric…

Fix Ollama streaming tool calls + remove tool-iteration token cap (cl…

Upgrade vendored llama.cpp so local models route tools reliably (gate…

Navigation Menu

Repository navigation

A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent #1329

Overview

Problem Statement

Proposed Solution

Scenario Matrix

Acceptance Criteria

Technical Specifications

Reference Files

Acquisition

Test Procedure

Non-Goals

Related Issues

Note: special-token coverage check (E4B → 12B)

Capability signal observed on E4B (baseline for the A/B)

A/B trial results: Gemma 4 12B (Q4_K_M) vs E4B on the corrected agent loop

Acquisition (reproducible)

Headline: 12B is not viable on 16 GB for this agent. Recommendation: keep E4B.

Blocker 1 — Memory: the f16 KV cache, not the weights, busts the budget

Blocker 2 — Behavior at 8K: 12B stops calling tools and fabricates results

Blocker 3 — Latency

Scenario matrix (side by side)

Acceptance criteria

Recommendation — keep E4B; do not add 12B to the catalog yet

What would make a larger model worth re-testing (prerequisites, not part of this issue)

Reproduce

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent #1329

Description

Overview

Problem Statement

Proposed Solution

Scenario Matrix

Acceptance Criteria

Technical Specifications

Reference Files

Acquisition

Test Procedure

Non-Goals

Related Issues

Activity

malibio commented on Jun 5, 2026

Note: special-token coverage check (E4B → 12B)

malibio commented on Jun 5, 2026

Capability signal observed on E4B (baseline for the A/B)

malibio commented on Jun 5, 2026

malibio commented on Jun 5, 2026

A/B trial results: Gemma 4 12B (Q4_K_M) vs E4B on the corrected agent loop

Acquisition (reproducible)

Headline: 12B is not viable on 16 GB for this agent. Recommendation: keep E4B.

Blocker 1 — Memory: the f16 KV cache, not the weights, busts the budget

Blocker 2 — Behavior at 8K: 12B stops calling tools and fabricates results

Blocker 3 — Latency

Scenario matrix (side by side)

Acceptance criteria

Recommendation — keep E4B; do not add 12B to the catalog yet

What would make a larger model worth re-testing (prerequisites, not part of this issue)

Reproduce

malibio commented on Jun 5, 2026

malibio commented on Jun 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions