Repository navigation
Redesign agent-matrix eval scenarios around NodeSpace's actual dev-workflow use case #1977
Description
Overview
All 16 scenarios in the agent-matrix eval (scripts/eval/fixtures/agent-matrix.ts) are generic personal/hobby CRUD tasks — tracking equipment, music albums, and venue bookings — rather than scenarios reflecting NodeSpace's actual positioning as context infrastructure for AI-native development. Model-lock decisions (ADR-046, ADR-056) and ongoing model re-evaluation work are scored against this matrix, meaning native-model selection is currently being decided by a model's ability to track laser cutters and music albums, not by its ability to do NodeSpace's actual job.
Problem Statement
The current scenario prompts:
- "I want to keep a record of the equipment my team checks out, whether it's been returned, and what each item costs to replace" (schema creation)
- "Start tracking albums I mean to listen to"
- "I also need a tracker for the venues I book"
- "Put down Kind of Blue, it's by Miles Davis"
- "Add a task to schedule the chip upgrade on the Polestar"
These exercise the right tool mechanics — create_schema, create_node, update_node, search_nodes, resolve_query — with realistic argument complexity (nested fields, enum coreValues, multi-step read-then-write sequences). But the domain is generic personal-database CRUD, unrelated to what NodeSpace is actually for.
Per the strategy docs (../nodespace-docs/strategy/vision.md, beliefs.md, principles.md), NodeSpace's target user is a developer or dev team using it as "context infrastructure for AI-native development": specifications, architecture decisions, tasks tied to features, and playbook-driven workflows, connecting "why it was built" to "what was built."
principles.mdSet up Tauri + Svelte project structure #1: "Code is artifact. Context is infrastructure."principles.mdSvelte frontend renders with placeholder panels #7: "Playbooks enforce context discipline" — an installable bundle of node-type definitions and executable rules.beliefs.md: "the spec defines what needs to be built, the constraints, the acceptance criteria, the architectural boundaries, before the agent writes a single line of code."
None of the current 16 scenarios touch specs, architecture decisions, playbooks, task-to-feature linkage, or architecture-decision retrieval — the dev-workflow use cases the product claims to serve. A model that excels at "track my albums" tells us little about whether it can, for example, retrieve the architecture decision that constrains a task before implementing it, or correctly link a bug fix back to the spec it violated.
This surfaced during #1956 (Gemma 4 12B re-evaluation): the matrix was the basis for judging whether 12B (in either GGUF source) or an experimental MoE variant cleared an "acceptable bar," but that bar itself has never been validated against NodeSpace's actual use case.
Proposed Solution
Design a scenario set themed around NodeSpace's actual dev-workflow use cases, for example:
- Creating a spec node and linking it to the architecture decision that constrains it.
- Retrieving the ADR that governs a given task before answering a question about it.
- Creating or updating a playbook's trigger/condition.
- Linking a bug fix back to the spec or acceptance criterion it violated.
- Querying "what decisions constrain this feature" (relationship traversal, not just property filtering).
- Task-to-feature linkage (creating a task and associating it with the feature/spec it implements).
Decide whether this replaces the existing 16 scenarios outright or runs alongside them as a second fixture (the existing set may still hold value as a generic tool-mechanics floor, separate from domain-fit).
Acceptance Criteria
- New scenario set (or replacement scenarios) designed around specs, architecture decisions, playbooks, and task-to-feature linkage, matching the vision/beliefs/principles docs' description of NodeSpace's target use case
- Decision made and documented on whether this replaces
scripts/eval/fixtures/agent-matrix.ts's existing 16 scenarios or runs as an additional fixture - New/changed prompts do not share wording with seeded guidance or skill content, verified by
guidance_is_not_contaminated_by_eval_prompts(packages/agent/src/agent_guidance.rs), which parsesprompt:literals out of every file inscripts/eval/fixtures/ - Scoring/expectation model reused or extended following the existing
Expectationtypes inscripts/eval/fixtures/agent-matrix.ts(toolOnce,toolSequence,minProperties, etc.) rather than inventing a parallel scoring mechanism -
bun run test:scriptspasses (the scoring-logic unit tests for the fixture) - Documented in
../nodespace-docs/development/agent-eval.mdif the fixture structure or scoring approach changes
Technical Specifications
Reference Files
- Scenario fixture:
scripts/eval/fixtures/agent-matrix.ts— current 16 scenarios,Expectationtype model, scoring logic - Scoring tests:
scripts/eval/fixtures/agent-matrix.test.ts— unit tests for the scoring function (no model/daemon needed) - Contamination guard:
packages/agent/src/agent_guidance.rs(guidance_is_not_contaminated_by_eval_prompts) — enforces that scenario prompts don't overlap with seeded guidance/skill wording - Eval harness docs:
../nodespace-docs/development/agent-eval.md— the shared runner, preflight gate, and setup instructions - Strategy docs:
../nodespace-docs/strategy/vision.md,beliefs.md,principles.md— the target-user/use-case framing this issue proposes scenarios align to
Non-Goals
- Re-running model comparisons against the new scenarios — that is separate follow-up work once the new scenario set exists.
- Changing the routing eval (
scripts/eval/fixtures/routing.ts) or the tool-call parser fixtures (chat::parser::tests) — this issue is scoped to the end-to-end agent-matrix eval only.
Related Issues
- Expose Gemma 4 12B as a selectable native model (once verified on adequate hardware) #1956 (where this gap was identified during Gemma 4 12B re-evaluation)
Activity
Expose Gemma 4 26B-A4B as an optional model tier, park 12B, fix per-m…
malibio commentedon Aug 17, 2026
Scope question: this issue covers the eval, but the guidance it measures is business-domain too
Filed #2118 (golden-prompt runner) and #2119 (assembly snapshot gate) alongside this. While scoping them, measured the domain-vocabulary skew across every model-facing surface, not just the matrix:
packages/agent/src/local_agent/tools.rspackages/agent/src/skill_pipeline.rsscripts/eval/fixtures/routing.tsscripts/eval/fixtures/agent-matrix.ts(business = invoice/customer/receivable/billing/booking/equipment/venue/vendor/payment/client; dev = bug/PR/commit/repo/sprint/ticket/issue/deploy/release/backlog/incident/code-review/ADR/spec/milestone/epic)
The matrix scenarios this issue targets are equipment checkout, albums, venues, and one car mod — none dev-flavored. But the guidance is worse, and it is the causal surface:
skill_pipeline.rsteachestitle_template = "Invoice #{{invoice_number}}", a fullInvoiceschema linked tocustomer, clarification examples built from "I found 3 invoices — Invoice #1 ($500)…", and a Node Creation description enumerating "Project, Customer, Invoice".tools.rsuses "Mark the $500 invoice as paid" asresolve_query's canonical example, in a parameter description the model reads every turn.Per ADR-038 Finding 2, the worked example in the guidance is what produces correct structured output — the model copies the pattern. So these examples are the template, not decoration.
Why it matters for this issue specifically: ADR-064 rule 3 makes skill descriptions index keys tuned against a retrieval benchmark. Retargeting the benchmark to dev intents while the descriptions and worked examples stay invoice-flavored tunes the two against each other in opposite directions — results would likely get worse with nothing actually wrong.
Recommendation: keep #1977 scoped to the eval (matrix + routing fixtures), and file the guidance/tool-schema example rewrite (
skill_pipeline.rs,tools.rs) as a separate issue that lands with or before it. Keeping them apart preserves the contamination boundary #1932 exists to enforce — golden/eval prompts and guidance examples must stay lexically distinct, or a passing scenario proves only that the model memorized its own answer key.Proceeding on that reading unless redirected. Happy to fold it all into #1977 instead if you'd rather have one unit of work.
malibio commentedon Aug 17, 2026
Sequencing: #2118 → #2120 → #2119 → #1977
These four are one line of work. Recording the order and the reasoning so it survives a context reset.
Why the snapshot gate is third, not first
It was initially argued for first, on the grounds that snapshotting today's assembly turns four independently-edited surfaces (
agent_guidance.rs,tools.rs,skill_pipeline.rs,routing.rs) into a reviewable diff — which is real, and would have caught thecoreValues/core_valuescontradiction ADR-064 records as shipped.But the snapshot would capture today's invoice-flavored prompt, and #2120 then rewrites it wholesale. Nearly every diff during that window is intended, so the gate's actual value — catching unintended drift — is thin exactly when it would be running. Capturing the golden once, after the rewrite settles, is cleaner than regenerating it through two rounds of deliberate change.
The risk this ordering carries, and its mitigation
The failure mode is the #1917 pattern repeating: tune by hand, get a good result, close the issue, never build the gate. That is not hypothetical — it is what happened, and
live_stage1_golden_prompts.rsstill carries the header comment saying the snapshot tests are "not yet written."Mitigation is in #2120's acceptance criteria rather than in the ordering: every tuned prompt is checked in as a #2118 case file. The work is durable whether or not #2119 lands promptly, and #2119 becomes the step that converts those case files into a production contract rather than the only thing standing between tuning and lost work.
For reading the current assembled prompt while deciding what to cut,
NODESPACE_PROMPT_DUMPalready dumps it at the native-path chokepoint — the gate is not needed for that.Adjacent, not blocking
friendly_namein schema tools) — independent_seedreconciliation, though evals still need a daemon rebuild since guidance is compiled inmalibio commentedon Aug 17, 2026
Revised sequencing: #2118 → #2122 → #2119 → #2120 → #1977
Supersedes the earlier sequencing comment. The chain gained a step and #2120 was rescoped, because the previous plan had no issue that actually produced the golden corpus — #2119 would have asserted against goldens nobody authored.
The split that matters
#2120 previously mixed what should the prompt say with edit the files that emit it. That is the exact conflation this effort exists to remove — a failure could be a bad prompt or bad plumbing, indistinguishable.
Now:
A failure in #2122 is a prompt failure. A failure in #2120 is a plumbing failure. By construction.
Why #2119 moved ahead of #2120
The gate is what defines #2120's done condition. "Make plumbing match" has no test without it.
The governing requirement across all of them
Goldens must be production-shaped: same blocks, same structure, same tool surface as what NodeSpace actually sends — with the content of those blocks being what is tuned. A minimal hand-authored prompt is a useful control but not a target; scenario 6 is the cautionary case, at 3/3 hand-authored vs 0/8 in production for the same task.
malibio commentedon Aug 21, 2026
Baseline (worktree issue-1977-dev-workflow-eval): Frontend 4794 passed / 227 files, 0 failed. (An initial run showed 2 known load-flaky perf benchmarks; a clean re-run passed 227/227.)
malibio commentedon Aug 21, 2026
Scope decision: re-theme in place, do not add a second fixture
AC 2 asks for a documented decision on replace-vs-add. Recommendation: replace the domain, keep every mechanic — one fixture, not two. Proceeding on this unless redirected.
Reasoning:
noExtraTypes,minProperties,noRetry+minCalls,toolSequence) are properties of theExpectationmodel, not of the album/laser-cutter vocabulary. Re-theming keeps every one of them, and every scenarioid, so baseline diffing still joins.What changes
Scenario ids and expectations stay byte-identical — only prompts and the chain's entities are re-themed, so no baseline row is orphaned:
tasktype, still exercising defined-but-unsetdue_date/status.What is genuinely new
The issue's strongest point is relationship traversal, which the matrix has never covered at all. Adding one new group (new ids, so nothing existing is disturbed) for
create_relationship+get_related_nodes— "which decision constrains this" traversal. This needs a newExpectationkind only if the existing ones cannot express it; I expecttoolSequencecovers it, and will not invent a parallel scoring mechanism (AC 4).Winnability constraint I am holding to
Every re-themed prompt has to be answerable by the schema the chain itself builds (the trap documented on scenario 9 and in #1846), and no prompt may share a 5-word run with seeded guidance (
guidance_is_not_contaminated_by_eval_prompts). Dev vocabulary raises that risk materially — seeded skill descriptions already say "Spec, ADR, Ticket" — so wording will be checked against the guard, not assumed.Re-theme the agent matrix around NodeSpace's dev-workflow domain (#2232)