Make the agent eval diagnostic before recording a baseline; revisit ADR-056's Scenario 5 claim

Overview

The agent eval cannot currently distinguish "the model failed" from "the harness failed" — both render as a low score. Until it can, re-running it does not produce a baseline worth recording. This issue makes the eval diagnostic first, then records the honest baseline.

Supersedes this issue's original framing ("re-run and record the number"), which assumed a single clean run would yield a usable baseline. Contamination was one problem with the old figure; it was not the only one.

Why a re-run alone is not enough

1. The apparatus is less reliable than the effects being measured. Three separate false results are on record in this system: the eval assertion that could not see a failed schema creation, the documented false pass in scripts/results-e4b.json, and — during the prompting spike — six harness bugs, each of which produced a clean, plausible number and each of which reversed a conclusion. None was detectable from aggregate scores; every one was found by reading raw model output.

2. Pass/fail over 12 scenarios is too coarse. Run-to-run variance on identical code has been observed at 10/9/8. A two-scenario effect sits inside that noise. Per-rule boolean scoring — several independent checks per scenario instead of one bit — resolved substantially smaller effects during the spike, because each scenario contributes several observations rather than one.

3. Stale guidance is measured silently. Guidance lives in compiled-in constants seeded into the DB on first run, and seeding skips by node type — so an eval run measures whatever was seeded first unless the DB is purged and the daemon rebuilt. Fixing seed delivery removes this hazard at the source; until then the eval must verify and report which guidance it actually measured.

4. The configuration under test is about to change. ADR-064 deletes most of the resident prose, rewrites skill descriptions for retrieval, and changes at least one tool's return contract. A baseline recorded now describes a system that will not exist, and would then be cited as a regression reference it cannot support.

Corrections to this issue's earlier content

It instructed re-running scripts/aichat-matrix.ts. That file does not exist; the fixture is scripts/eval/fixtures/agent-matrix.ts.
The 11/12 figure it flags as contaminated is also unusable for the reasons above, so "re-record it cleanly" understates the work.

Work

Make the eval diagnostic

Retain raw model output for every scored generation, to a JSONL trace. This is the single highest-value change: every false result on record was diagnosable in one look at one raw generation and invisible in aggregates
Fail loudly on impossible uniformity. A 0% or 100% rate across every arm or scenario is a harness signature, not a result. Abort with a diagnostic rather than printing a plausible table
Exclude degenerate empty generations from the denominator. The locked model intermittently opens a turn and emits nothing; the agent loop documents this. Scoring it as a failure silently deflates every cell
Report which guidance was measured — verify seeding actually applied, and surface it in the run output, so a stale-guidance run cannot be mistaken for a real one
Score per-rule where possible, not only per-scenario pass/fail, so effects smaller than the current noise floor are resolvable
Preserve the existing per-scenario failure reason in whatever is committed

Then record the baseline

Run against the ADR-056 locked model once the ADR-064 changes have landed, and record that as the baseline
Report enough repetitions to characterize variance, not a single number
Watch scenarios 4 / 8c / 8d specifically — INSTANCE vs TYPE was the guidance edit most likely to cost a small model comprehension, since it lost its concrete example
Check whether node_type="widget" appears in output. The Node Creation skill's example uses deliberately inert placeholders; if the model copies them, the fix is a different inert domain, not restoring the old example

Independent of the above

Revisit ADR-056's Scenario 5 claim. The ADR cites that scenario passing as evidence that collapsing execute_query into search_nodes removed tool ambiguity. The structural argument stands on its own, but the scenario's pass was not independent confirmation — the guidance contained the prompt and the exact call. Update the ADR to rest on the structural argument, or re-derive the evidence from the honest run

Notes

Contamination itself is already guarded: guidance_is_not_contaminated_by_eval_prompts parses prompts out of the eval fixtures at test time and fails the build if any reappears across agent_guidance.rs, skill_rules.rs, every seeded skill's markdown_content, and the seeded base prompt nodes. Rewording eval prompts is therefore safe.

A reference harness demonstrating the diagnostic practices above — raw-output retention, uniformity assertions, arm-distinctness checks — is at ~/nodespace/spike-1802-harness/ (not committed; its README documents its own defects).

Architecture Impact

Several issues in the ADR-064 workstream carry acceptance criteria of the form "no regression against the matrix." Those are unmeasurable until this lands, so this issue gates the verification of that work even though it does not block the implementation.

AC-5 is done — completed in PR #1763 (805a60e), so this issue is now only the ADR-056 follow-up.

Honest baseline: 9/12 on gemma-4-e4b-q4km, vs 11/12 recorded under contamination. Committed to scripts/results-e4b.json with provenance (model, date, guidance commit, host memory).

Two premises in this issue and #1760 were wrong, found by actually running it:

scripts/results-e4b.json did not exist in the repo. The contaminated 11/12 lived only in a scratch file — there was nothing to overwrite. It exists now.
A DB purge is mandatory, and was written down nowhere. Seeding skips nodes that already exist, so a reused database keeps the OLD prompt/skill nodes and the run silently measures guidance you already changed. The daemon binary must be rebuilt for the same reason — guidance is compiled-in string constants. My first two runs were invalid this way: a stale release binary reseeded the pre-fix skill_pipeline.rs example, which I caught only by querying the seeded DB for the old text and finding it still there.

Remaining scope for this issue: revisit ADR-056's Scenario 5 claim. Scenario 5 still passes on decontaminated guidance, so the tool-collapse conclusion survives — but the ADR should rest on the structural argument, since the scenario's original pass was not independent evidence.

The three failures are genuine agent defects, not rewording artifacts:

4. Instance creation — calls create_schema for an instance request, then errors. This starves scenario 6, which has no node to update.
6. Update — cascades from 4.
8c — retries create_node after an error (violates the no-blind-retry rule).

Scenario 8a/8b needed one more rewording pass: my first attempt ("Set up somewhere to note down albums") dropped the tracking vocabulary the DATABASE=SCHEMA rule keys on, and the model asked for confirmation instead of acting. A control run confirmed the ORIGINAL wording still fires create_schema on decontaminated guidance — so that was my artifact, not a finding. Now "Start tracking albums...".

Failure ownership from the 9/12 honest baseline, for anyone picking this up:

Scenario 6 (update) → resolve_query must return a resolved node, not query fragments #1637 (already open, assigned). Agent guidance contains verbatim aichat-matrix prompts — the eval is measuring recall of planted examples, not generalization #1760 removed the wrong hardcoded node_type literal that issue's analysis identified; the decomposition problem remains. Not cleanly measurable until Agent emits malformed tool calls: title_template referencing undefined fields, and calls with empty arguments #1775 lands, since Scenario 4 fails to create the node Scenario 6 needs.
Scenarios 4 + 8c → Agent emits malformed tool calls: title_template referencing undefined fields, and calls with empty arguments #1775 (new): instance request misrouted to create_schema, and create_node retried after an error.

Remaining scope for THIS issue is just the ADR-056 Scenario 5 review.

Address review: fail closed on the destructive check, abort on mid-ru…

Shared eval runner with a preflight gate (closes #1759)

Address review: fail closed on the destructive check, abort on mid-ru…

Shared eval runner with a preflight gate: stop scoring scenarios when…

Status: the ADR-056 item is already done; the run itself is blocked on environment isolation

Last checkbox (revisit ADR-056's Scenario 5 claim) — done, in nodespace-docs 29b3580. The ADR no longer cites the scenario passing as evidence. It now reads:

The argument rests on construction rather than measurement: with a single query tool the ambiguity cannot arise, because the tool it was ambiguous with no longer exists.

Scenario 6's entry was corrected in the same commit to name the actual cause (wrong type reference → exact-match lookup miss → silent empty resolution → fabricated completion) rather than describing symptoms, and the paragraph states that scores measured under contamination describe the harness as much as the model. Re-deriving Scenario 5 evidence from an honest run is still worthwhile, but the ADR no longer depends on it.

The re-run attempt, and what stopped it

Attempted against 989ef079 with the model loaded. The new preflight gate refused to score, correctly:

The daemon had been launched with NODESPACED_DB_PATH=/tmp/nodespaced-test/nodespace.db, which does not isolate — the registry at ~/.nodespace/databases.toml pins default_database to the real path and overrides it. The gate's remedy (NODESPACE_HOME, then confirm via grep served_db_path) is correct and should be the documented pattern.

This is worth recording because the trap is silent and pre-dates the gate: earlier runs on this machine used the same NODESPACED_DB_PATH invocation and did write eval artifacts into the real database — invoice_tracker, book_reading_list, contact_profile schemas plus their description children, and instances "Invoice for $500 due next Friday" (×2), "Dune by Frank Herbert", "Jane Doe", along with the per-group chat nodes. Anyone reproducing on a machine with an existing ~/.nodespace should check for those before trusting a run, and clean them up.

scripts/aichat-1329-trial.md still documents the non-isolating NODESPACED_DB_PATH invocation in its reproduce steps. That is the source of the pattern and should be corrected (or removed, since #1759 already proposes moving eval setup documentation into nodespace-docs).

Confirmed working

#1758's fix holds on the machine that previously floored to 4096 with no override:

Chat context fits configured window: n_ctx=32768 total_ram=17.2GB weights=5.3GB

So the run is unblocked as soon as it is launched under NODESPACE_HOME.

Correction to my previous comment — most of it was written against stale information, and a code block was dropped in posting. Superseding it:

The 9/12 honest baseline already answers what I was attempting. My re-run was redundant; I had not seen the results above before starting. Disregard the framing that the run was still outstanding.

Still accurate and worth keeping:

The preflight gate refused to score because the daemon was serving the real user database. The dropped block read:

── agent-matrix: ENVIRONMENT NOT USABLE ──
The daemon is serving the REAL user database (~/.nodespace/database/nodespace.db).
  This eval creates schemas, chat nodes, and instances; running it here would
  write test data into live user data.

NODESPACED_DB_PATH does not isolate — ~/.nodespace/databases.toml pins default_database to the real path and overrides it. The gate's remedy (NODESPACE_HOME, confirm via grep served_db_path) is correct.

This trap pre-dates the gate, and earlier runs on this machine did write eval artifacts into the real database: invoice_tracker, book_reading_list, contact_profile schemas plus description children, and instances "Invoice for $500 due next Friday" (×2), "Dune by Frank Herbert", "Jane Doe", plus per-group chat nodes. Anyone reproducing against an existing ~/.nodespace should check for those.

This pairs with the DB-purge requirement already noted above: a reused database keeps old seeded prompt/skill nodes and may carry prior eval artifacts. scripts/aichat-1329-trial.md still documents the non-isolating NODESPACED_DB_PATH invocation — worth correcting or removing.

Also confirmed: #1758's fix holds with no override on the machine that previously floored to 4096 — Chat context fits configured window: n_ctx=32768 total_ram=17.2GB weights=5.3GB.

On the remaining ADR-056 scope: already done in nodespace-docs 29b3580, pushed before this comment. Scenario 5's entry now rests on the structural argument ("with a single query tool the ambiguity cannot arise, because the tool it was ambiguous with no longer exists") rather than citing the scenario's pass. Scenario 6's entry was corrected in the same commit, and the section states that contaminated scores describe the harness as much as the model. Given Scenario 5 still passes at 9/12, that conclusion is now independently supported — so this issue's last checkbox looks complete.

Update following #1815 and #1828

Sequencing narrows to one issue. At filing, this waited on "the ADR-064 changes" as a set. Since then #1699, #1637, #1820, #1816, #1810, and #1815 have all landed. #1813 (resident prompt reduction) is now the only remaining blocker on recording a real baseline — everything else the eval would need to measure against is already on main.

#1815 adds a dimension this issue didn't originally scope: the two-stage routing path itself. The eval currently plans to score per-rule action outcomes. It should also characterize:

Whether Stage 1 routes (route_query) or clarifies (route_clarify) for each scenario, not just whether the final action succeeded
Tool-call round count per scenario, so the ~13% latency overhead Implement ADR-038 Stage-1 and Stage-2 gates; shipped loop is the rejected single-turn pull #1815 measured in isolation (packages/agent/tests/routing_latency.rs) can be cross-checked against real matrix scenarios, not just synthetic prompts
Whether Stage-2 candidate injection actually fired when expected — a scenario that silently skips routing and falls through to baseline behavior would otherwise look identical to one that routed correctly, in a pass/fail score

#1815 caught a bug that is a direct instance of what this issue exists to prevent. Its PR (#1826) found looks_like_narrated_tool_call only detected tool_name(...) syntax — a model emitting the JSON array form as text slipped past it, so a routed turn could silently do no work while the loop reported nothing. That's exactly "harness failure rendering as model failure." It's fixed upstream in the agent loop now, but the eval's own tool-call detection (wherever it parses model output independent of the loop) should be checked against both call-syntax forms as part of the diagnostic work below, so a regression there doesn't silently corrupt a future baseline the same way.

#1828 is a related but out-of-scope note, not new work here. It found mistral:7b suppresses tool-calling on any injected candidate block. Confined to a non-shipped model; no action needed in this issue, mentioned only so a future reader doesn't wonder whether it's related.

Added acceptance criteria

Baseline recorded once Reduce the resident prompt to identity and policy (measured 50% vs 73% for identity-only) #1813 lands (not "ADR-064" generally — that set is otherwise complete)
Per-scenario reporting includes whether Stage 1 routed or clarified, and the tool-call round count, not just the final pass/fail
The eval's own tool-call parsing is verified against both tool_name(...) and JSON-array call syntax, matching the fix in Implement ADR-038 Stage-1 and Stage-2 gates; replace the rejected single-turn pull #1826

Scoping note before starting implementation:

Issue #1764 has two phases: (A) make the eval diagnostic — raw-output JSONL trace, uniformity guard, empty-generation exclusion from the denominator, guidance-provenance reporting, per-rule scoring (already landed via #1778/#1805) — plus the ADR-056 Scenario 5 doc revisit; and (B) record the actual baseline.

Phase B is gated on the rest of the ADR-064 workstream landing: #1811 (delete inert READ-THEN-WRITE rule) and #1800 (delete create_schema description-inference route) are both still open. The issue itself says recording a baseline now would describe a system that's about to change and would then be cited as a regression reference it can't support.

Proceeding with Phase A now (diagnostic harness improvements + ADR-056 doc fix), and leaving baseline recording until #1811/#1800 close. Flagging here in case that's not the intended scope.

Correct on the phasing, and one of the two Phase-B gates is already clear.

#1811 is done — closed just now. It was satisfied as a side effect of #1820's routing cleanup (the READ-THEN-WRITE rule was in fact routing prose, so #1820 deleted it along with the rest), not via its own PR, which is why it was still showing open. Verified on current main: the rule is gone from TOOL_STRATEGY_RULES, and #1820's own test asserts it stays gone.

#1800 is genuinely still open and does gate Phase B — it's a real, separate defect (the description-parameter double-meaning / 27% silent-wrong-type rate) that nothing else in this workstream has touched.

So: proceed with Phase A now, as planned. Phase B is gated on #1800 alone, not #1800+#1811. Re-check #1800's status before recording the baseline — if it's still open when Phase A finishes, Phase B waits; if it's landed by then, proceed straight to the baseline run.

This was the intended scope — good catch flagging it rather than guessing.

Baseline: 4111/4112 tests passed (170/171 files) — 1 failure is the known machine-load-flaky architecture-benchmarks.test.ts (scaling-factor threshold), not a regression. Starting Phase A implementation.

Phase B is now unblocked. #1800 (the last remaining gate per the earlier scoping note) closed. Both items this issue's Phase B was waiting on are done — #1811 (already satisfied via #1820) and now #1800.

If Phase A's diagnostic work is done or close, proceed straight to recording the baseline. Nothing else in the ADR-064 workstream is outstanding to wait on.

Make the agent eval diagnostic: raw-output trace, uniformity guard, e…

Fix multiline/embedded-marker corruption in raw-output log scraping (…

Rename seed_prompt_nodes to seed_agent_guidance_nodes; fix eval's sta…

Phase B complete: honest baseline recorded

#1800 closed (merged as #1833), clearing the last ADR-064 gate. Rebased the diagnostic-harness branch onto latest main, then ran the actual baseline against a purged, freshly-seeded isolated daemon on commit 57bf1088 (Core Identity, Workspace Context Template, Tool Strategy Guide, Response Formatting Rules, Tool Call Formatting — 5 agent-guidance nodes; 8 skills — versions recorded in provenance.guidance).

Headline: 9/12 (median of three runs: 6/12, 9/12, 8/12 — mean 7.7/12, ~64%), consistent with this suite's previously-documented 10/9/8 variance band. Committed to scripts/results-e4b.json (run 2, other two summarized inline) with scripts/results-e4b.trace.jsonl alongside it — raw model output for every scored turn, so any finding below is re-verifiable without re-running.

Checklist

Run against the locked model once ADR-064 landed — done, commit 57bf1088/38b9ccbb
Report enough repetitions to characterize variance — 3 agent-matrix runs (6/9/8), 2 routing runs (10/15, 10/15)
Watch scenarios 4/8c/8d (INSTANCE vs TYPE) — passed 2/3, 3/3, 1/3 respectively. No systematic regression from the guidance edit that dropped the concrete example; 8d's 1/3 is worth watching in a larger future run but isn't conclusive on its own
Check for node_type="widget" leakage — zero occurrences across all three runs' raw traces and the full daemon log. The placeholder rewrite is holding
ADR-056 Scenario 5 — already resolved in an earlier session (nodespace-docs 29b3580); the ADR now rests on the structural argument rather than citing the scenario's pass

What the raw traces actually show

Scenario 6 (Update via indirect reference) failed in all three runs. The model asks the user for a node ID/URI instead of calling resolve_query, even though resolve_query is confirmed present in the offered tool list every time (verified via the daemon log's "system prompt and tools prepared" line — this isn't a routing failure, the tool was offered and the model just didn't reach for it). Owned by resolve_query must return a resolved node, not query fragments #1637.
create_schema JSON-truncation errors recurred across scenarios 3/8a/8b at 2/3 frequency each. The raw log shows genuinely malformed JSON mid-generation ("EOF while parsing a value at line 1 column 192") on schemas with several fields — a real generation defect on complex nested arguments, distinct from the Delete the create_schema description-inference route: an undocumented second meaning for the description parameter, with a 27% silent-wrong-type rate #1800/coreValues class of prose-contradiction bug this suite previously caught.
Zero empty-generation exclusions, no uniformity-guard trips in any run — nothing here reads as a harness artifact.
Routing eval (not committed as a separate file, noted in caveats): 10/15 twice, identical failure set both times — systematic rather than noise. At least one (ambiguous-client-contacts) looks like a scoring-heuristic gap: the model's reply does list disambiguating options in prose but doesn't match the isClarification keyword check.

Bug found while running this

seed_prompt_nodes seeds nodes with root_node_type: "agent-guidance", not "prompt" — renamed under ADR-064/#1699 but the function name was never updated. This meant scripts/eval/preflight.ts's guidance-provenance check was silently checking a node type that no longer exists (zero results, correctly recorded, but misleadingly so — it looked like guidance verification was broken rather than checking the wrong type). Fixed both the function name (seed_agent_guidance_nodes) and the eval's type list, in commit 38b9ccbb.

All work is on PR #1831.

Strong close — real diagnostic infrastructure (raw traces, provenance-tagged guidance versions, a caught bug in the eval's own plumbing), not a rubber-stamp number. One correction and two new issues to file from what the traces surfaced.

Correction: Scenario 6's failure is not owned by #1637

#1637 is already closed. Its fix was the handoff — resolve_query returning a resolved node instead of query fragments the model had to route itself, measured 100% vs 0%. That shipped.

What you found here is a different failure on the same tool: resolve_query is confirmed present and offered every time, and the model doesn't reach for it at all — it asks the user for a node ID instead. That's an invocation gap, not a handoff gap. #1637's fix can't own this because it solved a different mechanism on the same tool; closing #1637 doesn't reopen to "own" a new bug against it. This needs its own issue.

Two things to file

1. resolve_query invocation gap (Scenario 6, 3/3 runs). The model has the tool, doesn't use it, asks the user instead. Worth checking against #1815's Stage-2 candidate-injection design — if resolve_query isn't in the routed tool surface for this scenario's matched skill, that would explain non-invocation cleanly and point at a skill tool_whitelist gap rather than a model-judgment failure. Your raw traces + daemon log should show which tools were actually offered in the failing turns, not just that resolve_query exists somewhere in the registry.

2. create_schema JSON-truncation on complex nested arguments (3/8a/8b, 2/3 frequency each). Genuinely different bug class from #1809/#1800 — not a prose/schema contradiction, a real generation defect ("EOF while parsing a value") on schemas with several fields. Worth its own issue with the raw trace lines cited, since this is a new failure mode this workstream hasn't characterized yet.

Worth a look, not urgent

Routing eval's isClarification scoring gap (ambiguous-client-contacts) — the model's answer is arguably correct (lists disambiguating options in prose) but the keyword check doesn't recognize it. Sounds like a harness/scoring issue, not a model failure — matches this workstream's recurring "is it the model or the harness" question. Worth a quick look before trusting the 10/15 routing number as a model-quality signal rather than partially a scoring-gap signal.
8d's 1/3 — you flagged it as not conclusive on its own, agreed; worth another eyeball if Re-measure one-hop related-schema enrichment at higher power (0%->33% at n=9 in #1817) #1834's related-schema re-measurement or any future run touches the same scenario, but not worth a standalone issue at this n.

Want me to file the two issues above, or are you filing them from the raw traces directly?

Correction to the finding above: scenario 6's failure is NOT #1637 (closed) — I misattributed it. #1637 was about resolve_query's return contract once called, and its own closing evidence cites 100% success on this exact scenario phrasing ("the 2400 one...") once the model calls the tool. That fix is confirmed present and working in the code this baseline measured.

What this baseline actually found is a different failure: the model doesn't call resolve_query at all in 2 of the 3 runs (run 1's failure is a cascade from scenario 4 never creating a node to reference — a different, reasonable clarifying question, not this pattern). In runs 2 and 3 the model asks the user directly for a node ID/URI instead, despite resolve_query being confirmed present in the offered tool list and the guidance rule instructing its use for indirect references being present verbatim in skill_pipeline.rs.

Filed as its own issue: #1835. scripts/results-e4b.json's caveats and the trace file are corrected in commit 100bd1bb (also fixes an unrelated stale doc-comment reference caught in PR re-review).

Make the agent eval diagnostic before recording a baseline; revisit A…

Navigation Menu

Repository navigation

Make the agent eval diagnostic before recording a baseline; revisit ADR-056's Scenario 5 claim #1764

Overview

Why a re-run alone is not enough

Corrections to this issue's earlier content

Work

Make the eval diagnostic

Then record the baseline

Independent of the above

Notes

Architecture Impact

Status: the ADR-056 item is already done; the run itself is blocked on environment isolation

The re-run attempt, and what stopped it

Confirmed working

Update following #1815 and #1828

Added acceptance criteria

Phase B complete: honest baseline recorded

Checklist

What the raw traces actually show

Bug found while running this

Correction: Scenario 6's failure is not owned by #1637

Two things to file

Worth a look, not urgent

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make the agent eval diagnostic before recording a baseline; revisit ADR-056's Scenario 5 claim #1764

Description

Overview

Why a re-run alone is not enough

Corrections to this issue's earlier content

Work

Make the eval diagnostic

Then record the baseline

Independent of the above

Notes

Architecture Impact

Activity

malibio commented on Jul 28, 2026

malibio commented on Jul 28, 2026

malibio commented on Jul 28, 2026

Status: the ADR-056 item is already done; the run itself is blocked on environment isolation

The re-run attempt, and what stopped it

Confirmed working

malibio commented on Jul 28, 2026

malibio commented on Jul 30, 2026

Update following #1815 and #1828

Added acceptance criteria

malibio commented on Jul 30, 2026

malibio commented on Jul 30, 2026

malibio commented on Jul 30, 2026

malibio commented on Jul 30, 2026

malibio commented on Jul 31, 2026

Phase B complete: honest baseline recorded

Checklist

What the raw traces actually show

Bug found while running this

malibio commented on Jul 31, 2026

Correction: Scenario 6's failure is not owned by #1637

Two things to file

Worth a look, not urgent

malibio commented on Jul 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions