Repository navigation
Re-baseline eval:agent under the corrected graph capture #3789
Description
Overview
Re-record the eval:agent baseline under the corrected graph capture. Results recorded while the capture was blind are not comparable with it.
Problem Statement
The shared graph capture (scripts/eval/graph.ts) was found to be vacuous for the end-state-graded fixtures:
- It read schema ids from a
nodesarray thatschema listdoes not emit (it emitsschemas), so with the fixture's emptytypeslist no node type was ever queried. - It pulled 200 nodes per type, while a fresh database seeds more than 300
textnodes and the daemon clamps a query to 500 rows without saying so, so a written note fell off the end of the list.
While it was blind, every "nothing was written" clause passed over a write and no "was created" clause could pass. Both are fixed in the eval:tasks PR, and the runner now voids a run on a blind capture.
When the CLI began emitting schemas is not established, so which recorded eval:agent baselines were measured blind is not known. Any recorded number that rests on an end-state clause, and any document that cites one, should be treated as unverified until this is done.
Proposed Solution
- Establish from history when
schema listbegan emittingschemas, and so which recorded baselines are affected. - Run
bun run eval:agentfor 3 reps on currentmainwith the contention check done first, and record it as the new baseline. - Compare the end-state verdicts with the last recorded baseline and list the scenarios whose result changed.
- Find the documents that cite an
eval:agentscore and mark any that predate the fix.
Acceptance Criteria
- The date or commit the capture became blind is stated
- A 3-rep
eval:agentbaseline is recorded on the corrected capture, with the contention check noted - The scenarios whose verdict changed are listed with the reason
- Documents that cite a pre-fix
eval:agentscore are identified
Related Issues
- the
eval:tasksPR that fixed the capture
Activity
malibio commentedon Oct 13, 2026
Findings. (1) Blind window: schema list began emitting 'schemas' (not 'nodes') in f2835dd (2026-10-02, #3490); fixed in cc1dba3 (2026-10-13). The agent-matrix fixture has types: [] so every end-state clause in that window was vacuous. The 200-node truncation dates from the capture's introduction, 8cf57e1 (2026-08-22). No eval:agent baseline was recorded in the window: the only committed one (2026-07-30) predates graph scoring and is trajectory-scored, so it is not comparable. (2) New baseline, 3 reps, a6bb9aa, e4b, purged isolated daemon restarted between reps: pass^3 13/20, pass^1 12.67/18, 1 flipped; 0 Metal OOM in the daemon logs; no cargo/rustc/nextest/model process before launch. A first run was voided: its first daemon was never stopped by my between-runs hook, two daemons held models, 26 Metal OOMs. (3) Verdict changes vs July on the 12 shared scenarios: 8a and 8e fail->pass 3/3; 8b and 8c pass->fail 0/3 (nothing was created: 8b emits create_schema,update_schema,create_schema and times out at 240s; 8c calls no tool); 6 still fails, 1/3. 1-5, 7, 8d pass in both. Scenarios 9-13 are new since July. 11c/11c2/11d and 12d are excluded when setup 11b/12c does not establish state. (4) Docs citing scores: ADR-056 (10/12, 11/12) and agent-eval.md (6/12, 9/12, 8/12) are July trajectory-scored numbers, pre-window; agent-eval.md now says so. (5) Separate defect found: the runner's guidance provenance read-back looks for _seed.key/_seed.version, which nothing writes now, so provenance.guidance is empty and the rep-to-rep guidance drift check is inert.
Re-baseline eval:agent on the corrected graph capture (closes #3789)