You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Agent eval scenarios for Creating a Task and Answering a Question, on a fixture current with the entity tier #3763
The agent eval fixture (packages/agent/goldens) predates the entity tier and the workflow procedures, so it exercises none of the skills shipped by #3722 and #3719 (Creating a Task, Answering a Question, Asking the User, Splitting a Plan into Tasks). #3722 closed with its last acceptance criterion unmet: "the agent evals on the scenarios the change touches".
Problem Statement
The confusion matrix and the live retrieval guards measure which skill is found, not whether an agent that has it follows it: checks before creating, a bug reproduced before it is shaped, a question put to the user rather than answered by the agent, a declined request left as an accepted decision.
Without scenarios, a change to either procedure's wording has no behavioural check.
Proposed Solution
Add eval scenarios for the new procedures and bring the fixture up to the current model, then record a baseline:
Creating a Task: a bug report is searched for (built and declined), reproduced or reported as not reproducible, then created with a brief, a checklist and its project, and is in Ready tasks.
A declined request: an existing accepted decision that declines it is found, and no task is created.
A question that stops work: a question task is created assigned to the person, blocking the stopped task, which keeps its status and assignee.
Answering a Question: the agent puts the question to the user and records the answer as theirs; it never answers it itself.
Acceptance Criteria
The fixture seeds the entity tier and the shipped workflow queues and skills
One scenario per bullet above, with a pass condition that does not depend on wording
A baseline run recorded (3 reps on these scenarios, then one full smoke run), with the contention check from the eval runbook done first
Any scenario that fails is either fixed in the procedure text or recorded as a known gap with the reason
Activity
malibio commentedon Oct 13, 2026
Frontend: 5899 passed (326 files)
Void an eval run on a blind or failed graph capture, and tolerate und…
Address review: page the snapshot queries past the daemon's 500-row c…
Add the tasks eval: creating a task and answering a question, graded …
Fix the eval snapshot reading no schemas and truncating at 200 nodes (c…
Void an eval run on a blind or failed graph capture, and tolerate und…
Address review: page the snapshot queries past the daemon's 500-row c…
Reset the reported done task through its checklist so the tasks eval …
Fix the tasks eval seed aborting a run once an agent changes the done…