Skip to content

Agent eval scenarios for Creating a Task and Answering a Question, on a fixture current with the entity tier #3763

Closed

Description

@malibio

Overview

The agent eval fixture (packages/agent/goldens) predates the entity tier and the workflow procedures, so it exercises none of the skills shipped by #3722 and #3719 (Creating a Task, Answering a Question, Asking the User, Splitting a Plan into Tasks). #3722 closed with its last acceptance criterion unmet: "the agent evals on the scenarios the change touches".

Problem Statement

  • The confusion matrix and the live retrieval guards measure which skill is found, not whether an agent that has it follows it: checks before creating, a bug reproduced before it is shaped, a question put to the user rather than answered by the agent, a declined request left as an accepted decision.
  • Without scenarios, a change to either procedure's wording has no behavioural check.

Proposed Solution

Add eval scenarios for the new procedures and bring the fixture up to the current model, then record a baseline:

  • Creating a Task: a bug report is searched for (built and declined), reproduced or reported as not reproducible, then created with a brief, a checklist and its project, and is in Ready tasks.
  • A declined request: an existing accepted decision that declines it is found, and no task is created.
  • A question that stops work: a question task is created assigned to the person, blocking the stopped task, which keeps its status and assignee.
  • Answering a Question: the agent puts the question to the user and records the answer as theirs; it never answers it itself.

Acceptance Criteria

  • The fixture seeds the entity tier and the shipped workflow queues and skills
  • One scenario per bullet above, with a pass condition that does not depend on wording
  • A baseline run recorded (3 reps on these scenarios, then one full smoke run), with the contention check from the eval runbook done first
  • Any scenario that fails is either fixed in the procedure text or recorded as a known gap with the reason

Related Issues

Activity

  1. malibio commented on Oct 13, 2026

    @malibio
    CollaboratorAuthor

    Frontend: 5899 passed (326 files)

  2. self-assigned this
    on Oct 13, 2026
  3. added 6 commits that reference this issue on Oct 13, 2026
    8e5bf74
    86fc712
    ca2248a
    f59eb29
    f79cbdc
    b626ceb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions