Skip to content

fix(skill-evals): 32 eval steps without a user-prompt-template.md get security-issue-import's prompt #1492

Closed

Description

@dpol1

What

When a fixtures directory has no user-prompt-template.md, the runner falls back to USER_PROMPT_TEMPLATE (tools/skill-evals/src/skill_evals/runner.py:194-197). That default (lines 96-110) is the security-issue-import template:

## Existing open trackers (corpus)
{corpus}
## Reporter roster (existing trackers mapped to reporter email)
{roster}
## Incoming report
{report}
Apply the semantic sweep and reporter-identity check. Return JSON only.

33 step directories under tools/skill-evals/evals/ have no template. One is security-issue-import/step-2a-semantic-sweep, the step the default was written for. The other 32 belong to other skills:

  • every step of release-announce-draft, release-archive-sweep, release-audit-report, release-keys-sync, release-prepare, release-rc-cut, release-verify-rc and release-vote-tally (30 steps);
  • reviewer-routing/step-0-preflight;
  • non-asf-profile-smoke/step-release-backend-preflight.

For those steps the model gets an empty "Existing open trackers" section, an empty "Reporter roster" section, the case input under "Incoming report", and an instruction to run a semantic sweep and a reporter-identity check that has nothing to do with the step under test.

Why it matters

The system prompt is correct: it carries the step extracted from the skill. The user prompt is not: it asks for an import-skill task. A PASS on these steps says less than it appears to, and a FAIL may come from the prompt rather than the skill. AGENTS.md makes the eval suite the check for every skill change, and almost all of the affected steps are in the release family. This does not mean these evals fail; I have not measured how many verdicts change with a neutral prompt.

Suggested fix

  1. Make the default template neutral: the {report} slot and "Return JSON only.", nothing else. str.format() ignores the unused corpus / roster arguments, so the call site does not change.
  2. Give security-issue-import/step-2a-semantic-sweep its own user-prompt-template.md holding today's default, so its prompt stays the same.
  3. Re-run the 32 affected steps and note any verdict that moves.

The alternative is to make the template mandatory and have the runner fail when it is missing, which means 32 new files.

Found via

Reading the runner's prompt assembly while mapping the eval harness. The 33 directories are those with a step-config.json and no user-prompt-template.md on main (37670575).

Activity

  1. added a commit that references this issue on Oct 5, 2026
    215067f
  2. added 2 commits that reference this issue on Oct 5, 2026
    cad4dbb
    78032a9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions