Repository navigation
fix(skill-evals): 32 eval steps without a user-prompt-template.md get security-issue-import's prompt #1492
Description
What
When a fixtures directory has no user-prompt-template.md, the runner falls back to USER_PROMPT_TEMPLATE (tools/skill-evals/src/skill_evals/runner.py:194-197). That default (lines 96-110) is the security-issue-import template:
## Existing open trackers (corpus)
{corpus}
## Reporter roster (existing trackers mapped to reporter email)
{roster}
## Incoming report
{report}
Apply the semantic sweep and reporter-identity check. Return JSON only.
33 step directories under tools/skill-evals/evals/ have no template. One is security-issue-import/step-2a-semantic-sweep, the step the default was written for. The other 32 belong to other skills:
- every step of
release-announce-draft,release-archive-sweep,release-audit-report,release-keys-sync,release-prepare,release-rc-cut,release-verify-rcandrelease-vote-tally(30 steps); reviewer-routing/step-0-preflight;non-asf-profile-smoke/step-release-backend-preflight.
For those steps the model gets an empty "Existing open trackers" section, an empty "Reporter roster" section, the case input under "Incoming report", and an instruction to run a semantic sweep and a reporter-identity check that has nothing to do with the step under test.
Why it matters
The system prompt is correct: it carries the step extracted from the skill. The user prompt is not: it asks for an import-skill task. A PASS on these steps says less than it appears to, and a FAIL may come from the prompt rather than the skill. AGENTS.md makes the eval suite the check for every skill change, and almost all of the affected steps are in the release family. This does not mean these evals fail; I have not measured how many verdicts change with a neutral prompt.
Suggested fix
- Make the default template neutral: the
{report}slot and "Return JSON only.", nothing else.str.format()ignores the unusedcorpus/rosterarguments, so the call site does not change. - Give
security-issue-import/step-2a-semantic-sweepits ownuser-prompt-template.mdholding today's default, so its prompt stays the same. - Re-run the 32 affected steps and note any verdict that moves.
The alternative is to make the template mandatory and have the runner fail when it is missing, which means 32 new files.
Found via
Reading the runner's prompt assembly while mapping the eval harness. The 33 directories are those with a step-config.json and no user-prompt-template.md on main (37670575).
Activity
fix(skill-evals): give template-less eval steps a neutral user prompt (…
feat(bitbucket): add guarded cloud PR merge (#1471)
feat(tools): add asf-nexus and wire Nexus staging check into verify-rc (