Measure Whether the Fleet's Skills Actually Trigger, Rather Than Assuming Their Descriptions Do #1447

The fleet ships 24 skills whose description frontmatter decides whether they fire, and nothing
measures whether they actually do. Every description here has been hand-tuned against intuition,
and two known failures say the tuning is not reliable on its own: a skill's procedure losing to a
generic session instruction, and repo-worktree needing its rule re-learned as "start every task
in its own worktree" after the skill itself did not surface it at task start.

A skill that does not trigger is indistinguishable, from inside a session, from a skill that does
not exist. That is the failure this would measure.

What the measurement is

Skill activation as a classification problem, one labeled prompt set per skill, scored as precision
and recall.

• Recall is the under-trigger problem: of the prompts this skill owns, how many actually fire
  it. This is the repo-worktree failure above.
• Precision is the over-trigger problem: when it does fire, how often it should have. This is
  what a suite catches that no amount of reading a description does, because a description written
  to fire more reliably fires on its neighbours' prompts too.
• Negative prompts are the part most easily skipped and the reason the suite is worth building:
  prompts inside the fleet's domain that no skill should own. Without them a description can be
  widened until recall is perfect and the skill fires on everything.

Prompts must be realistic user phrasings rather than paraphrases of the description, or the suite
measures string similarity instead of triggering.

Candidate harnesses

None is an Anthropic project, so "official" is not available here and the choice is a judgment
about maintenance risk rather than a lookup.

• UiPath/coder_eval: YAML config, one success criterion
  per skill per row, aggregates precision, recall, F1 and a confusion matrix, and gates a build on
  suite_thresholds such as a recall minimum. Published to PyPI. The writeup behind it is
  https://coder-eval.com/blog/does-your-claude-skill-trigger and its case study ran 1,001 prompts
  over 20 skills with roughly 50 negatives, which is close to this fleet's shape.
• adewale/skill-eval-harness: paired variants,
  trace artifacts, runner adapters.
• OpenAI's eval-skills writeup for the method
  rather than as a runnable harness.
• SkillTester benchmarks utility and security of agent skills,
  a different question, listed so it is not mistaken for this one.

Scope boundary, so this is not confused with the reviewer question

This measures triggering in an agent that already loads the skills tree, which is Claude Code,
Codex and opencode. It says nothing about whether GitHub's hosted Copilot reviewer reads
.github/skills/ at all, because every harness here drives a local agent that is handed the
directory. That question needs a live review and is being measured separately under #1321.

Why it is worth deciding now

#1321's Option 1 retires four carried documents and keeps .github/skills/ as the surface that
carries the rules a reviewer and an on-host agent must honor. That option is only sound if the
skills reliably fire. Today that is an assumption, and this issue is what would turn it into a
number.

Open questions

1. Which harness, given none is first-party and a suite is a maintenance commitment.
2. The prompt-set size and who authors it. Descriptions written by the same seat that writes the
   prompts will score better than they deserve.
3. Cadence and cost. A four-figure prompt suite on every pull request is not affordable, so this is
   plausibly a manual gate before a skill description changes, or a scheduled run, rather than a
   pull request check.
4. Which model tier the suite runs against, since triggering is a model behavior and a result on
   one tier does not transfer to another.