feat(eval-harness): add eval-harness skill for worktree-based code evaluation

[baleen37]

Summary

Adds a new me:eval-harness skill that evaluates code changes or compares two implementations using isolated git worktrees and parallel subagents.

Changes

• plugins/me/skills/eval-harness/SKILL.md — new skill with worktree-based A/B evaluation (Phase 0: setup, Phase 1: parallel subagents, Phase 2: model grader/judge, Phase 3: structured report)
• tests/me/eval-harness.bats — 15 BATS tests covering all skill requirements
• plugins/me/skills/eval/SKILL.md — added cross-reference pointing code eval use cases to me:eval-harness
• docs/superpowers/plans/2026-03-16-eval-harness-skill.md — implementation plan

Tests

• All 15 eval-harness tests pass
• Full test suite (50 tests) passes

Breaking

None — existing me:eval skill unchanged (prompt A/B comparison).

Summary by CodeRabbit

• Documentation

  • Introduced eval-harness skill for comparing two code variants using isolated environments with parallel subagent evaluation and optional model grading.
  • Includes comprehensive implementation plan with multi-phase workflow, input specifications, and structured reporting format.
  • Updated existing evaluation tool documentation to reference eval-harness for non-prompt evaluation use cases.

• Tests

  • Added test suite to validate eval-harness skill configuration and required documentation fields.

[coderabbitai]

📝 Walkthrough

Walkthrough

This change introduces a comprehensive plan and complete scaffolding for a new eval-harness skill that compares two code variants using isolated git worktrees, parallel subagent evaluation, optional model grading, and standardized reporting. Includes documentation, test definitions, and cross-reference updates.

Changes

Cohort / File(s)	Summary
Eval-Harness Skill Definition docs/superpowers/plans/2026-03-16-eval-harness-skill.md, plugins/me/skills/eval-harness/SKILL.md	Introduces comprehensive skill definition with four-phase workflow: isolated worktree setup, parallel subagent evaluation, optional model grading, and structured reporting. Includes use-case guidelines, input format specification, phase-by-phase operations, and cleanup procedures.
Test Scaffolding tests/me/eval-harness.bats	Adds Bash/BATS test suite validating skill file structure, required frontmatter fields (name, description), and content verification for key concepts: worktree isolation, parallelism, model/code graders, input variants, winner determination, tie handling, anonymization, and cleanup.
Eval Skill Cross-Reference plugins/me/skills/eval/SKILL.md	Updates guidance to direct non-prompt evaluation use cases to me:eval-harness skill instead of direct eval usage.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• feat(skills): add eval-harness skill and eval command #499 — Related PR also adding and documenting the eval-harness skill with matching SKILL.md structure and eval tooling documentation for the same feature area.

Poem

🐰 Hopping through worktrees, left and right,
Two variants dance in parallel light,
The judge rabbit reads their anonymous case,
Declares the winner with dignified grace—
Then tidies the burrows, all clean and bright! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding a new eval-harness skill for worktree-based code evaluation, which aligns with the primary content of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/eval-harness-skill

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/me/eval-harness.bats (1)

56-58: Minor redundancy in pattern.

The -i flag already makes the match case-insensitive, so "tie\|Tie" is redundant—"tie" alone suffices. Not a bug, just a nit.

💡 Optional simplification

 `@test` "eval-harness: skill documents tie as possible outcome" {
-    grep -qi "tie\|Tie" "$SKILL_FILE"
+    grep -qi "tie" "$SKILL_FILE"
 }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/me/eval-harness.bats` around lines 56 - 58, In the test "eval-harness:
skill documents tie as possible outcome" update the grep pattern used against
$SKILL_FILE to remove the redundant alternation; since grep is invoked with -i,
replace the pattern "tie\|Tie" with just "tie" so the case-insensitive match is
sufficient and the test remains correct and simpler.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@tests/me/eval-harness.bats`:
- Around line 56-58: In the test "eval-harness: skill documents tie as possible
outcome" update the grep pattern used against $SKILL_FILE to remove the
redundant alternation; since grep is invoked with -i, replace the pattern
"tie\|Tie" with just "tie" so the case-insensitive match is sufficient and the
test remains correct and simpler.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 83d620a1-3b6e-4da8-8e34-35362f755304

📥 Commits

Reviewing files that changed from the base of the PR and between eb86517 and dcfdf42.

📒 Files selected for processing (4)

• docs/superpowers/plans/2026-03-16-eval-harness-skill.md
• plugins/me/skills/eval-harness/SKILL.md
• plugins/me/skills/eval/SKILL.md
• tests/me/eval-harness.bats