Skip to content

feat(skills): add eval-harness skill and eval command - #499

Merged
baleen37 merged 1 commit into
mainfrom
feat/eval-harness-skill
Feb 15, 2026
Merged

feat(skills): add eval-harness skill and eval command#499
baleen37 merged 1 commit into
mainfrom
feat/eval-harness-skill

Conversation

@baleen37

@baleen37 baleen37 commented Feb 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add skills/eval-harness/SKILL.md — eval-driven development (EDD) skill with capability/regression eval types, pass@k/pass^k metrics, code/model/human grader types, and define→implement→evaluate→report workflow
  • Add commands/eval.md/eval slash command with define, check, report, list, and clean subcommands for managing eval definitions in .claude/evals/

Test plan

  • Verify /eval command triggers correctly in Claude Code session
  • Verify eval-harness skill loads when EDD-related queries are asked
  • Test /eval define <name> creates eval definition file
  • Test /eval list shows existing evals
  • Verify SKILL.md frontmatter follows plugin skill conventions (third-person description, trigger phrases)

Summary by CodeRabbit

Release Notes

  • Documentation
    • Added comprehensive Eval Harness documentation covering Evaluation-Driven Development (EDD) concepts, eval types, grading methods, metrics, and workflow guidance.
    • Added Eval command documentation detailing subcommands for defining, checking, reporting, listing, and managing evaluations with examples.

Add eval-driven development (EDD) skill with capability/regression eval
types, pass@k metrics, grader types, and workflow guidance. Add /eval
command with define, check, report, list, and clean subcommands.
@baleen37
baleen37 enabled auto-merge (squash) February 15, 2026 17:17
@coderabbitai

coderabbitai Bot commented Feb 15, 2026

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

This pull request adds two documentation files establishing eval-driven development practices: commands/eval.md provides a command reference guide with subcommands and workflows, while skills/eval-harness/SKILL.md offers comprehensive documentation on evaluation-driven development methodology, grading strategies, and implementation examples.

Changes

Cohort / File(s) Summary
Eval Command Documentation
commands/eval.md
Introduces the eval command with subcommands (define, check, report, list, clean), including argument parsing, directory structure under .claude/evals/, and examples of capability and regression evaluations.
Eval Harness Skill Documentation
skills/eval-harness/SKILL.md
Comprehensive guide on Evaluation-Driven Development (EDD) covering methodology, eval types (Capability/Regression), grading options (Code-Based/Model-Based/Human), metrics (pass@k, pass^k), workflow stages, storage conventions, and detailed examples including authentication eval implementation.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 A harness for evals, so clever and bright,
Commands laid out for the developer's night,
Define, check, and report with such care,
Testing AI's wit—oh what a fair pair!
Evaluation-driven, we hop toward the light! 🌟

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/eval-harness-skill

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@baleen37
baleen37 merged commit 7054048 into main Feb 15, 2026
4 of 5 checks passed
@baleen37
baleen37 deleted the feat/eval-harness-skill branch February 15, 2026 17:18
baleen-release-app Bot pushed a commit that referenced this pull request Feb 15, 2026
# [5.27.0](v5.26.0...v5.27.0) (2026-02-15)

### Features

* **skills:** add eval-harness skill and eval command ([#499](#499)) ([7054048](7054048))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant