Improve evals.

[adampoit]

• Improve skills based on evals.
• Move evals to gate draft-release.

[not-adam]

Mira PR Walkthrough

This PR overhauls the eval infrastructure by extracting scenario intent contracts into immutable JSON files (evals/intents/*.json), isolating the synthetic user-driver policy into protected templates (evals/user-driver/*.md), and enforcing contract integrity through a local Pi extension guard (eval-intent-guard.ts), CI checks (evals:contracts), and comprehensive tests. It simultaneously improves Patchlane worker skills (fork-setup, workspace, sync-patches, health-check, migration) to carry workflow requirements rather than embedding them in driver prompts or scenario code. The release pipeline is restructured to move live eval gating from publish.yml into draft-release.yml.

graph LR
    intent["evals/intents/*.json"]
    ud["evals/user-driver/*.md"]
    guard[".pi/extensions/eval-intent-guard.ts"]
    skills["skills/*/SKILL.md"]
    ci[".github/workflows/ci.yml"]
    draft[".github/workflows/draft-release.yml"]

    intent -->|"loaded & validated by"| guard
    ud -->|"protected by"| guard
    guard -->|"enforced in CI via"| ci
    guard -->|"gated on draft release via"| draft
    skills -->|"carry workflow rules for"| intent
    ci -->|"contract hash check"| intent
    ci -->|"contract hash check"| ud

Loading

Confidence: 4/5   ◉◉◉◉○   Safe with minor fixes

• The extensiveness and structural soundness of tests, CI gating, and contract integrity checks make this PR well-defended, but the large number of changed files (38) and deep refactoring across eval scenarios, runner logic, and skill definitions warrants careful review—particularly for unintended behavioral drift in assertions, backward compatibility of transcript parsing, and any edge cases in the Pi extension shell command classification.

Blast Radius — 2 dependent repositories, 3 total references

adampoit/mira — 1 reference
adampoit/wezterm — 2 references

42 files reviewed

---

Comment @not-adam help to get the list of available commands and usage tips.