Repository navigation
feat(harness): return failed output checks to the Agent - #217
Merged
Merged
Conversation
Five direct runs (@nuxtjs/robots, @nuxt/scripts, @unhead/vue, which-nuxt, retriv) and their independent reviews found the same gaps. Every run hand-wrote a code-block extractor. Most review findings came from prose claims, which the Skill never called examples. One run tested npm latest (1.x) while the branch published 2.0 betas. Node fetch dropped the Host header the robots run needed. Reviewers had no definition of the three finding levels. Add extract-blocks.mjs, --header and response headers to serve-fixture, an internal-package gate, the branch version rule, prose-claim and failure-input checks, and defined review levels. The Harness requests now state the Skillgen output shape and keep fixtures out of the output.
A run whose output failed a deterministic check ended as InvalidSkill, and the session was destroyed before the Agent saw why. The Agent still held its context and could have fixed the issue in one turn. The Harness now sends the issues back in the same session, up to two times. The @unhead/vue Skill shipped a frontmatter description with quotes and % that read as broken. The check now requires one plain YAML line without double quotes, backticks, or %, and the Skill says so.
harlan-zw
added
the
harlan-agent-review
harlan-github-agent
Bot
added
harlan-agent-review-required
1 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔗 Linked issue
Related to harlan-zw/which-nuxt#13, unjs/unhead#1009, skilld-dev/retriv#21, nuxt-modules/robots#339, nuxt/scripts#950
❓ Type of change
📚 Description
Five direct
generate-package-skillruns, each followed by an independentreview-skillrun, shipped the pull requests above. Each review found errors or warnings in the generated Skill, and most findings came from prose claims the Skill never called examples. One run tested npmlatest(1.x) while the branch published 2.0 betas. Every run hand-wrote a code-block extractor. The@unhead/vueSkill also shipped a description with quotes and%that read as broken. Nothing caught it, because the Harness checks output only after the Agent finishes, then ends the run.The Harness now returns failed output checks to the Agent in the same session, up to two times, and requires the description to be one plain YAML line.
generate-package-skillgains the run findings: an internal-package gate, the branch version rule, prose claims and failure inputs as examples, the Skillgen output shape, and clearer README andfilesrules. It also bundlesscripts/extract-blocks.mjs.serve-fixture.mjsgains--headerand records response headers.review-skilldefines its three levels and tests the version the Skill names.