fix(skills): require actionable CLI feedback repros#2498
Conversation
e35b768 to
4bb1e67
Compare
4bb1e67 to
797826f
Compare
james-russo-rames-d-jusso
left a comment
There was a problem hiding this comment.
Reviewed at 797826f1. Docs + contract test.
Blockers
(none)
Concerns
(none)
Nits
(none)
Green notes
🟢 Repro packet template is compact + comprehensive. Five fields (REPRO COMMAND, EXPECTED / ACTUAL, EXACT ERROR, OUTCOME, WORKAROUND) cover what a maintainer actually needs to triage without ambiguity. OUTCOME enumerates the three failure modes the CLI can produce (output correct | output corrupt | fallback succeeded | hard exit | command hung) — better than prose that leaves fallback-vs-hard-exit implicit.
🟢 SKILL.md brief entry is byte-consistent with the full reference. The one-paragraph summary in SKILL.md names the rerunnable command, expected vs actual, exact error, outcome, and workaround — same five slots the reference expands. Pin the packet template in one place; the summary paraphrases without re-authoring.
🟢 Contract test asserts every required field. renderReference asserts all five header strings, skill asserts the packet-name mention. If the reference drifts silently (a maintainer trims WORKAROUND: in a copy pass), the test fires. Also asserts skill contains "reproduction packet" so the SKILL.md pointer can't be lost either.
🟢 skills-manifest.json hash updated correctly. Only hyperframes-cli.hash changes (from 1065ea34fecfda50 → 05f7c65e6f9db100); files: 11 unchanged; no other skill entry drifts. Consistent with the gen-skills-manifest output pattern from prior CLI-feedback PRs.
🟢 Privacy language kept intact. "publishes a minimal reproduction to a public URL" + "opt-in and consent-gated; the issue is never auto-submitted" preserved verbatim in preview-render.md. Also added "redact secrets and credentials" + "share a project link only when … safe to share" — tightens the guard-rail without softening the existing one.
Verdict framing
Right scope, single-skill, no cross-skill byte-identity considerations (nothing in this PR touches the faceless-explainer/pr-to-video/product-launch-video shared triplet). LGTM from my side.
vanceingalls
left a comment
There was a problem hiding this comment.
Verdict: APPROVE (RIGHT)
Small, well-tested prompt contract widening: bug/friction feedback now carries a compact REPRO / EXPECTED-ACTUAL / EXACT ERROR / OUTCOME / WORKAROUND packet; clean runs stay short. The source-contract test asserts the shipped strings, so drift between the docs and the invariant is caught in CI.
Verified
- Test assertions in
coreSkillContent.test.tsmatch the exact strings added toSKILL.md(reproduction packet) andreferences/preview-render.md(REPRO COMMAND:,EXPECTED / ACTUAL:,EXACT ERROR:,OUTCOME:,WORKAROUND:). - Cross-skill drift audit:
--file-issueandhyperframes feedbacknarrative appear only in the two files this PR touches (GitHub code search: 2 hits total, both edited here). No sibling skill scaffold or reference (lint-validate-inspect.md, other CLI refs) carries parallel bug-repro guidance that would need updating. - SKILL.md pointer
The required packet format and privacy warning live in references/preview-render.mdmatches the file that carries the packet template. - Manifest hash
1065ea34fecfda50 → 05f7c65e6f9db100forhyperframes-cliis consistent with the two edited files —Skills: manifest in syncis green. - Doc-only contract; no runtime code change; no backward-compat risk for external skill authors.
Non-blocking nits (follow-up if you like)
references/preview-render.mdnever names the reproduction packet the SKILL.md now points to — the phrase only appears in the intro sentence above the code fence. A**Reproduction packet**sub-heading above the fenced block would make the round-trip from SKILL.md obvious.- The rewrite drops concrete trigger examples the old prose called out (
Killed: 9,Target closed, and the4K/<video>/font-family: 苹方triggers). The OUTCOME categorical (output correct | output corrupt | fallback succeeded | hard exit | command hung) covers the exit modes acceptably; the loss is a minor teaching-value hit for agents cold-reading the reference. expect(renderReference).toContain("<LABEL>:")asserts presence anywhere. A future edit that scatters the labels into disjoint prose (breaking the packet code fence) keeps the test green while defeating the intent. Not worth blocking on; tighten to match a contiguous fenced block only if the packet becomes tooling-load-bearing.
CI: Prior run 29453867207 was cancelled by the copy-cleanup force-push. Run 29453962500 is all-green on completed jobs including Test: skills, Skills: manifest in sync, Fallow audit, Format, Lint, SDK unit + contract + smoke, Studio: load smoke, Preview parity, preview-regression, and CLI: npx shim on ubuntu/macos. Remaining lanes are in-progress and have no plausible failure surface on a docs+test PR. mergeStateStatus: BLOCKED reflects REVIEW_REQUIRED, not CI red.
Review by Via
Extend the CLI feedback reproduction packet (heygen-com#2498) with a fifth mandated field, `COMPOSITION_STRUCTURE:`, and enforce presence of `REPRO COMMAND:` / `COMPOSITION_STRUCTURE:` at feedback-submit time. - Skill + reference now specify `COMPOSITION_STRUCTURE:` — a privacy-preserving structural anatomy (element census + attribute presence + timeline shape + delta + defect location) — required for any rating <=7 that describes a visual defect. - `buildCompositionCensus()` + `renderCompositionCensusBlock()` auto-fill the block from composition HTML so agents don't ask the human user to hand-count `<video>` / `<img>` / sub-comp mounts. Counts + presence flags only — no file paths, no src URLs, no user text. - `hyperframes feedback` soft-warns (never blocks) when a non-10 `--comment` is missing `REPRO COMMAND:`, and when a rating-<=7 visual-defect comment is missing `COMPOSITION_STRUCTURE:`. The warning points at the auto-census helper so agents remediate themselves. - `coreSkillContent.test.ts` locks the new literal in both the skill and the reference file, following heygen-com#2498's pattern. Extends heygen-com#2498. Follow-up: no change to `doctorSummary` generation, no change to the feedback-submission API endpoint, no refactor of heygen-com#2498's doc-content Jest test. Signed-off-by: Via
Summary
CLI defect feedback from agents now arrives with a rerunnable reproduction packet instead of a symptom-only summary. Clean-run reports stay concise; bug and friction reports capture the exact command and working directory, expected versus actual behavior, verbatim error, output outcome, and workaround.
Project sharing remains optional. The skill asks for a link only when one is already available and safe to share.
A source-contract regression test keeps that reporting format in the shipped skill, and the published skills manifest is refreshed for update detection.
Test plan
bunx vitest run packages/cli/src/commands/coreSkillContent.test.ts packages/cli/src/utils/skillsManifest.test.ts— 59 passedbun run --cwd packages/cli typecheckbun run --cwd packages/cli gen:skills-manifest --checkbunx oxfmt --check packages/cli/src/commands/coreSkillContent.test.ts skills/hyperframes-cli/SKILL.md skills/hyperframes-cli/references/preview-render.md skills-manifest.jsonbunx oxlint packages/cli/src/commands/coreSkillContent.test.ts