Skip to content

fix(skills): require actionable CLI feedback repros#2498

Merged
miguel-heygen merged 1 commit into
mainfrom
fix/feedback-repro-reporting
Jul 15, 2026
Merged

fix(skills): require actionable CLI feedback repros#2498
miguel-heygen merged 1 commit into
mainfrom
fix/feedback-repro-reporting

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

CLI defect feedback from agents now arrives with a rerunnable reproduction packet instead of a symptom-only summary. Clean-run reports stay concise; bug and friction reports capture the exact command and working directory, expected versus actual behavior, verbatim error, output outcome, and workaround.

Project sharing remains optional. The skill asks for a link only when one is already available and safe to share.

A source-contract regression test keeps that reporting format in the shipped skill, and the published skills manifest is refreshed for update detection.

Test plan

  • bunx vitest run packages/cli/src/commands/coreSkillContent.test.ts packages/cli/src/utils/skillsManifest.test.ts — 59 passed
  • bun run --cwd packages/cli typecheck
  • bun run --cwd packages/cli gen:skills-manifest --check
  • bunx oxfmt --check packages/cli/src/commands/coreSkillContent.test.ts skills/hyperframes-cli/SKILL.md skills/hyperframes-cli/references/preview-render.md skills-manifest.json
  • bunx oxlint packages/cli/src/commands/coreSkillContent.test.ts

@miguel-heygen
miguel-heygen force-pushed the fix/feedback-repro-reporting branch from e35b768 to 4bb1e67 Compare July 15, 2026 21:54
@miguel-heygen
miguel-heygen force-pushed the fix/feedback-repro-reporting branch from 4bb1e67 to 797826f Compare July 15, 2026 21:59

@james-russo-rames-d-jusso james-russo-rames-d-jusso left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 797826f1. Docs + contract test.

Blockers

(none)

Concerns

(none)

Nits

(none)

Green notes

🟢 Repro packet template is compact + comprehensive. Five fields (REPRO COMMAND, EXPECTED / ACTUAL, EXACT ERROR, OUTCOME, WORKAROUND) cover what a maintainer actually needs to triage without ambiguity. OUTCOME enumerates the three failure modes the CLI can produce (output correct | output corrupt | fallback succeeded | hard exit | command hung) — better than prose that leaves fallback-vs-hard-exit implicit.

🟢 SKILL.md brief entry is byte-consistent with the full reference. The one-paragraph summary in SKILL.md names the rerunnable command, expected vs actual, exact error, outcome, and workaround — same five slots the reference expands. Pin the packet template in one place; the summary paraphrases without re-authoring.

🟢 Contract test asserts every required field. renderReference asserts all five header strings, skill asserts the packet-name mention. If the reference drifts silently (a maintainer trims WORKAROUND: in a copy pass), the test fires. Also asserts skill contains "reproduction packet" so the SKILL.md pointer can't be lost either.

🟢 skills-manifest.json hash updated correctly. Only hyperframes-cli.hash changes (from 1065ea34fecfda5005f7c65e6f9db100); files: 11 unchanged; no other skill entry drifts. Consistent with the gen-skills-manifest output pattern from prior CLI-feedback PRs.

🟢 Privacy language kept intact. "publishes a minimal reproduction to a public URL" + "opt-in and consent-gated; the issue is never auto-submitted" preserved verbatim in preview-render.md. Also added "redact secrets and credentials" + "share a project link only when … safe to share" — tightens the guard-rail without softening the existing one.

Verdict framing

Right scope, single-skill, no cross-skill byte-identity considerations (nothing in this PR touches the faceless-explainer/pr-to-video/product-launch-video shared triplet). LGTM from my side.

Review by Rames D Jusso

@vanceingalls vanceingalls left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE (RIGHT)

Small, well-tested prompt contract widening: bug/friction feedback now carries a compact REPRO / EXPECTED-ACTUAL / EXACT ERROR / OUTCOME / WORKAROUND packet; clean runs stay short. The source-contract test asserts the shipped strings, so drift between the docs and the invariant is caught in CI.

Verified

  • Test assertions in coreSkillContent.test.ts match the exact strings added to SKILL.md (reproduction packet) and references/preview-render.md (REPRO COMMAND:, EXPECTED / ACTUAL:, EXACT ERROR:, OUTCOME:, WORKAROUND:).
  • Cross-skill drift audit: --file-issue and hyperframes feedback narrative appear only in the two files this PR touches (GitHub code search: 2 hits total, both edited here). No sibling skill scaffold or reference (lint-validate-inspect.md, other CLI refs) carries parallel bug-repro guidance that would need updating.
  • SKILL.md pointer The required packet format and privacy warning live in references/preview-render.md matches the file that carries the packet template.
  • Manifest hash 1065ea34fecfda50 → 05f7c65e6f9db100 for hyperframes-cli is consistent with the two edited files — Skills: manifest in sync is green.
  • Doc-only contract; no runtime code change; no backward-compat risk for external skill authors.

Non-blocking nits (follow-up if you like)

  1. references/preview-render.md never names the reproduction packet the SKILL.md now points to — the phrase only appears in the intro sentence above the code fence. A **Reproduction packet** sub-heading above the fenced block would make the round-trip from SKILL.md obvious.
  2. The rewrite drops concrete trigger examples the old prose called out (Killed: 9, Target closed, and the 4K / <video> / font-family: 苹方 triggers). The OUTCOME categorical (output correct | output corrupt | fallback succeeded | hard exit | command hung) covers the exit modes acceptably; the loss is a minor teaching-value hit for agents cold-reading the reference.
  3. expect(renderReference).toContain("<LABEL>:") asserts presence anywhere. A future edit that scatters the labels into disjoint prose (breaking the packet code fence) keeps the test green while defeating the intent. Not worth blocking on; tighten to match a contiguous fenced block only if the packet becomes tooling-load-bearing.

CI: Prior run 29453867207 was cancelled by the copy-cleanup force-push. Run 29453962500 is all-green on completed jobs including Test: skills, Skills: manifest in sync, Fallow audit, Format, Lint, SDK unit + contract + smoke, Studio: load smoke, Preview parity, preview-regression, and CLI: npx shim on ubuntu/macos. Remaining lanes are in-progress and have no plausible failure surface on a docs+test PR. mergeStateStatus: BLOCKED reflects REVIEW_REQUIRED, not CI red.

Review by Via

@miguel-heygen
miguel-heygen merged commit 3a71a03 into main Jul 15, 2026
56 of 68 checks passed
@miguel-heygen
miguel-heygen deleted the fix/feedback-repro-reporting branch July 15, 2026 22:14
pull Bot pushed a commit to chizee/hyperframes that referenced this pull request Jul 17, 2026
Extend the CLI feedback reproduction packet (heygen-com#2498) with a fifth
mandated field, `COMPOSITION_STRUCTURE:`, and enforce presence of
`REPRO COMMAND:` / `COMPOSITION_STRUCTURE:` at feedback-submit time.

- Skill + reference now specify `COMPOSITION_STRUCTURE:` — a
  privacy-preserving structural anatomy (element census + attribute
  presence + timeline shape + delta + defect location) — required for
  any rating <=7 that describes a visual defect.
- `buildCompositionCensus()` + `renderCompositionCensusBlock()`
  auto-fill the block from composition HTML so agents don't ask the
  human user to hand-count `<video>` / `<img>` / sub-comp mounts.
  Counts + presence flags only — no file paths, no src URLs, no user
  text.
- `hyperframes feedback` soft-warns (never blocks) when a non-10
  `--comment` is missing `REPRO COMMAND:`, and when a rating-<=7
  visual-defect comment is missing `COMPOSITION_STRUCTURE:`. The
  warning points at the auto-census helper so agents remediate
  themselves.
- `coreSkillContent.test.ts` locks the new literal in both the skill
  and the reference file, following heygen-com#2498's pattern.

Extends heygen-com#2498. Follow-up: no change to `doctorSummary` generation, no
change to the feedback-submission API endpoint, no refactor of
heygen-com#2498's doc-content Jest test.

Signed-off-by: Via
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants