Skip to content

Review agent should detect repo-specific references in scaffolded skill definitions #1908

Open

Description

@fullsend-ai-retro

What happened

On PR #1840, the code agent modified files under internal/scaffold/fullsend-repo/skills/pr-review/ — these are generic skill definitions deployed to all repos via scaffolding. The changes hardcoded docs/plans/ path references in both SKILL.md and correctness.md. The review bot ran twice and posted findings about internal consistency, scope creep, and pattern ordering, but never flagged the portability problem. The human reviewer (ralphbean) caught it in 3 inline comments requesting removal of all path-specific references, leading to a CHANGES_REQUESTED verdict and a fix cycle.

What could go better

The review bot should have a portability/generality check for changes to scaffolded skill definitions (internal/scaffold/). These files are templates deployed across many repos — hardcoded paths like docs/plans/ will break or be misleading in repos with different directory structures. The bot's first run came close (it noticed the docs/plans/ path and raised a design question) but framed it as a detection strategy ambiguity rather than a portability violation. This is a high-confidence finding: the human explicitly called this out as the primary issue, and it required a full fix cycle to resolve.

Proposed change

Add a portability check to the review agent's analysis when the diff touches files under internal/scaffold/. This could be implemented as:

  1. In the pr-review skill's SKILL.md (internal/scaffold/fullsend-repo/skills/pr-review/SKILL.md): Add a dispatch rule or review dimension for changes to scaffold templates that checks for repo-specific assumptions (hardcoded paths, org-specific references, repo-structure dependencies).

  2. Alternatively, in the correctness sub-agent (internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md): Add guidance that when reviewing changes to files under internal/scaffold/, the agent should verify that all references are generic and portable across different repo structures.

The check should flag any literal directory paths (e.g., docs/plans/, src/components/) in scaffold templates as potential portability issues, unless they reference well-known conventional paths (e.g., .github/, CLAUDE.md).

Validation criteria

The next 3 PRs that modify files under internal/scaffold/ and introduce repo-specific path references should have the review agent flag the portability concern without requiring human intervention. Measure by checking whether human reviewers still need to manually catch path-specificity issues in scaffold changes.


Generated by retro agent from #1840

Activity

  1. added
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    component/harnessAgent harness, config, and skills loading
    priority/mediumNormal priority, plan for next cycle
    on Jun 4, 2026
  2. fullsend-ai-triage commented on Jun 4, 2026

    @fullsend-ai-triage

    This is a stale-issue relevance check rather than an implementation trigger, per the request above. Here's what changed and what's still true:

    Still relevant. The underlying gap -- no review-time check for hardcoded, repo-specific paths introduced into generic skill/sub-agent definitions -- remains unaddressed. Current SKILL.md dispatch classification (step 3b/3c) and the correctness, cross-repo-contracts, style-conventions, docs-currency, and intent-coherence sub-agent definitions were all inspected directly; none contain a portability check for templated skill content.

    What moved. The file paths named in the original proposal no longer exist. internal/scaffold/fullsend-repo/ in this repo no longer has a skills/ subtree at all (only providers/, templates/, scripts/, profiles/, .github/). The pr-review skill and its sub-agents (including correctness.md) now live in the fullsend-ai/agents repo under skills/pr-review/. Any implementation needs to target that repo/path instead.

    Related work. #1909 tracks the code-agent side of the same PR #1840 incident and is also still open/unresolved -- the two remain complementary as previously noted.

    No conflicting work in flight. No open PR in either fullsend-ai/fullsend or fullsend-ai/agents references this issue or implements a portability check.

    No ready-to-code trigger is being recommended -- this is left for human prioritization as requested.


    Labels: Aligns labeling with the repo's convention for other 'review agent should detect X' feature issues (e.g. #1622, #1660, #3477, #1562), which are consistently tagged feature and component/skills.


    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
    Previous run

    Triage Summary

    Category: Feature — add a portability check to the review agent for scaffold template changes.

    Problem: When PRs modify files under internal/scaffold/, the review agent does not check whether the changes introduce repo-specific references (hardcoded paths, org-specific assumptions). On PR #1840, the correctness sub-agent noticed a docs/plans/ reference but framed it as a design ambiguity rather than a portability violation, requiring a human reviewer to catch the real issue.

    Recommended approach: Add scaffold-portability guidance to either the pr-review SKILL.md dispatch rules or the correctness.md sub-agent definition. When the diff touches internal/scaffold/, the review should flag literal directory paths that aren't well-known conventional paths (.github/, CLAUDE.md, etc.).

    Related but distinct: #1909 addresses the same root incident but targets the code agent (preventing it from introducing repo-specific paths in scaffold files). This issue targets the review agent (catching such paths during review). Both are complementary.

    Proposed test case:

    Scenario: Review agent flags hardcoded paths in scaffold templates
      Given a PR that modifies internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/correctness.md
      And the diff introduces a literal path like "docs/plans/"
      When the review agent runs
      Then it produces a portability finding flagging the hardcoded path
    
    Scenario: Well-known paths do not trigger false positives
      Given a PR that modifies a scaffold template
      And the diff references ".github/" or "CLAUDE.md"
      When the review agent runs
      Then no portability finding is produced
    
    Edge cases: relative paths (../foo), paths inside code fences,
    bare conventional names (docs/ without subdirectory)
    

    Labels: Retro-generated feature request targeting the review agent's handling of scaffold templates, medium priority since it requires human fallback but is not blocking.

  3. fullsend-ai-prioritize commented on Jun 8, 2026

    @fullsend-ai-prioritize

    RICE Priority Score: 2.4

    Score breakdown
    Dimension Score Reasoning
    Reach 1.5 Scaffold templates under internal/scaffold/ are deployed across all repos, so portability bugs in these files propagate widely. However, the trigger condition (PRs modifying scaffold files) is a relatively small subset of all PRs, limiting direct reach. Scored 1.5 for affecting multiple teams/repos indirectly.
    Impact 1 The current workaround is human review catching hardcoded paths, which adds a fix cycle (as happened on PR #1840). This is a noticeable workflow improvement — prevents CHANGES_REQUESTED rounds — but is not blocking since humans can catch it. Scored 1 for medium impact.
    Confidence 0.8 The issue is very well-described with a concrete incident (PR #1840), clear proposed solutions, specific test cases, and validation criteria. The triage comment confirms scope and distinguishes it from the related #1909. Scored 0.8 for high confidence.
    Effort 0.5 The fix involves adding portability guidance to existing skill definition files (SKILL.md or correctness.md) — essentially prompt engineering with a clear pattern to check for. Well-scoped with two concrete implementation options provided. Scored 0.5 for a simple, well-scoped change.

    Formula: (1.5 x 1 x 0.8) / 0.5 = 2.4

  4. ralphbean commented on Sep 22, 2026

    @ralphbean
    Member

    /fs-triage lots of things have changed between when this was originally filed and now across both the https://github.com/fullsend-ai/fullsend and https://github.com/fullsend-ai/agents repos. Is this still relevant? This is a stale-issue review, not an implementation request: even if the issue is still a valid bug and has sufficient information, do not recommend or trigger ready-to-code yet. Leave it for human prioritization after confirming whether it remains relevant. Do not close a still-relevant bug merely to avoid coding it.

  5. fullsend-ai-triage commented on Sep 22, 2026

    @fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 12:05 AM UTC · Completed 12:10 AM UTC

    Commit: b9e5099 · View workflow run →

    Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $0.46

  6. added
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    and removed
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    on Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/reviewReview agentcomponent/harnessAgent harness, config, and skills loadingcomponent/skillsfeatureFeature-category issue awaiting human prioritizationpriority/mediumNormal priority, plan for next cycletriagedTriaged but awaiting human prioritization

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions