Skip to content

Review agent: add behavioral-change detection for dispatch and workflow files #1069

Open

Description

@fullsend-ai-retro

What happened

PR #906 renamed all /fs-* slash command patterns to unprefixed versions in dispatch workflow files, silently breaking all slash commands. The review agent approved PR #906 without detecting this behavioral regression. PR #1063 was then needed to fix it. The dispatch workflow files (reusable-dispatch.yml, per-repo dispatch.yml) define the command routing that all agents depend on — changes to command patterns are high-impact behavioral changes that should be flagged even when the code is syntactically valid.

The failed retro agent run 25994807322 on this PR also identified this gap but could not file the proposal due to schema validation failures.

What could go better

The review agent's code-review skill does not have specific guidance for evaluating dispatch/workflow file changes. These files have implicit contracts (command patterns must match what docs, skills, and agent definitions reference) that are not captured in any schema or test — they are only discoverable through cross-referencing.

Confidence: Medium. I am confident that PR #906's regression was missed by the review agent. I am moderately confident that adding cross-reference checking guidance to the code-review skill would catch similar regressions. However, the review agent did use a subagent on PR #1063 to cross-reference /fs- patterns across the codebase, so the capability exists — it just was not applied on the original PR #906.

Uncertainty: I have not reviewed the PR #906 review agent transcript to confirm why the regression was missed. It may have been a different root cause (e.g., the review agent was not triggered, or the diff was too large).

Proposed change

In the code-review skill definition (internal/scaffold/fullsend-repo/skills/code-review.md or equivalent), add a review dimension for dispatch/workflow file changes that instructs the agent to:

  1. When the diff modifies comment pattern matching (e.g., issue_comment body checks), cross-reference the patterns against docs, agent definitions, and skill files to verify consistency.
  2. Flag any command pattern rename as a high-severity behavioral change requiring explicit justification.
  3. When workflow files change dispatch routing, verify that all referenced agent types still have matching handlers.

This is a targeted addition to the existing review dimensions, not a new skill.

Validation criteria

  1. A PR that renames slash command patterns in dispatch files triggers a high-severity finding about behavioral change.
  2. The review agent cross-references command patterns against at least agent definitions and docs when reviewing dispatch file changes.
  3. No false positives on dispatch file changes that do not alter command routing (e.g., adding a new command alongside existing ones).

Generated by retro agent from #1063

Activity

  1. fullsend-ai-triage commented on May 17, 2026

    @fullsend-ai-triage

    Re-triaged for relevance as requested. The overall architecture has moved on since May: the review-agent skills now live in fullsend-ai/agents (skills/code-review, skills/pr-review/sub-agents), not the fullsend-repo scaffold path this issue originally cited, which no longer holds skill files in that form.

    More importantly, the three related issues the prior triage flagged (#890/#1911, #852, #1060) have all since been closed and merged (via PRs #1912, #1807, #1156). Together they added: consumer-completeness checks for new dispatch/enum values, generic rename/removal staleness grepping (including doc references), and mandatory findings for permission-manifest changes. That generic staleness check is a real mitigation for the PR #906 class of regression — but it reports every identifier rename, including a dispatch/workflow command-pattern rename, as a flat medium-severity finding, with no dispatch-routing-specific escalation or explicit handler-consistency verification. So the specific gap this issue calls out — treating a slash-command routing rename as the high-severity behavioral change it is — remains open.

    Two other open issues touch adjacent ground without duplicating this one: #5332 (review agent should validate dispatch infrastructure fixes across all deployment modes) and #1309 (mandatory finding when a diff modifies the review agent's own behavior files). Worth coordinating with if this is picked up.

    The issue is still valid and actionable, but per the requester's instruction this triage is not recommending or triggering ready-to-code — it's being left for human prioritization to decide whether the remaining, narrower scope is still worth implementing now.


    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
    Previous run

    Triage Summary

    This is a well-defined feature request to add behavioral-change detection for dispatch/workflow files to the code-review skill.

    Problem: PR #906 renamed slash command patterns in dispatch workflow files, breaking all slash commands. The review agent approved the change without detecting the regression, and a follow-up fix PR was needed.

    Root cause hypothesis: The code-review skill has no review dimension for dispatch/workflow file changes. Command pattern renames are syntactically valid but semantically breaking, and nothing instructs the review agent to cross-reference patterns against docs, skills, and agent definitions.

    Proposed change: Add guidance to the code-review skill to:

    1. Cross-reference command patterns against docs and agent definitions when comment pattern matching is modified
    2. Flag command pattern renames as high-severity behavioral changes
    3. Verify dispatch routing handlers when workflow files change

    Related issues:

    These are complementary, not duplicates, but coordinated implementation could reduce redundancy.

    Proposed test case:

    Scenario: Command pattern rename triggers high-severity finding
      Given a PR diff that renames a slash command pattern in a dispatch workflow file
        (e.g., changing '/fs-review' to '/review' in an issue_comment body check)
      When the code-review skill evaluates the diff
      Then a high-severity finding is emitted about behavioral change to command routing
      And the finding references the need to verify consistency with docs and agent definitions
    
    Scenario: Adding new command does not trigger false positive
      Given a PR diff that adds a new command pattern alongside existing ones
      When the code-review skill evaluates the diff
      Then no behavioral-change finding is emitted for the new addition
    

    Labels: This is a feature request to enhance the review agent's code-review skill with a new review dimension. It is ready for implementation with clear scope and validation criteria.

  2. ifireball commented on May 26, 2026

    @ifireball
    Member

    /fs-prioritize

  3. fullsend-ai-prioritize commented on May 31, 2026

    @fullsend-ai-prioritize

    RICE Priority Score: 1.8

    Score breakdown
    Dimension Score Reasoning
    Reach 1.5 Dispatch workflow files define command routing for all slash commands across the platform. When these break (as happened with PR #906), all users are affected. However, this issue is a preventive measure for the review agent, so reach is based on the frequency of dispatch file changes and the breadth of impact when regressions slip through. Multiple strategic customers depend on slash commands working reliably.
    Impact 1.5 The PR #906 regression broke all slash commands silently, requiring an emergency fix PR #1063. Adding behavioral-change detection to the review agent would catch this class of high-severity regression before merge, representing a significant pain point prevention. The impact per occurrence is very high, but this is a guardrail rather than a direct user-facing fix.
    Confidence 0.8 High confidence. The issue is exceptionally well-described with a clear root cause analysis, concrete proposed changes to the code-review skill, validation criteria, and related issues identified by triage. The scope is well-bounded — adding review dimensions to an existing skill definition. The only gap is not having reviewed the PR #906 review agent transcript.
    Effort 1 Medium effort. The change is a targeted addition to the code-review skill definition, not an architectural change. However, it requires understanding the cross-referencing contract between dispatch files, docs, agent definitions, and skill files. The guidance must be precise enough to catch renames without false-positiving on additions. Related issues (#890, #852, #1060) suggest coordinated implementation could be beneficial.

    Formula: (1.5 x 1.5 x 0.8) / 1 = 1.8

  4. ralphbean commented on Sep 21, 2026

    @ralphbean
    Member

    /fs-triage lots of things have changed between when this was originally filed and now across both the https://github.com/fullsend-ai/fullsend and https://github.com/fullsend-ai/agents repos. Is this still relevant? This is a stale-issue review, not an implementation request: even if the issue is still a valid bug and has sufficient information, do not recommend or trigger ready-to-code yet. Leave it for human prioritization after confirming whether it remains relevant. Do not close a still-relevant bug merely to avoid coding it.

  5. fullsend-ai-triage commented on Sep 21, 2026

    @fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 4:54 PM UTC · Completed 4:59 PM UTC

    Commit: eb8149a · View workflow run →

    Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $0.45

  6. added
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    and removed
    triagedTriaged but awaiting human prioritization
    on Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/reviewReview agentcomponent/harnessAgent harness, config, and skills loadingfeatureFeature-category issue awaiting human prioritizationtriagedTriaged but awaiting human prioritization

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions