Skip to content

docs(agents): require direct review-thread replies - #1679

Closed
ggallen wants to merge 1 commit into
fullsend-ai:mainfrom
ggallen:docs/review-thread-replies
Closed

ggallen wants to merge 1 commit into
fullsend-ai:mainfrom
ggallen:docs/review-thread-replies

Conversation

@ggallen

@ggallen ggallen commented Oct 14, 2026

Copy link
Copy Markdown
Member

Summary

Require agents to reply to existing GitHub inline review threads through the dedicated REST reply endpoint using the top-level comment ID. Resolve threads separately only after verifying the finding is fixed, and leave unfixed threads open.

The instruction also avoids explicit review creation/submission and the GraphQL thread-reply mutation, which produced separate COMMENTED review records during PR #8441 cleanup.

Verification

  • make lint passed.
  • Commit is DCO signed off.

Signed-off-by: Greg Allen <gallen@redhat.com>
@ggallen
ggallen requested a review from a team October 14, 2026 23:43
@ggallen

ggallen commented Oct 14, 2026

Copy link
Copy Markdown
Member Author

Opened in error; this was intended as a local persistent instruction, not a pull request.

@ggallen ggallen closed this Oct 14, 2026
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Require direct replies to existing GitHub review threads

📝 Documentation 🕐 Less than 10 minutes

Grey Divider

AI Description

• Direct agents to reply through GitHub’s REST endpoint using the top-level review comment ID.
• Require fix verification before replying, then resolve threads separately; leave unfixed threads
 open.
• Prohibit review submission and the GraphQL thread-reply mutation to avoid separate review records.
Diagram

graph TD
  A["Existing thread"] --> B{"Finding fixed?"}
  B -->|Yes| C["REST reply"] --> D["Resolve thread"]
  B -->|No| E["Leave open"]
Loading
High-Level Assessment

Keep the focused instruction in AGENTS.md. It specifies the intended GitHub endpoint and separates replying from resolution; review creation and the GraphQL mutation are explicitly unsuitable for this workflow.

Files changed (1) +10 / -0

Documentation (1) +10 / -0
AGENTS.mdSpecify how agents reply to existing review threads +10/-0

Specify how agents reply to existing review threads

• Adds a rule requiring the REST thread-reply endpoint with the top-level comment ID. Agents must verify the fix, resolve fixed threads separately, leave unfixed threads open, and avoid review submission or the GraphQL reply mutation.

AGENTS.md

@ggallen
ggallen deleted the docs/review-thread-replies branch October 14, 2026 23:43
@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (1) 📜 Skill insights (0)

Grey Divider


Action required

1. Review bots cannot reply to threads 🐞 Bug ≡ Correctness
Description
The new instruction requires agents to perform a writable GitHub REST mutation, but the review agent
only has read-only REST and GraphQL permissions. Whenever the review agent handles a fixed existing
thread, it cannot use the mandated endpoint from its sandbox, and the post-review path still
publishes through the generic review operation instead.
Code

AGENTS.md[R173-176]

+When responding to an existing inline pull request review thread, always use
+`POST /repos/{owner}/{repo}/pulls/{pull_number}/comments/{comment_id}/replies`
+with the top-level review comment ID. Do not use a review creation/submission
+operation or the GraphQL `addPullRequestReviewThreadReply` mutation for thread
Relevance

●●● Strong

Accepted precedents address missing write permissions and documentation mismatches around GitHub
comment publishing.

PR-#38
PR-#766

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new policy requires a POST mutation for every existing-thread reply. The review agent
documentation says its GitHub permissions are read-only, while the post-review script exposes only a
generic fullsend post-review publisher at the workflow boundary; no direct-reply operation is
provided in the repository.

AGENTS.md[173-179]
agents/review.md[235-242]
scripts/post-review.sh[123-129]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

Issue description
The new repository instruction mandates direct REST replies to existing GitHub review threads, but the review agent runs with read-only forge permissions and the post-review workflow publishes through the generic review operation. As a result, the required behavior cannot be performed by the agent that is expected to follow this instruction.

Fix Focus Areas
- AGENTS.md[173-179]
- agents/review.md[235-242]
- scripts/post-review.sh[123-129]

Recommended Fix
Implement the direct-reply and separate-thread-resolution behavior in the trusted post-review workflow, passing the top-level comment ID and reply body from validated agent output, or explicitly scope this instruction to an agent with a writable GitHub API path. Ensure unfixed threads remain open and avoid creating or submitting pending reviews for replies.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Remediation recommended

2. Thread replies skew review audits 🔗 Cross-repo conflict ≡ Correctness
Description
The new guidance makes agents reply through the direct REST endpoint, which GitHub may record as a
COMMENTED review, while autonomy-analysis classifies every review authored by
fullsend-ai-review[bot] as a verdict. Those reply records can be paired with the latest finding
and counted by the verdict-mismatch audit, distorting review measurements for PRs using this
workflow.
Code

AGENTS.md[R173-176]

+When responding to an existing inline pull request review thread, always use
+`POST /repos/{owner}/{repo}/pulls/{pull_number}/comments/{comment_id}/replies`
+with the top-level review comment ID. Do not use a review creation/submission
+operation or the GraphQL `addPullRequestReviewThreadReply` mutation for thread
Relevance

●●● Strong

Recent review-state precedents show the team accepts fixes preventing downstream review records from
distorting verdict projections.

PR-#1636

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The PR instructs agents to use the direct inline-comment reply endpoint and explicitly acknowledges
that GitHub may associate the reply with a COMMENTED review. The downstream collector classifies
bot-authored reviews solely by login, and the audit pairs every such record with a finding and
evaluates its state, so a reply-created record can enter verdict analysis.

agents -> autonomy-analysis
AGENTS.md[173-179]
External repo: fullsend-ai/autonomy-analysis, skills/gather-review-observations/scripts/fetch-pr-activity.sh [48-54]
External repo: fullsend-ai/autonomy-analysis, skills/audit-pr-review-gap/scripts/check-verdict-mismatch.py [81-122]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Direct GitHub thread replies may create `COMMENTED` review records, but autonomy-analysis currently classifies every review from `fullsend-ai-review[bot]` as a review verdict. This can make a reply appear to be an additional verdict and can produce incorrect verdict-mismatch audit results.

## Fix Focus Areas
- skills/gather-review-observations/scripts/fetch-pr-activity.sh[48-54]
- skills/audit-pr-review-gap/scripts/check-verdict-mismatch.py[81-122]

## Recommended Fix
Update autonomy-analysis to identify and exclude review records created as a consequence of direct inline-thread replies, or otherwise distinguish actual post-review verdicts from `COMMENTED` reply records before classifying them as `review_agent_verdict`. Add a fixture covering a bot-authored direct thread reply that produces a `COMMENTED` review and verify it does not create an extra verdict or mismatch pair.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Grey Divider

Context sources
✅ Compliance rules (platform): 57 rules
✅ Skills: 4 invoked
  code-review
  code-implementation
  pr-review
  docs-review
✅ Cross-repo context — repo relationships
  Explored: repo: fullsend-ai/autonomy-analysis (sha: 46717359) — View relationship
  Explored: repo: fullsend-ai/fullsend (sha: 02438e1c) — View relationship
Review mode: Auto: 🚀 Fast: Localized documentation guidance with no runtime behavior or high-risk logic.

Grey Divider

Tip of the day
💡 Did you know, you can switch off images and animations for a plain-text comment

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread AGENTS.md
Comment on lines +173 to +176
When responding to an existing inline pull request review thread, always use
`POST /repos/{owner}/{repo}/pulls/{pull_number}/comments/{comment_id}/replies`
with the top-level review comment ID. Do not use a review creation/submission
operation or the GraphQL `addPullRequestReviewThreadReply` mutation for thread

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Review bots cannot reply to threads 🐞 Bug ≡ Correctness

The new instruction requires agents to perform a writable GitHub REST mutation, but the review agent
only has read-only REST and GraphQL permissions. Whenever the review agent handles a fixed existing
thread, it cannot use the mandated endpoint from its sandbox, and the post-review path still
publishes through the generic review operation instead.
Agent Prompt
Issue description
The new repository instruction mandates direct REST replies to existing GitHub review threads, but the review agent runs with read-only forge permissions and the post-review workflow publishes through the generic review operation. As a result, the required behavior cannot be performed by the agent that is expected to follow this instruction.

Fix Focus Areas
- AGENTS.md[173-179]
- agents/review.md[235-242]
- scripts/post-review.sh[123-129]

Recommended Fix
Implement the direct-reply and separate-thread-resolution behavior in the trusted post-review workflow, passing the top-level comment ID and reply body from validated agent output, or explicitly scope this instruction to an agent with a writable GitHub API path. Ensure unfixed threads remain open and avoid creating or submitting pending reviews for replies.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Comment thread AGENTS.md
Comment on lines +173 to +176
When responding to an existing inline pull request review thread, always use
`POST /repos/{owner}/{repo}/pulls/{pull_number}/comments/{comment_id}/replies`
with the top-level review comment ID. Do not use a review creation/submission
operation or the GraphQL `addPullRequestReviewThreadReply` mutation for thread

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Thread replies skew review audits 🔗 Cross-repo conflict ≡ Correctness

The new guidance makes agents reply through the direct REST endpoint, which GitHub may record as a
COMMENTED review, while autonomy-analysis classifies every review authored by
fullsend-ai-review[bot] as a verdict. Those reply records can be paired with the latest finding
and counted by the verdict-mismatch audit, distorting review measurements for PRs using this
workflow.
Agent Prompt
## Issue description
Direct GitHub thread replies may create `COMMENTED` review records, but autonomy-analysis currently classifies every review from `fullsend-ai-review[bot]` as a review verdict. This can make a reply appear to be an additional verdict and can produce incorrect verdict-mismatch audit results.

## Fix Focus Areas
- skills/gather-review-observations/scripts/fetch-pr-activity.sh[48-54]
- skills/audit-pr-review-gap/scripts/check-verdict-mismatch.py[81-122]

## Recommended Fix
Update autonomy-analysis to identify and exclude review records created as a consequence of direct inline-thread replies, or otherwise distinguish actual post-review verdicts from `COMMENTED` reply records before classifying them as `review_agent_verdict`. Add a fixture covering a bot-authored direct thread reply that produces a `COMMENTED` review and verify it does not create an extra verdict or mismatch pair.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 11:45 PM UTC · Completed 11:50 PM UTC

Commit: d47384b · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $0.94

@fullsend-ai-review

Copy link
Copy Markdown

Review skipped — this PR is already closed.

The /fs-review command only reviews open PRs/MRs.

Posted by fullsend pre-review check

@fullsend-ai-review

Copy link
Copy Markdown

Review skipped — this PR is already closed.

The /fs-review command only reviews open PRs/MRs.

Posted by fullsend post-review check

This branch was previously deployed

1 inactive deployment
dev — d47384be Deployed Oct 14, 2026 by ggallen via behaviour #797
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant