Skip to content

Review agent should check CI status before approving PRs #369

Closed as not planned

Description

@ralphbean

What happens

The review agent currently approves PRs without checking whether CI status checks are passing. This can result in approving changes that fail tests, linting, or other automated verification.

Example: konflux-ci/oras-container#295 (review) — the review agent approved this PR while CI was failing.

What should happen

Before approving a PR, the review agent should check the status of all CI checks and factor that into its review decision.

Potential approaches:

  1. Block approval on any failing check: Refuse to approve if any status check is failing
  2. Distinguish required vs. optional: Only block on required checks (per branch protection rules)
  3. Approve with caveat: Approve the code quality but add a comment noting failing checks
  4. Comment only: Don't approve, but post review comments noting both code assessment and CI status

Pending/running checks: If checks haven't completed yet (pending or in-progress), the review agent should acknowledge this state and either:

  • Wait for checks to complete before reviewing
  • Review code quality but note that CI verification is incomplete
  • Skip review entirely until CI provides signals

Fork PR edge case: When CI requires manual approval to run (e.g., on fork PRs), the agent should note that CI status is unavailable rather than treating it as passing.

Context

This is an enhancement for the review agent, not a blocker for initial deployment. However, it's important for preventing the agent from giving false signals about PR readiness.

Related work:

Activity

  1. deboer-tim commented on Apr 28, 2026

    @deboer-tim

    This happens fairly often with human reviewers in Podman Desktop. If the tests are still running then often a reviewer will just approve and rely on required tests passing. If a minor check has failed then humans will often approve with a comment - sometimes even if the check is not required, and trusting that the person will fix. If the failure was major, some will do a review already; others won't even start.

    My point is mostly about conventions - teams may have different expectations. Fullsend reviewer doesn't use up human time but may want to wait until PR is ready to save $. So may need to be configurable or pick up conventions from the repo.

  2. fullsend-ai-triage commented on May 5, 2026

    @fullsend-ai-triage

    RICE Priority Score: 0.8

    Score breakdown
    Dimension Score Reasoning
    Reach 1.5 This affects all users of the review agent. The example is from konflux-ci (strategic customer), and deboer-tim (openkaiden, another strategic customer) commented with relevant feedback about team conventions. Two of three strategic customers are directly engaged with this issue.
    Impact 1 The review agent approving PRs with failing CI gives false signals about PR readiness, but it's not blocking — CI results are still visible and human reviewers can catch issues. The issue author explicitly notes this is an enhancement, not a blocker. The improvement is noticeable but workarounds exist.
    Confidence 0.8 Well-described issue with a concrete example PR, clear problem statement, multiple solution approaches outlined, and substantive community feedback from a strategic customer. The scope is clear, though the design decision (which approach to take, configurability needs raised by deboer-tim) adds some uncertainty.
    Effort 1.5 Requires checking CI status via GitHub API, handling multiple edge cases (pending checks, fork PRs, required vs optional checks), and likely needs configurability per deboer-tim's feedback about team conventions. Touches the review agent's approval logic and may need design decisions before implementation.

    Formula: (1.5 × 1 × 0.8) / 1.5 = 0.8

  3. maruiz93 commented on Jun 25, 2026

    @maruiz93
    Contributor

    An example of this: konflux-ci/tekton-integration-catalog#185

    The review agent says LGTM but the CI is failing.

  4. ralphbean commented on Sep 8, 2026

    @ralphbean
    MemberAuthor

    /fs-triage lots of things have changed between when this was originally filed and now across both the https://github.com/fullsend-ai/fullsend and https://github.com/fullsend-ai/agents repos. Is this still relevant?

  5. fullsend-ai-triage commented on Sep 8, 2026

    @fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 9:34 PM UTC · Completed 9:40 PM UTC

    Commit: e4749da · View workflow run →

    Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.98

  6. added
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    and removed
    type/ideaExploratory idea or new problem area
    on Sep 8, 2026
  7. fullsend-ai-triage commented on Sep 8, 2026

    @fullsend-ai-triage

    This feature request is still relevant. Investigation of the current review agent implementation in both fullsend-ai/fullsend and fullsend-ai/agents confirms that no CI status checking has been added — the review agent's sub-agent roster (correctness, security, intent-coherence, style-conventions, docs-currency, cross-repo-contracts, risk-assessment, challenger) has no CI dimension, and neither the pre-review nor post-review scripts query check status.

    Related work

    Several narrower issues track specific aspects of this problem:

    All remain open with no PRs addressing them.

    Implementation notes

    The post-review.sh script already has a pattern for downgrading approve → comment (used for protected-path enforcement). A similar gate for CI status could be added there as a defense-in-depth layer. For richer integration, a CI-status sub-agent or pre-check step in the pr-review orchestrator could feed CI signals into the review synthesis.

    Per community feedback, the behavior should likely be configurable per repo to accommodate different team conventions around CI expectations.

    # Proposed test cases (conceptual)
    def test_approve_downgraded_when_required_check_fails():
        """Verdict should not be 'approve' when a required CI check is failing."""
        pr = mock_pr(checks=[{"name": "tests", "status": "FAILURE", "required": True}])
        result = run_review(pr)
        assert result["action"] != "approve"
        assert "CI" in result["body"] or "check" in result["body"].lower()
    
    def test_pending_checks_acknowledged():
        """Review should note when CI checks are still running."""
        pr = mock_pr(checks=[{"name": "tests", "status": "PENDING", "required": True}])
        result = run_review(pr)
        assert "pending" in result["body"].lower() or "in progress" in result["body"].lower()
    
    def test_fork_pr_ci_unavailable():
        """Fork PRs where CI hasn't been triggered should note unavailable status."""
        pr = mock_pr(checks=[], is_fork=True)
        result = run_review(pr)
        assert "unavailable" in result["body"].lower() or "not run" in result["body"].lower()

    Labels: Removing type/idea since the issue has been validated by multiple users with concrete examples and is a confirmed feature gap, not an exploratory idea.


    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
  8. deboer-tim commented on Sep 9, 2026

    @deboer-tim

    FYI we had some discussion on the contributor call today, and @mrizzi and I chatted offline.

    Review agent can be extended to look at additional aspects (my naive assumption is that would be generally more efficient than multiple review agents), but if we're building a mostly-automated platform the fundamental behaviour difference is between 'smart coder' or 'smart reviewer' - or more simply 'who is responsible for the next action'?

    On most of the teams I work with, review/CI are asynchronous and the coder is fully responsible for getting the PR in an approved state - deciding how to respond to all feedback including failed PR checks. Hence issues like #642, #5350, fullsend-ai/agents#399, and fullsend-ai/agents#565 - i.e. code/fix automatically reacting by itself.

    On Marco's team the review(bot) does more: essentially waiting to look at all sources of feedback, summarizing, and then triggering the coder with more specific instructions. This isn't just configuration of one agent, but a difference in responsibilities between the three current agents.

    I think this merits discussion as we should either decide to only support one path (which would cancel a whole class of issues) or this is a new multi-agent 'who is responsible for reacting when X' config requirement.

  9. ifireball commented on Sep 17, 2026

    @ifireball
    Member

    See fullsend-ai/agents#1333 - we are making the fix agent also fix CI. Also further discussion on agent should go in the agents repo. Closing this here for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/reviewReview agentfeatureFeature-category issue awaiting human prioritizationtriagedTriaged but awaiting human prioritizationtype/featureNew capability request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions