Skip to content

Add create-check-run safe output for multi-agent PR analysis #33597

Closed

Description

@strawgate

Goal is to let workflows surface their progress and result as a GitHub [Check Run](https://docs.github.com/en/rest/checks/runs) on the triggering commit/PR. Specifically:

Check Runs are NOT PR reviews, a single user account / GitHub token can create N check runs and can decide on the pass/fail state of each check run and can provide summary/details for the check runs.

  • On workflow startup: post a check run in in_progress status (e.g. "Agent running…"). This may be difficult and may not be necessary as the Agent itself will show as in progress anyway.
  • On workflow completion: create/update that check run with a conclusion of success / failure / neutral, plus a title, summary, and text/details produced by the agent.

This gives users a first-class signal in the PR checks UI and on commits, distinct from issue/PR comments. PR Review Comments are restricted to the PR diff but check runs can report any problem allowing users to run several agents against a PR each verifying certain important criteria is met.

Sketch of the config

safe-outputs:
  create-check-run:
    name: "My Agent"   # check run name shown in the UI
    max: 1             # only one check run per workflow run

It might be desirable to allow the agent to specify the check run name (especially if we're to support having max > 1.

safe output emits a single JSONL line, e.g.:

{"type": "create_check_run", "conclusion": "failure", "title": "3 issues found", "summary": "...", "text": "..."}

Permissions needed: checks: write (+ contents: read for SHA resolution) + pull_request: read for seeing title/body/comments, etc.

Implementation notes

Pattern would follow create_issue (~7 files):

  • Schema entry in pkg/parser/schemas/main_workflow_schema.json
  • Config parser in pkg/workflow/safe_outputs_config.go
  • Handler registration in pkg/workflow/compiler_safe_outputs_handlers.go
  • Permissions in pkg/workflow/safe_outputs_permissions.go
  • Runtime handler at actions/setup/js/create_check_run.cjs
  • Plus the new pre-agent + conclusion-job plumbing for the bidirectional state

Activity

  1. changed the title [-]Add `create-check-run` safe output for workflow status reporting[/-] [+]Add `create-check-run` safe output for multi-agent PR analysis[/+] on May 20, 2026
  2. strawgate commented on May 20, 2026

    @strawgate
    ContributorAuthor

    Examples of what check_runs look like on a PR from my pr-rocket tool which runs many agents and reports each agent's findings as individual check runs:
    Image

    Image
  3. locked and limited conversation to collaborators on May 20, 2026
  4. unlocked this conversation on May 20, 2026
  5. locked and limited conversation to collaborators on May 20, 2026
  6. unlocked this conversation on May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions