Skip to content

fix(host-service): count PRs a merge queue closed as merged - #9712

Open
joshrchambers wants to merge 1 commit into
ParadevAI:mainfrom
joshrchambers:graphite-merged-prs
Open

joshrchambers wants to merge 1 commit into
ParadevAI:mainfrom
joshrchambers:graphite-merged-prs

Conversation

@joshrchambers

@joshrchambers joshrchambers commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • The host now counts a PR as merged when GitHub shows it closed but its squashed commit is on the base branch. Graphite's merge queue merges this way.
  • With this, the usage and leaderboard counts of merged agent PRs, and the cost per PR, include PRs merged through Graphite.

Why / Context

Graphite's merge queue runs CI on a draft PR, pushes the squashed commit to main itself, then closes the original PR. GitHub never sets merged_at on either PR. The host stored these PRs as closed, and countAgentPrsByDay (which reads pull_requests.merged_at) left them out.

Real example, remilabs/lidar-server#309:

  • GitHub shows it CLOSED, merged: false, closed by graphite-app.
  • Its draft CI PR, chore (desktop): 1hr to 4hr autoupdate notification #313, is also closed and not merged.
  • main has bcca8af Optionally keep a small roof face… (#309).
  • GET /commits/bcca8af/pulls returns nothing, so GitHub doesn't link the commit to the PR.

How It Works

  • New utils/landed-merge/: for a closed, unmerged PR, lists commits on its base branch from 3 h before the close to 10 min after. A commit whose subject line ends in (#<number>) means the PR landed. The commit's time becomes mergedAt.
  • PR refresh: runs that lookup for closed, unmerged PRs and stores a match as merged. Each PR is looked up once per host process. A row already stored merged is not looked up again, and stays merged when GitHub returns CLOSED (GitHub never turns a merged PR into a closed one).
  • Startup pass: at host start, checks linked PRs from the last 30 days that are stored closed with no mergedAt. This covers rows stored before this change and PRs of archived workspaces, which the refresh skips.
  • github-query now reads closed_at and base.ref into the PR node, as optional fields.
  • Lookups go through fetchFromGitHub (gh first, then Octokit, behind the availability gate). A failed lookup logs a warning and leaves the PR closed.

Manual QA

Testing

  • bun run typecheck (host-service): passes
  • bunx biome check packages/host-service/src/runtime/pull-requests: clean
  • bun test src/runtime/pull-requests src/trpc/router/usage/history: 198 pass
  • New tests:
    • a Graphite-style closed PR becomes merged, with only one commits lookup over two refreshes
    • a revert commit that mentions the PR doesn't count
    • the startup pass marks a PR of an archived workspace as merged
    • subject-line matching (rejects (#3090) and a mention in the message body)
  • Full bun test in host-service: 2884 pass, 1 fail. The failure is workspace PR history > history accumulates across branch moves, an existing timing flake: after an unlink, history is sorted by linkedAt alone, and on a slow full run both links can get the same millisecond. It passes 8/8 run on its own, and the new code doesn't run for the open PRs it uses.

Known Limitations

  • Only commits whose subject ends in (#N) are found (squash merges, which is what Graphite does). A merge whose commit has a different subject won't be counted.
  • One page of 100 commits per lookup. On a base branch with more than 100 commits in the 3 h window, a PR could be missed.
  • These PRs now show as merged instead of closed in the host's own PR views. Views that read GitHub directly still show closed.

Risks / Rollout

  • Risk: low. The change only adds GitHub calls for closed, unmerged PRs (one per PR per host process; the conditional ETag cache makes repeat calls cheap), and only ever turns closed into merged.
  • Rollback: revert this PR. Rows already marked merged stay merged, which is correct for PRs that really landed.

Summary by cubic

Fixes PR counting so PRs closed by a merge queue still count as merged in usage and leaderboard stats, even though GitHub never sets merged_at.

For closed, unmerged PRs, the host now checks the base branch for a commit whose subject ends in (#<PR number>) near the close time and stores the PR as merged with that commit's time. A one-time pass at host start covers closed PRs from the last 30 days, including those of archived workspaces. Each PR is looked up once per host process, and failed lookups leave the PR closed.

  • Lookups only match squash-merge subjects; other merge styles won't be counted.
  • A stored merged PR stays merged when GitHub reports it closed, since GitHub never reverses a merge.

Written for commit 0d0d488. Summary will update on new commits.

View guided diff

Summary by CodeRabbit

  • Bug Fixes
    • Closed pull requests that were merged through the repository’s merge queue are now recognized as merged, with the landed commit time recorded when available.
    • Existing merged status is preserved when GitHub reports a pull request as closed.
    • Recent closed pull requests linked to archived workspaces are also checked during startup backfill.

Graphite's merge queue pushes the squashed commit to the base branch
itself and then closes the PR, so GitHub never sets merged_at. The host
stored these PRs as closed, and the usage and leaderboard counts of
merged agent PRs left them out.

For a closed, unmerged PR, look for a commit on its base branch whose
subject ends in "(#<number>)" near the close time. When one is found,
store the PR as merged with that commit's time. A one-time pass at host
start checks closed PRs from the last 30 days, including PRs of archived
workspaces, which the refresh skips.
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Walkthrough

The runtime now identifies landed merge commits for closed pull requests without a GitHub merge timestamp. It records matching pull requests as merged during refresh and startup backfill, using the commit date as the merge time.

Changes: Landed merge detection

Layer / File(s) Summary
Pull request data and landed-commit lookup
packages/host-service/src/runtime/pull-requests/utils/github-query/*, packages/host-service/src/runtime/pull-requests/utils/landed-merge/*
Normalized pull request data now includes the close time and base branch. New lookup utilities retrieve closed pull request details and search base-branch commits for a subject ending in the exact pull request number.
Refresh and startup backfill
packages/host-service/src/runtime/pull-requests/pull-requests.ts, packages/host-service/src/runtime/pull-requests/pull-requests.test.ts
Refresh checks for landed commits and records matching pull requests as merged. Startup backfills linked closed pull requests updated within 30 days. Tests cover matching subjects, non-matching revert subjects, lookup caching, and backfill for an archived workspace.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant PullRequestRuntimeManager
  participant PullRequestRows
  participant GitHub
  PullRequestRuntimeManager->>PullRequestRows: Select linked closed pull requests updated within 30 days
  PullRequestRuntimeManager->>GitHub: Retrieve pull request and base-branch commits
  GitHub-->>PullRequestRuntimeManager: Return close details and commit data
  PullRequestRuntimeManager->>PullRequestRows: Update matching closed pull requests with landed merge time
Loading

Suggested reviewers: owieschon

Merge Risk: Low · up to 0d0d4

Merge-queue PRs are now counted as merged in the common case. In rare cases, such as GitHub being unreachable at startup or a very busy base branch, a PR can stay shown as closed until the host restarts. This is bounded and recoverable, so the change is mergeable with owner awareness.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title uses conventional commit format and names the main change: the host-service now counts PRs closed by a merge queue as merged. It matches the changeset, which detects landed squash-merge comm…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description covers what changed and why (Graphite merge queue closes PRs without setting merged_at, so usage and leaderboard counts miss them), gives a concrete real-world example, explains how th…
Full details: Title check

Explanation

The title uses conventional commit format and names the main change: the host-service now counts PRs closed by a merge queue as merged. It matches the changeset, which detects landed squash-merge commits for closed, unmerged PRs and stores them as merged.

Full details: Description check

Explanation

The description covers what changed and why (Graphite merge queue closes PRs without setting merged_at, so usage and leaderboard counts miss them), gives a concrete real-world example, explains how the lookup, refresh, and startup backfill work, and documents manual QA, automated test results, known limitations, and rollout risk. The template's "What & why" and "How I tested it" sections are both substantively filled. The checklist items are not ticked and the PR title is not in the description, but the title is shown in the objectives as conventional commit format. Those are non-critical gaps for a PASS.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional comments: 1

packages/host-service/src/runtime/pull-requests/pull-requests.ts-1656-1656: 1

1656-1656: Major · Functional Correctness · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Expire negative landed-merge results.

If the first lookup runs before a matching commit becomes available on the base branch, it caches null. Even if that commit arrives within the supported ten-minute lookahead, every later lookup returns the cached result. Explicit refresh also cannot retry it.

Give negative results a retry deadline or invalidate them on explicit refresh. Keep in-flight deduplication and successful matches cached.

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3a2f0a4c-3666-4a5e-bb13-38298ae39214
Commits

Reviewing files that changed from the base of the PR and between 340f4ae and 0d0d488.

Files selected for processing: 8

  • packages/host-service/src/runtime/pull-requests/pull-requests.test.ts
  • packages/host-service/src/runtime/pull-requests/pull-requests.ts
  • packages/host-service/src/runtime/pull-requests/utils/github-query/github-query.test.ts
  • packages/host-service/src/runtime/pull-requests/utils/github-query/github-query.ts
  • packages/host-service/src/runtime/pull-requests/utils/github-query/types.ts
  • packages/host-service/src/runtime/pull-requests/utils/landed-merge/index.ts
  • packages/host-service/src/runtime/pull-requests/utils/landed-merge/landed-merge.test.ts
  • packages/host-service/src/runtime/pull-requests/utils/landed-merge/landed-merge.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

@joshrchambers

Copy link
Copy Markdown
Contributor Author

@harshithmullapudi this fixes the merged agent PR count for repos that merge through Graphite's merge queue. Graphite closes the PR instead of merging it, so these PRs were left out of the count and the cost per PR. Tested against real Graphite-merged PRs; details are in the description. Happy to adjust the approach.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant