Skip to content

Monthly report of precision per reviewer and repository, and cost per fixed finding #48

Closed

Description

@polewskm

Background. A scheduled workflow should summarize how useful the bot was each month:

  • precision per reviewer and per repository: fixed ÷ (fixed + dismissed), with acknowledged and ignored shown separately
  • findings per PR
  • total cost split by provider (the Claude API credits vs Bedrock, which is billed to the maintainer's employer)
  • cost per fixed finding

Anthropic's usage-report API doesn't cover Bedrock, so spend must come from the engine's own per-call records. For scale: about $1.25 per review on average, roughly $425/month at 340 PRs, of which about $225 lands on Bedrock past the $200 monthly credits.

Current state. No scheduled workflow (.github/workflows has ci, code-review, release and wif-smoke-test). Per-review cost appears only in each summary comment's text.

What to do.

  1. Add .github/workflows/monthly-report.yml: schedule: cron '0 6 1 * *' plus workflow_dispatch, with permissions: issues: write, contents: read and GITHUB_TOKEN only (all repos are public, so reading comments needs no other token).
  2. tools/report.ts: for each repo where the app is installed, list PRs closed in the previous month, read the outcomes marker (Record each finding's outcome when a pull request closes #47) and the summary comments' cost, and aggregate.
  3. Open one issue, "Review report YYYY-MM", with tables per reviewer and per repo, provider split, cost per fixed finding, and the month-over-month change.

Acceptance. A workflow_dispatch run for the current month opens an issue whose totals match a hand count on two PRs.

Related. Depends on #47.


Filed from the 2026-10-31 audit of the NCodeGroup review-bot and project-template design against what was built. This issue stands alone: everything needed to do it is above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions