Skip to content

Add pre-run file creation for unresolved threads #497

Closed

Description

@strawgate

We have pr review comments in a json file, we should also probably make a file for unresolved threads, resolved threads, etc in a file too so that the agent doesnt have to do it itself.

See https://github.com/elastic/ai-github-actions-playground/actions/runs/22530481954/job/65269085660?pr=809 for the various permutations the agent ended up making

Glob "/tmp/pr-context/issue-*.json"
  └ No matches found
● List unresolved review threads with comments
  $ jq -r '.[] | select(.isResolved==false) | "THREAD\t\(.id)\t\(.path):\(.line // .startLine //
  0)\tOUTDATED=\(.isOutdated)\n" + (.comments[] | "COMMENT\t\(.databaseId)\t\(.author.login)\t" +
  (.body|gsub("\n";" "))) + "\n---"' /tmp/pr-context/review_comments.json
  └ 2 lines...
● Todo started: gather-context
● Read /tmp/pr-context/review_comments.json lines 1-120
  └ 120 lines read
● Inspect review_comments JSON shape
  $ python - <<'PY'
  import json
  p='/tmp/pr-context/review_comments.json'
  obj=json.load(open(p))
  print(type(obj).__name__)...
  └ 14 lines...
● Todo queried:
● List unresolved review threads
  $ jq -r '.[] | select(.isResolved==false) | "THREAD\t\(.id)\t\(.path):\(.line // .startLine //
  0)\tOUTDATED=\(.isOutdated)\n" + (.comments.nodes[] |
  "COMMENT\t\(.databaseId)\t\(.author.login)\t" + (.body|split("\n")[0])) + "\n---"'
  /tmp/pr-context/review_comments.json
  └ 40 lines...
● Print unresolved thread comment details
  $ python - <<'PY'
  import json,textwrap
  p='/tmp/pr-context/review_comments.json'
  arr=json.load(open(p))
  for t in arr:...
  └ 519 lines...
● Create session plan file
  $ cat > /home/runner/.copilot/session-state/a90484f2-d19b-431a-a650-9ca0155e32c1/plan.md <<'EOF'
  # Plan
  1. Inspect unresolved PR review threads and map each to current code.
  2. Implement minimal, safe fixes for clearly actionable issues.
  3. Run targeted tests/lint relevant to touched files....
  └ 3 lines...
● Read peek/scripts/elasticsearch-mocks.mjs
  └ 273 lines read
● Read peek/src/components/cluster-health/clusterHealthUtils.ts
  └ 215 lines read
● Read peek/src/components/cluster-health/NodeDetailTable.tsx
  └ 206 lines read
● Read peek/src/components/cluster-health/OverviewView.tsx
  └ 135 lines read
● Read peek/src/components/cluster-health/RefreshPicker.tsx
  └ 61 lines read
● Read peek/src/components/cluster-health/ResilienceSignalsView.tsx
  └ 121 lines read
● Read peek/src/hooks/useClusterHealthData.ts
  └ 236 lines read
● Read peek/src/routes/manifest.ts
  └ 370 lines read
● Read peek/src/components/cluster-health/TaskBacklogView.tsx
  └ 76 lines read
● Read peek/tests/unit/clusterHealthUtils.test.ts
  └ 248 lines read
● Grep "PRIORITY_ORDER|groupPendingTasks|parseNumber\("
(/home/runner/work/ai-github-actions-playground/ai-github-actions-playground/peek/src)
  └ 7 lines found
● Show full bodies for selected review comments
  $ python - <<'PY'
  import json
  ids={2867974903,2867974917,2867975557}
  arr=json.load(open('/tmp/pr-context/review_comments.json'))
  for t in arr:...
  └ 199 lines...

maybe we should make review comments into files based on the comment id or something? maybe also write every piece of review feedback to its own file based on id too? with threads split per file into resolved and unresolved or something like that?

let's think through all the dimensions we'd want to make this super easy for the Agent to find what it wants

Activity

  1. github-actions commented on Feb 28, 2026

    @github-actions
    Contributor
    3 related issues

    What is this? | From workflow: Duplicate Issue Detector

    Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

  2. github-actions commented on Feb 28, 2026

    @github-actions
    Contributor

    TL;DR: this should be implemented as a small additive change in pr-context—precompute unresolved/resolved thread JSON files so downstream agents stop re-deriving thread state from review_comments.json.

    Recommendation

    Add two pre-run artifacts in .github/workflows/gh-aw-fragments/pr-context.md:

    • /tmp/pr-context/review_threads_unresolved.json (isResolved == false)
    • /tmp/pr-context/review_threads_resolved.json (isResolved == true)

    Keep the object schema identical to entries in review_comments.json (no transformation), keep existing threads/(path).json behavior, and update prompt text to prefer the new files (with fallback to review_comments.json if missing).

    Findings
    • pr-context currently creates only the aggregate thread file and per-file thread files, not status-split aggregates:
      • .github/workflows/gh-aw-fragments/pr-context.md:75
      • .github/workflows/gh-aw-fragments/pr-context.md:78-89
      • .github/workflows/gh-aw-fragments/pr-context.md:120-121
    • Consumers explicitly ask agents to derive unresolved threads from the aggregate file:
      • .github/workflows/gh-aw-pr-review-addresser.md:125
      • .github/workflows/gh-aw-pr-review-addresser.md:148
      • .github/workflows/gh-aw-mention-in-pr.md:174
    • Related prior work already established the broader PR-context artifact model and thread metadata (id, isResolved, isOutdated, databaseId) in merged PR #459; this issue is a natural incremental extension.
    • I did not find evidence that per-comment files are required for current workflow logic; current reply/resolve flows already rely on IDs present in thread objects.
    Verification

    I verified by targeted repository searches:

    No matches for split files in pr-context source:
      pattern: review_threads_(unresolved|resolved).json | threads_unresolved.json | threads_resolved.json
      file: .github/workflows/gh-aw-fragments/pr-context.md
    
    Matches showing unresolved-thread derivation burden:
      .github/workflows/gh-aw-pr-review-addresser.md:125
      .github/workflows/gh-aw-pr-review-addresser.md:148
      .github/workflows/gh-aw-mention-in-pr.md:174
    
    Current thread artifact creation points:
      .github/workflows/gh-aw-fragments/pr-context.md:75
      .github/workflows/gh-aw-fragments/pr-context.md:78
      .github/workflows/gh-aw-fragments/pr-context.md:120
    

    I also ran a 3-way parallel investigation (context generator angle, consumer angle, prior-work/testing angle). All three converged on the same minimal fix direction: precomputed unresolved/resolved artifacts, not a broad redesign.

    Detailed Action Plan
    1. In .github/workflows/gh-aw-fragments/pr-context.md (immediately after writing review_comments.json), add:
      • jq '[.[] | select(.isResolved == false)]' /tmp/pr-context/review_comments.json > /tmp/pr-context/review_threads_unresolved.json
      • jq '[.[] | select(.isResolved == true)]' /tmp/pr-context/review_comments.json > /tmp/pr-context/review_threads_resolved.json
    2. Update the manifest table in the same file to document both new artifacts and intended semantics.
    3. Update consumer prompts to prefer the split files:
      • .github/workflows/gh-aw-pr-review-addresser.md (Step 1 / Step 4 references)
      • .github/workflows/gh-aw-mention-in-pr.md (resolve-thread guidance)
        Include fallback text to review_comments.json filtering.
    4. Recompile generated workflow lock files (make compile) so source and .lock.yml stay in sync.
    5. Validate with make lint and a targeted search for new filenames in source + generated lock files.
    6. Defer per-comment-file generation unless a concrete consumer requirement appears (keeps scope/complexity low and avoids file explosion).
    Related Items
    Type Link Relevance
    Issue #497 Current request
    Issue #494 Adjacent UX concern around PR-context exposure in responses
    Issue #321 Broader PR review feedback automation context
    PR #459 Added/standardized PR-context review-thread data and guidance
    File .github/workflows/gh-aw-fragments/pr-context.md Source of PR-context artifact generation
    File .github/workflows/gh-aw-pr-review-addresser.md Consumer that currently derives unresolved threads at runtime
    File .github/workflows/gh-aw-mention-in-pr.md Consumer with same unresolved-thread derivation pattern

    What is this? | From workflow: Issue Triage

    Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions