Skip to content

Do not repeat previous findings #405

Closed

Description

@strawgate

We should give Agents stronger instructions for avoiding filing the same issues twice.

We should have a step (maybe a safe-output?!) that the Agent can run to get issue numbers and titles of the last ~100 suggestions made by that agent to help ensure that the Agent doesn't repeat the same suggestions again.

Let's think through a good way to do this without adding specific instructions to every workflow (a fragment maybe? how can we insert the title prefix for each agent into the script? etc)

Activity

  1. github-actions commented on Feb 25, 2026

    @github-actions
    Contributor
    3 related issues

    What is this? | From workflow: Duplicate Issue Detector

    Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

  2. github-actions commented on Feb 25, 2026

    @github-actions
    Contributor

    TL;DR: the lowest-risk path is to add a single reusable duplicate preflight to gh-aw-fragments/safe-output-create-issue.md (which all create-issue workflows already import), and use each workflow’s title-prefix to fetch/review the last ~100 prior reports before creating a new issue.

    Recommendation

    Implement duplicate prevention centrally in the shared safe-output-create-issue fragment, not per-workflow prompts, then enforce a consistent title-prefix for every workflow that can call create_issue.

    This gives one place to maintain the behavior, matches your “no per-workflow duplication” goal, and uses tools already available (github-search_issues + pagination) without waiting on new MCP APIs.

    Findings
    • create-issue guidance is centralized but currently only documents API limits, not duplicate-preflight behavior:
      • .github/workflows/gh-aw-fragments/safe-output-create-issue.md:1-14
    • Duplicate checks are currently inconsistent and workflow-specific (some have explicit queries, many do not):
      • .github/workflows/gh-aw-agent-suggestions.md:92-96 (explicit duplicate search)
      • .github/workflows/gh-aw-flaky-test-investigator.md:99-102 (explicit duplicate check)
      • .github/workflows/gh-aw-branch-actions-detective.md:114-118 (prefix-based check)
    • Post-facto duplicate handling exists, but it comments after issue creation rather than preventing new duplicate filings:
      • .github/workflows/gh-aw-duplicate-issue-detector.md:72-73, 121-127
    • Coverage evidence from repo scan:
      • 20 workflows can create issues.
      • 12 include explicit duplicate guidance; 8 do not.
      • 19/20 define safe-outputs.create-issue.title-prefix; gh-aw-plan.md is the only missing one.
      • All 20/20 create-issue workflows import gh-aw-fragments/safe-output-create-issue.md.
    Verification

    I ran repository scans to verify current behavior:

    $ python - <<'PY'
    from pathlib import Path
    root=Path('/home/runner/work/ai-github-actions/ai-github-actions/.github/workflows')
    files=sorted(root.glob('gh-aw-*.md'))
    create=[]
    with_dup=[]
    for f in files:
        t=f.read_text()
        if 'create-issue:' in t:
            create.append(f.name)
            if 'github-search_issues' in t or 'Check for duplicates' in t or 'title prefix' in t:
                with_dup.append(f.name)
    print('create_issue_workflows',len(create))
    print('with_duplicate_guidance',len(with_dup))
    print('without_duplicate_guidance',len(create)-len(with_dup))
    PY
    create_issue_workflows 20
    with_duplicate_guidance 12
    without_duplicate_guidance 8
    $ python - <<'PY'
    from pathlib import Path
    root=Path('/home/runner/work/ai-github-actions/ai-github-actions/.github/workflows')
    missing=[]
    for f in sorted(root.glob('gh-aw-*.md')):
        t=f.read_text()
        if 'create-issue:' in t and 'title-prefix:' not in t:
            missing.append(f.name)
    print('missing_title_prefix',len(missing))
    print('\n'.join(missing))
    PY
    missing_title_prefix 1
    gh-aw-plan.md
    $ python - <<'PY'
    from pathlib import Path
    root=Path('/home/runner/work/ai-github-actions/ai-github-actions/.github/workflows')
    missing=[]
    for f in sorted(root.glob('gh-aw-*.md')):
        t=f.read_text()
        if 'create-issue:' in t and 'gh-aw-fragments/safe-output-create-issue.md' not in t:
            missing.append(f.name)
    print('create_issue_without_fragment',len(missing))
    PY
    create_issue_without_fragment 0
    Detailed Action Plan
    1. Add centralized duplicate-preflight instructions in:

      • .github/workflows/gh-aw-fragments/safe-output-create-issue.md
      • Add a required preflight section before any create_issue call:
        • derive prefix from safe-outputs.create-issue.title-prefix (when present)
        • run github-search_issues against open+closed issues with in:title "(prefix)"
        • paginate (perPage: 20-30) until ~100 recent items are reviewed
        • if the proposed finding is already tracked, call noop instead of create_issue
        • if creating, include a short “Duplicate checks” block in the issue body listing checked issue numbers
    2. Close the one metadata gap by adding title-prefix to:

      • .github/workflows/gh-aw-plan.md (safe-outputs.create-issue block at lines 73-77)
      • This makes preflight deterministic for all issue-creating workflows.
    3. Codify a strict matching rule in the same fragment:

      • “Same root cause/same requested change + overlapping scope => duplicate, use noop”
      • “Shared area but different actionable change => not duplicate, proceed.”
    4. Optionally harden with a future MCP helper (separate follow-up):

      • Add a safe-output read helper that returns last N issues for the current workflow prefix.
      • This can replace manual github-search_issues query construction later, but is not required for immediate rollout.
    5. Validation pass:

      • make compile
      • make lint
      • Spot-check at least one workflow with and one without previous explicit duplicate guidance to confirm fragment-level behavior is uniformly present.
    Related Items
    Type Link Relevance
    Issue #405 Request to avoid repeating prior findings
    File .github/workflows/gh-aw-fragments/safe-output-create-issue.md:1-14 Central place to add duplicate-preflight behavior
    File .github/workflows/gh-aw-plan.md:73-77 Only create-issue workflow block missing title-prefix
    File .github/workflows/gh-aw-agent-suggestions.md:92-96 Existing workflow-level duplicate check pattern
    File .github/workflows/gh-aw-duplicate-issue-detector.md:72-73,121-127 Duplicate detection currently post-creation (comment), not pre-creation guard
    Issue #354 Prior request around report issue lifecycle behavior
    Issue #357 Closed follow-up on issue lifecycle defaults
    PR #355 Made close-older configurable for scheduled reports
    PR #358 Changed defaults for close-older behavior across detector/auditor workflows

    What is this? | From workflow: Issue Triage

    Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

  3. strawgate commented on Feb 26, 2026

    @strawgate
    ContributorAuthor

    /ai let's make title-prefix an input that can be configured in the workflow_call (with a default) and then let's just use that title-prefix var for the issue search script that's part of the fragment? Make sure you run compile and ensure it ends up in the lock file correctly.

    That would work I think

  4. github-actions commented on Feb 26, 2026

    @github-actions
    Contributor

    TL;DR: the exact change should be to add a configurable issue-title-prefix workflow input (with default) in gh-aw-plan, wire it to safe-outputs.create-issue.title-prefix, and use that same input in the duplicate-search instructions in the shared safe-output-create-issue fragment.

    Concrete edits
    1. Add a new workflow_call input and wire title-prefix in .github/workflows/gh-aw-plan.md (current relevant block is at gh-aw-plan.md:20-77):
    # .github/workflows/gh-aw-plan.md
    on:
      workflow_call:
        inputs:
          issue-title-prefix:
            description: "Title prefix for issues created by this workflow"
            type: string
            required: false
            default: "[plan] "
    
    safe-outputs:
      activation-comments: false
      create-issue:
        max: ${{ inputs.create-issue-max }}
        title-prefix: "${{ inputs.issue-title-prefix }}"
    1. Extend .github/workflows/gh-aw-fragments/safe-output-create-issue.md (currently only limitations at safe-output-create-issue.md:1-13) to include duplicate preflight guidance that uses the prefix input:
    ## Duplicate Preflight (required before `create_issue`)
    
    1. Search prior issues by title prefix:
       - `github-search_issues` with query
         `repo:{owner}/{repo} is:issue in:title "${{ inputs.issue-title-prefix }}" sort:updated-desc`
    2. Review at least the most recent ~100 matching issues (paginate with `perPage: 20-30`).
    3. If the same root cause/request already exists (open or closed), do **not** create a new issue; call `noop` and reference the existing issue number(s).
    1. Update consumer docs for plan inputs in gh-agent-workflows/plan/README.md:25-32 to include issue-title-prefix.
    Why this matches your ask
    • gh-aw-plan.md currently has no title-prefix input and no safe-outputs.create-issue.title-prefix (gh-aw-plan.md:20-77).
    • The shared create-issue fragment currently has only API limits and no duplicate-search script/instructions (safe-output-create-issue.md:1-13).
    • This makes the prefix configurable via workflow_call with a default, and then uses that same variable for the duplicate search logic in the shared fragment.
    Verification run

    I ran:

    make compile

    Result: compile completed with the repository’s existing known errors in gh-aw-mention-in-issue-no-sandbox.md and gh-aw-mention-in-pr-no-sandbox.md (threat-detection vs sandbox mismatch), unrelated to this change request.


    What is this? | From workflow: Mention in Issue

    Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions