Bug Hunter Improvements #185

Activity

1. github-actions commented on Feb 20, 2026

   

   github-actionsBot

   on Feb 20, 2026 – with GitHub Actions

   Contributor

   More actions

   TL;DR: Bug Hunter is not explicitly told to run full test suites, but the current prompt strongly nudges it toward test-driven repro (minimal failing test), so it can spend time re-running tests even when CI is already green; I recommend tightening the prompt to prefer targeted repro and explicitly avoid redundant full-suite runs.

   Recommendation

   Update .github/workflows/gh-aw-bug-hunter.md to keep the reproducibility bar high while reducing unnecessary test execution:

   1. Keep mandatory reproduction, but explicitly prefer the smallest targeted command and avoid rerunning full test suites that are already green unless needed for a new failing signal.
   2. Relax wording from “must be covered by a minimal failing test” to “must have deterministic reproduction (failing check or clearly incorrect output/behavior).”
   3. Add an explicit skip rule for “re-running broad existing test suites with no new failure hypothesis.”

   Findings

   • Current Bug Hunter prompt emphasizes test framing:
     • .github/workflows/gh-aw-bug-hunter.md:79 — requires a bug that can be covered by a minimal failing test.
     • .github/workflows/gh-aw-bug-hunter.md:101 — “Can be expressed as a minimal failing test.”
     • .github/workflows/gh-aw-bug-hunter.md:92-95 — reproduction is mandatory (good, should stay).
   • There is no explicit instruction today to skip redundant full-suite reruns when status is already known/green.
   • Related historical context:
     • #89 and PR #90 focused on duplicate prevention/labeling, not test-execution strategy.
     • #158 clarified commit-history lookback behavior.
     • #135 added “run required commands” guidance for code-changing workflows; Bug Hunter is detection/reporting and should still optimize for focused repro over broad reruns.

   Verification

   $ rg -n "test|tests|passing|failing" .github/workflows/gh-aw-bug-hunter.md
   79: Find a single reproducible, user-impacting bug ... minimal failing test.
   101: - Can be expressed as a minimal failing test ...
   136: > [File path + outline of test]
   
   $ web_fetch https://github.com/elastic/beats/actions/runs/22206903339
   Status Success; artifact list available from run page summary.
   

   I could confirm prompt wording locally; the linked run page summary is accessible, but detailed command logs are not included in that summary fetch.

   Detailed Action Plan

   1. Edit .github/workflows/gh-aw-bug-hunter.md:
      • At :79, reword goal to emphasize deterministic repro with minimal targeted commands (not broad suite reruns).
      • In Data Gathering step 3 (:92-95), add a bullet: if CI/tests are already green for the same scope, do not rerun full suites unless there is a concrete failure hypothesis.
      • In “What to Look For” (:99-102), replace strict “minimal failing test” language with “deterministic failing check or clear behavior mismatch.”
      • In “What to Skip” (:103-109), add “rerunning broad test suites without a new hypothesis.”
   2. Recompile workflow locks (make compile) so .github/workflows/gh-aw-bug-hunter.lock.yml reflects prompt changes.
   3. Run lint/validation (make lint) and spot-check the compiled lock includes the new guidance text.
   4. Observe 2-3 subsequent Bug Hunter runs to verify behavior shifts toward focused repro and fewer redundant test reruns.

   Related Items

   Type	Link	Relevance
   Issue	#185	Current request
   File	.github/workflows/gh-aw-bug-hunter.md:79,92-95,99-109,136	Prompt text driving Bug Hunter behavior
   Issue	#89	Prior duplicate-prevention improvements
   PR	#90	Implemented #89 prompt updates
   Issue	#158	Prior scope clarification (recent commits)
   Issue	#135	“Run required commands” guidance for code-changing workflows
   External Run	https://github.com/elastic/beats/actions/runs/22206903339	Referenced run showing concern

   ---

   What is this? | From workflow: Issue Triage Agent

   Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. for issue #185
2. strawgate commented on Feb 20, 2026

   

   strawgate

   on Feb 20, 2026

   ContributorAuthor

   More actions

   I want it to find a bug, build a test that shows that the bug exists, etc. It's not supposed to be just running tests hoping one fails and then calling that a bug
3. strawgate

   assigned

   Copilot

   and

   strawgate

   on Feb 20, 2026
4. strawgate

   closed this as completedin #189on Feb 20, 2026