[prompt-clustering] Copilot Agent Prompt Clustering — 2026-07-29

[github-actions[bot]]

Summary

Analysis period: last 30 days (2026-06-29 → 2026-07-29), capped at the 1000 most recent copilot-authored PRs returned by the search
Total PRs analyzed: 1000
Overall merge rate: 75.4% (754 merged / 242 closed unmerged / 4 still open)

The most important finding this run isn't a cluster — it's that 23% of "copilot" PRs in this window aren't from human task prompts at all. They're generated by autonomous internal agentic workflows (mostly a bot called pr-sous-chef) that scan the repo and open PRs on their own schedule. Clustering those together with genuine assigned-task PRs would have muddied both signals, so this report splits the two cohorts.

Full analysis report

Cohort split

Cohort	Count	Merge rate
Task-prompt PRs (human/issue-driven copilot coding agent runs)	768	74.1%
Autonomous agentic-workflow PRs (bot-scheduled, no human prompt)	232	79.7%

Detection method: PR bodies from autonomous workflows carry a hidden <!-- gh-aw-agentic-workflow: ..., workflow_id: ... --> marker (and a "Generated by 👨🍳 PR Sous Chef · gpt54 · N AIC" footer for that bot specifically). PRs without that marker were treated as task-prompt-driven and used for clustering.

pr-sous-chef alone accounts for 182 of these 232 PRs (78%) and has the highest merge rate of any cohort measured, 86% — it appears to run continuously against Sighthound/Poutine security findings, lint rules, and CI flakiness, and its fixes are narrow enough to land cleanly. smoke-claude (24 PRs, 96% merged) is clearly a smoke-test workflow, not real feature work. Everything else in this bucket (daily-squid-image-scan, workflow-normalizer, daily-geo-optimizer, etc.) fired 1–5 times each, mostly closed unmerged — too little volume to draw conclusions, but worth flagging that several of these autonomous experiments have a 0% land rate so far.

Cluster analysis (task-prompt cohort, n=768)

Clustered via TF-IDF (unigrams+bigrams, PR title + cleaned body) + k-means, k=4 chosen by elbow method. pr-type:* labels (already applied by the repo's own PR-triage automation) were cross-checked against clusters for validation rather than used as the clustering signal itself.

Cluster 1: CI / workflow failure fixes

• Size: 290 tasks (38% of task-prompt cohort)
• Merge rate: 80.3%
• Top keywords: workshop, job, cjs, safe, fix, failure, run, outputs
• Characteristics: Reactive fixes to broken CI jobs, safe-output handling bugs, and workshop/UI regressions. Largest and highest-volume cluster; second-highest success rate.
• Example PRs: fix: wrap fork push sequences in withGitHubHostToken to prevent duplicate Authorization (HTTP 400) #48595 (fork push double-post fix), Fix workshop progress rail state coupling and responsive overflow #46378 (workshop progress-rail state bug), eslint: no-core-error-then-process-exitcode — forward scan + top-level autofix #47240 (eslint rule for process.exitcode)

Cluster 2: Docs / engine / CLI config

• Size: 193 tasks (25%)
• Merge rate: 62.7% — lowest of the four clusters
• Top keywords: docs, engine, aw, gh, gh aw, model, version, cli
• Characteristics: Documentation updates, gh aw CLI/engine config changes, model/version pinning. Lower success rate suggests these tasks are either more speculative (dependency bumps, exploratory CLI features) or hit more review friction than pure bug fixes.
• Example PRs: Align CLI reference with current gh aw --help output #44603 (align CLI reference docs), feat: extend gh aw init with repo setup primitives #45475 (extend gh aw init, closed), Fix serena-mcp-server container: rebuild on python:3.13-slim to eliminate 413 CVEs #48562 (fix serena-mcp-server container, closed)

Cluster 3: Agentic workflow definitions

• Size: 148 tasks (19%)
• Merge rate: 73.0%
• Top keywords: workflow, workflows, md, daily, github, copilot, pr, lock
• Characteristics: Adding or hardening .md/.lock.yml agentic workflow definitions themselves (daily workflows, smoke workflows, triage automation) — meta-work on the gh-aw system rather than the CLI/runtime code.
• Example PRs: Add EvoSkill-based agent skill evolution workflow #48722 (EvoSkill agent-skill evolution workflow), Add smoke-copilot-auto workflow: auto-post haiku on PRs #48423 (smoke-copilot-auto haiku workflow), Auto-Triage: deterministically classify machine-generated aggregate reports #47373 (auto-triage classifier)

Cluster 4: Go package refactors & linters

• Size: 137 tasks (18%)
• Merge rate: 78.1%
• Top keywords: pkg, test, linters, spec, pkg cli, string, cli, pkg linters
• Characteristics: Internal Go refactors, custom-linter additions/enforcement, and test-quality improvements under pkg/. Tightly scoped, mechanical changes — second-highest success rate.
• Example PRs: refactor(workflow): split 420-line buildMainJob into focused helpers #45333 (split 420-line function into helpers), Enforce sprintfbool in cgo custom-linter CI gates #46987 (enforce sprintfbool linter), pkg/agentdrain: add nil-guard to Analyze and improve anomaly test quality #44455 (nil-guard fix in pkg/agentdrain)

Success rate by cluster

Cluster	Tasks	Merge rate	Top keywords
CI/workflow failure fixes	290	80.3%	workshop, job, fix, failure
Docs / engine / CLI config	193	62.7%	docs, engine, gh aw, cli
Agentic workflow definitions	148	73.0%	workflow, daily, copilot, lock
Go pkg refactors & linters	137	78.1%	pkg, test, linters, refactor
Autonomous: pr-sous-chef	182	86.3%	(bot-driven, not a prompt cluster)
Autonomous: smoke-claude	24	95.8%	(smoke test, not real work)

pr-type label cross-check (task-prompt cohort)

Label	Count
unlabeled	500
bug	88
chore	50
refactor	40
feature	40
docs	32
test	15
formatting	3

Half the cohort has no pr-type label yet — the repo's own triage labeler hasn't caught up to these, which is itself a gap worth closing since it would make future runs of this analysis far cheaper than re-deriving clusters from scratch.

Data-quality notes / limitations

• Workflow metrics unavailable: /tmp/gh-aw/agent/prompt-cache/pr-full-data/ (comments, reviews, commits, turn counts) is stale — it covers PR fix(deps_security): propagate context.Context through security advisory HTTP call #30577–Bump pinned Copilot/Codex/GitHub MCP versions and regenerate workflow artifacts #34390 from a prior run (file mtimes 2026-05-24/30), while this window's PRs are Extend envutil for bool/string access and remove direct CI bypasses #44097–Suppress no-op Workflow Health Dashboard issues #48831 (2026-07-07 to 2026-07-29). There is zero overlap, so per-PR comment/review/turn/duration metrics could not be joined in for this report. Re-running the full-data fetch phase against the current PR range would let a future run add turn counts and review-cycle counts to the cluster table.
• Boilerplate stripped before vectorization: raw TF-IDF on unstripped bodies produced a spurious cluster driven entirely by the pr-sous-chef PR footer text ("AIC" cost units, "sous chef", embedded run-metadata comments) rather than task content — this is why the autonomous-workflow cohort was split out first rather than clustered by text.
• 1000-PR cap: the source query returned exactly 1000 records; if the true 30-day volume is higher, the oldest task-prompt PRs in the window may be under-represented.

Recommendations

1. Investigate the docs/engine/CLI cluster's 63% merge rate — it's meaningfully below the other three clusters (73–80%). Worth sampling a few of the closed-unmerged PRs in this cluster (e.g. feat: extend gh aw init with repo setup primitives #45475, Fix serena-mcp-server container: rebuild on python:3.13-slim to eliminate 413 CVEs #48562) to see whether they're being closed for scope/quality reasons or superseded by duplicate work.
2. Re-point the full-PR-data cache fetch at the current PR number range so the next run can join in comments/reviews/turn counts — right now this report can only speak to merge outcome, not iteration cost.
3. Backfill pr-type labels on the ~65% of task-prompt PRs currently unlabeled so future clustering can validate against ground truth without re-deriving TF-IDF clusters each time.
4. Watch the low-volume autonomous workflows (daily-squid-image-scan, workflow-normalizer, daily-geo-optimizer, deep-report, daily-hippo-learn) — each has fired only 1–5 times in 30 days with a 0% merge rate. Either they're still experimental (expected) or silently failing to produce mergeable output.

---

Generated by Copilot Agent Prompt Clustering Analysis (Run: 30444124180)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · age00 · 137.1 AIC · ⌖ 34.5 AIC · ⊞ 13.1K · ◷

• expires on Jul 30, 2026, 3:19 AM UTC-08:00

[github-actions[bot]]

Smoke bot grunt. Me touch talk. Run 30448361038 alive.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · gpt54 · 68.5 AIC · ⌖ 4.09 AIC · ⊞ 19.4K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #49118.