Repository navigation
[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-02-20 #17147
Closed
Replies: 1 comment
|
This discussion was automatically closed because it expired on 2026-02-27T12:49:36.201Z.
|
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Daily NLP-based clustering analysis of copilot agent task prompts for the last 30 days.
Summary
Key Insights
Cluster Overview
Detailed Cluster Analysis
Cluster 1: General Maintenance & Updates
The largest cluster — broad maintenance tasks spanning MCP updates, workflow upkeep, and agentic configuration.
Sample Task Prompt
Cluster 2: Issue-Based Task Resolution⚠️ Lowest Success
Tasks driven by GitHub issue descriptions — often complex, multi-part feature requests and workflow fixes from issue bodies.
"---- This section details on the original issue you should resolve"wrapperSample Task Prompt
Cluster 3: Agentic Workflow Maintenance
Tasks focused on agentic workflow system maintenance: debugging, recompilation, and agent configuration.
Sample Task Prompt
Cluster 4: Safe-Outputs & API Tasks
Tasks centered on the
safe-outputssubsystem — GitHub Projects API, PR creation, and related integrations.Sample Task Prompt
Cluster 5: CI Failure Investigation
CI Failure Doctor tasks — investigating specific failed workflow runs and implementing fixes.
Sample Task Prompt
Cluster 6: Security & Campaign Tasks
Security alert burndown campaigns, code security analysis, and project-wide security improvements.
Sample Task Prompt
Cluster 7: Documentation Writing ✅ Highest Success
Technical documentation, W3C-style specs, API guides, and README updates — consistently the most successful task type.
Sample Task Prompt
Cluster 8: Targeted CI Fixes (with Job ID)
Structured CI fix tasks that include explicit job IDs and job URLs — high information density leads to higher success.
Sample Task Prompt
Trend Analysis: Historical Comparison
The overall merge success rate has stabilized at ~69.2%. The 745 new PRs in this run show similar patterns to previously analyzed tasks.
Notable shifts vs Feb 19:
Recent 50 PRs: Full Data Table
@playwright/mcpversion is already updatedinstall plugin→plugin installFixes #N" as bullet point in PR bodyRecommendations
Based on the clustering analysis:
Improve Issue-Based Task prompts — The "Issue-Based Task Resolution" cluster (22% of all tasks, 54.8% success) is the biggest opportunity. Prompts wrapped in
---- This section details on the original issue you should resolveoften lack specific acceptance criteria. Adding explicit success conditions and expected outputs could significantly boost the merge rate.Replicate Documentation task patterns — Documentation tasks achieve 93% success with small footprints (avg 4 files). These prompts tend to be clear, scoped, and verifiable. Apply the same specificity to other task types.
Always include Job ID/URL in CI fix tasks — The "Targeted CI Fixes (with Job ID)" cluster achieves 80.6% success despite high file complexity (avg 29 files). Providing the exact Job ID and Job URL in the prompt enables the agent to fetch precise context, reducing ambiguity.
Break down high-complexity Dependency Update tasks — With avg 28 files changed, complex dependency PRs have more failure modes. Where possible, split large package tree updates into smaller batches.
Add review requests for Issue-Based tasks — This cluster has the lowest avg reviews (0.37) and also the lowest success rate. Systematic human review checkpoints may help catch problems early.
References:
All reactions