Phase 2: Build Automated Triage Agent & Metadata Suggester #1690

Summary

Build an intelligent issue triage agent that analyzes issue content and suggests metadata improvements. This is Phase 2 of Epic #1679 (Comprehensive Issue Metadata Expansion), building on Phase 1 audit findings.

Objective

Create an agent capable of:

1. Analyzing issue content to detect missing/incorrect metadata
2. Suggesting improvements (type, assignee, project, milestone, relationships)
3. Validating templates for compliance
4. Mapping issue relationships (parent-child, blocked-by, blocking)

Deliverables

1. Issue Analyzer Module

File: scripts/agents/includes/issue-analyzer.js

Capabilities:

• Type Detection: Analyze title/description to suggest issue type

  • Keywords: "bug", "error", "broken" → type:bug
  • Keywords: "feature", "add", "support" → type:feature
  • Keywords: "epic", "initiative", "phase" → type:epic
  • Keywords: "story", "user", "as a" → type:story
  • Confidence scoring (80%+ = suggest, <50% = unclear)

• Area Detection: Extract area labels from content

  • Keywords: "frontend", "UI", "component" → area:frontend
  • Keywords: "backend", "API", "database" → area:backend
  • Keywords: "CI/CD", "workflow", "github actions" → area:ci
  • Keywords: "docs", "documentation", "readme" → area:docs
  • Keywords: "security", "auth", "vulnerability" → area:security

• Priority Assessment: Estimate priority from keywords

  • Severity: "critical", "blocking", "urgent" → priority:critical
  • Urgency: "ASAP", "high", "soon" → priority:important
  • Normal: Default to priority:normal

• Assignee Suggestions: Find potential assignees

  • Extract @mentions from description
  • Suggest team lead based on area
  • Parse "assigned to:", "owner:", "cc:" patterns

• Project/Milestone Detection: Find context clues

  • Parse "project:", "milestone:", "sprint:" patterns
  • Detect version numbers (v1.0, Q3 2026)
  • Identify release context from description

• Confidence Scoring: Rate each suggestion

  • 0-100% confidence per suggestion
  • Only return suggestions >70% confidence by default
  • Support --low-confidence flag for 50%+ suggestions

2. Metadata Suggester Module

File: scripts/agents/includes/metadata-suggester.js

Generates actionable suggestions:

• Type suggestion with confidence & rationale
• Area labels (multiple possible)
• Priority level with supporting evidence
• Assignee recommendations (primary + backup)
• Project suggestion
• Milestone estimation
• Custom field suggestions (domain, team, risk)
• Format: Structured object with reasoning

Output format:

{
  type: { suggestion: 'feature', confidence: 85, reason: 'Keywords: add, support' },
  area: [
    { label: 'frontend', confidence: 90, reason: 'UI, component mentioned' },
    { label: 'docs', confidence: 60, reason: 'documentation tagged' }
  ],
  priority: { level: 'important', confidence: 75, reason: 'High impact' },
  assignee: {
    primary: '@frontend-team',
    backup: '@design-team',
    confidence: 80
  },
  project: 'Q3 Release',
  milestone: 'v2.1.0'
}

3. Template Validator Module

File: scripts/agents/includes/template-validator.js

Validates issue description templates:

• Detect which template should be applied (based on issue type)

• Check for required sections:

  • Feature: Problem statement, acceptance criteria, DoR, DoD
  • Bug: Reproduction steps, expected vs actual, environment, DoR, DoD
  • Epic: Vision, success criteria, high-level tasks, DoR, DoD
  • Story: User narrative, acceptance criteria, DoR, DoD
  • Task: Clear objective, steps, success criteria, DoR, DoD

• Validate section quality:

  • Not empty
  • Has content (>20 characters)
  • Properly formatted
  • Complete checklists

• Report missing/broken sections:

  • Which sections present vs missing
  • Quality score (% complete)
  • Specific fixes needed

• Reuse logic from PR improvement: Enhance bulk issue template fixer with pagination support #1669 (template section detection)

4. Relationship Mapper Module

File: scripts/agents/includes/relationship-mapper.js

Maps issue relationships:

• Parent/Child Detection:

  • Parse: "parent: #XXXX", "child: #YYYY", "parent issue #XXXX"
  • Create parent-child links
  • Validate: No circular dependencies

• Blocking Detection:

  • Parse: "blocked by #XXXX", "blocks #YYYY", "waiting for #ZZZZ"
  • Extract PR blocks: "blocked by PR #XXXX"
  • Create blocking relationships

• Dependency Validation:

  • Check: Parent-child consistency (child should be simpler than parent)
  • Check: Blocking cycles (issue can't block itself transitively)
  • Warn: Long dependency chains (>5 levels deep)

• Output: Relationship graph

  • Parent: #XXXX
  • Children: [#YYYY, #ZZZZ]
  • Blocked by: [#AAAA, PR #BBBB]
  • Blocking: [#CCCC]

5. Integration Module

File: scripts/agents/issue-triage.agent.js

Main triage agent:

• Load all four sub-modules
• Orchestrate analysis (analyze → suggest → validate → map)
• Handle errors gracefully (skip problematic issues)
• Cache results (avoid re-analyzing same issue)
• Support batch processing (--batch=50)
• Progress logging (show % complete)
• Output formats: JSON, Markdown report, GitHub API payload
• Performance: Analyze 100+ issues/minute

6. Documentation

File: scripts/agents/TRIAGE-AGENT-GUIDE.md

Contents:

• Quick start example
• How accuracy is measured (confidence scores)
• How to review suggestions
• How to apply suggestions (manual vs batch)
• Limitations and edge cases
• Examples per issue type

7. Test Suite

File: scripts/agents/__tests__/issue-triage.agent.test.js

Tests (30+):

• Analyzer tests (8):

  • Type detection (bug, feature, epic, story, task)
  • Area label extraction
  • Priority assessment
  • Edge cases (ambiguous issues)

• Suggester tests (8):

  • Confidence scoring
  • Reasoning explanation
  • Multiple suggestions
  • Filtering by confidence threshold

• Validator tests (8):

  • Template detection
  • Section validation
  • Quality scoring
  • Broken template detection

• Mapper tests (6):

  • Relationship parsing
  • Circular dependency detection
  • Output formatting

Acceptance Criteria

Functional

• Analyzes 100+ issues without errors
• Generates suggestions in <500ms per issue
• Confidence scores are calibrated (suggested items have 80%+ accuracy)
• All four modules work independently & integrated
• Handles malformed/unusual issue descriptions gracefully
• Outputs valid JSON, Markdown, GitHub API payloads

Quality

• 30+ unit tests, all passing
• ESLint/Prettier compliance
• Front-matter validation passing
• Comprehensive documentation with examples
• All suggestion types covered

Accuracy

• Type detection: 85%+ accuracy (validated on sample of 20 issues)
• Area detection: 80%+ accuracy
• Priority assessment: 75%+ accuracy
• Assignee suggestions: Reasonable options presented
• Template validation: 95%+ accuracy (binary: valid/invalid)

Success Metrics

After building the agent:

• Can suggest metadata for any issue with 70%+ confidence
• Developers trust suggestions (low false-positive rate)
• Agent is 3+ hours faster than manual triage per 100 issues
• Ready to power Phase 3 automation

Implementation Approach

Architecture

scripts/agents/
├── issue-triage.agent.js                 # Main orchestrator
├── includes/
│   ├── issue-analyzer.js                 # Content analysis
│   ├── metadata-suggester.js             # Suggestion generation
│   ├── template-validator.js             # Template checking
│   └── relationship-mapper.js            # Relationship detection
├── TRIAGE-AGENT-GUIDE.md                 # User documentation
└── __tests__/
    └── issue-triage.agent.test.js        # Test suite

Key Features

• Modular design: Each module can be used independently
• Confidence scoring: Only high-confidence suggestions returned by default
• Extensible: Easy to add new detection types
• Non-destructive: Generates suggestions only, doesn't modify issues
• Batch processing: Handle 100+ issues efficiently
• Detailed output: Reasoning for every suggestion

Reuse from PR #1669

• GitHub API request helper
• Pagination patterns
• Template section detection logic
• Batch processing approach
• Error handling patterns

Definition of Done

• All 4 modules complete and integrated
• Agent tested on 50+ real issues
• Suggestions validated for accuracy (80%+)
• Documentation written with examples
• Test suite (30+) passing
• PR created and merged
• Epic Epic: Comprehensive Issue Metadata Expansion & Automated Triage System #1679 updated with completion status

Related Issues

• Epic: Epic: Comprehensive Issue Metadata Expansion & Automated Triage System #1679 (Comprehensive Issue Metadata Expansion)
• Phase 1: Phase 1: Create Issue Metadata Audit Script #1689 (Audit Script)
• Reference: PR improvement: Enhance bulk issue template fixer with pagination support #1669 (Bulk Issue Template Fixer)

Effort Estimate

• Story points: 13-21
• Timeline: 5-7 days
• Team: 1-2 engineers

🤖 Generated by Claude Code