P2: Test Expectations Misalignment — API Documentation & Coverage #1716

Overview

Priority: P2 (Testing/Quality)
Type: Test Quality & Documentation
Blocking: No (tests pass, but weak test coverage)
Effort: 4–5 hours
Related: PR #1713 (Phase 3 findings)

Problem Statement

The handle-needs-triage handler tests codify arbitrary behavior instead of verifying the documented handler API. Tests accept overly-permissive assertions that may mask logic bugs in the issue triage automation system.

Files Affected

• scripts/automation/__tests__/handle-needs-triage.test.js (lines 17, 27, 37, 46, 57, 50-58)
• scripts/agents/includes/handle-needs-triage.js (handler implementation)
• Docs: docs/HANDLER_API_REFERENCE.md (documentation)

Impact Assessment

• Current State: Non-blocking (tests pass 100%, but weak)
• Risk Scope: Issue triage handler inference logic, confidence score interpretation
• Blocking Tests: None (tests pass)
• CI Status: ✅ CI passes (tests pass, but assertions are weak)
• Quality Risk: Weak tests may mask logic bugs during refactoring

Problem Details

Issue 1: Arbitrary Default Behavior

Lines 17, 27, 37, 46, 57
Tests assert that confidence < 0.3 for generic input remains untyped, but tests don't verify:

• Why this threshold exists
• What the handler actually implements
• Whether behavior matches documentation

Issue 2: Overly-Permissive Assertions

Lines 50-58
Tests accept first zero-score entry instead of requiring meaningful confidence:

• Current: expect(result.type).toBe(firstZeroScoreEntry.type)
• Problem: Accepts any low-confidence match, doesn't verify meaningful inference
• Risk: May mask bugs in scoring logic

Issue 3: Weak Edge Case Coverage

Missing tests for:

• Empty issue content
• Mixed keywords (ambiguous typing)
• Low confidence scores
• Boundary conditions

Remediation Scope

1. Audit Handler Implementation

• Review handle-needs-triage.js actual behavior
• Document actual confidence thresholds
• Identify undocumented quirks or edge cases
• Map test assertions to handler code paths

2. Update Test Expectations

• Match test assertions to actual handler behavior
• Fix overly-permissive tests
• Document why tests assert what they do (WHY comments)
• Link test assertions to handler documentation

3. Add Edge Case Tests

• Empty issue content → should not crash, returns default
• Mixed keywords → ambiguous typing, return lowest-confidence option
• Low confidence scores → verify threshold behavior
• Boundary conditions → confidence exactly at threshold

4. Document Handler API

• Configuration options
• Confidence threshold behavior (≥0.3 for type, <0.3 untyped)
• Return value semantics (type field, confidence field, etc.)
• Known limitations and assumptions

Completion Criteria

• All test assertions justified
• Edge cases covered
• Handler API documented
• No weak tests that mask bugs
• All tests passing (existing + new)
• Code review approved
• Merged to develop

Testing

• Unit tests for each confidence level (0, 0.15, 0.3, 0.5, 1.0)
• Edge case tests (empty, ambiguous, boundary)
• Integration test with real issue data
• Documentation review

References

• Related Issue: Develop Branch Stability Initiative (chore: Establish develop branch stability baseline #1713)
• Test Status: Initial report in Phase 3 audit
• Handler: scripts/agents/includes/handle-needs-triage.js
• Tests: scripts/automation/__tests__/handle-needs-triage.test.js

Suggested Owner: @ash (or assign based on availability)
Timeline: After issue #1715 completes
Labels: type:testing-coverage, priority:important, area:automation, meta:needs-changelog