docs: spec 016 and 017 design findings from the #3500 review #3519

CodeRabbit raised 13 Major findings on #3500 against specs 016 and 017. These are design decisions for the spec owners, so most are tracked here rather than changed in the move. Part of #3464.

Status update (2026-09-26, after #3500 merged as 62c9344f31): the premise that #3500 would move the specs
"unchanged from their agreed sources" no longer holds. The move went through several review rounds, and the
mechanical defects found on the way were fixed in place rather than deferred. Six items below are now closed and
ticked; the remainder are still open and still need an owner decision. #3434, which carries the pre-merge copies of
the same files, will reintroduce the closed items if merged without a rebase — see the comment on that pull request.

Spec 016: changelog agent (owner decision)

• Token handling (contracts/cli-interface.md L41, L126, L139): the documented --github-token flag and GITHUB_TOKEN=ghp_xxxx example put tokens in shell history and process arguments; validate.sh forwards the token as an argument. The contract documents the shipped validator (docs: changelog-agent - Document specification decisions #3470 item 1), so fixing it means changing the validator to accept the token from the environment only.
• Authorisation for changelog-check-links (contracts/rest-api-interface.md L91–L100): only changelog-validate defines authentication. A caller-supplied github_repo must be checked against the authenticated installation, with the token restricted to that repository.
• One error payload schema: the data-model.md table defines error_code, expected_format and actual_value, but the example at L102 and spec.md L109 use error_type, and research.md L160–L172 uses type/actual/expected/current_value. Pick one and align all three.
  • Outcome: closed in docs: specs - land CI remediation, changelog agent and agent consolidation specs #3500 23cf74620c / 4c11647bff: one name (ErrorObject) and one field set across data-model.md, spec.md, research.md and the REST contract; no ValidationError remains.
• Issue vs PR link rules (data-model.md L212, L254): "merged only" (decided on refactor: changelog-agent - establish quality framework and spec 015 structure #3434) fails valid issue references, since issues have no merged state. Resolve the reference type first, then define a status rule per type.
  • Outcome: closed in docs: specs - land CI remediation, changelog agent and agent consolidation specs #3500 23cf74620c: a linked pull request must exist and be merged; an issue need only exist, since an issue has no merged state.
• Changelog exemption by branch prefix (spec.md L101, research.md L210–L236): Q1 decided "automatic bypass for chore/ and deps/". CodeRabbit notes that a user-facing change on such a branch then skips validation. Confirm the decision, or switch to the meta:changelog-exempt label (research.md L320) as the trusted signal.
  • Outcome: closed in docs: specs - land CI remediation, changelog agent and agent consolidation specs #3500 4af8501b79: the pseudocode now mirrors changelog-unified.yml (bot author, docs-only diff, explicit meta:no-changelog) and the branch name is never consulted. Note the meta:changelog-exempt label it suggested does not exist; the canonical pair is meta:needs-changelog / meta:no-changelog.

Spec 017: CI failure remediation (marked Resolved 2026-09-22)

The findings concern the evidence standard of a completed investigation. Apply them to the method if 017 is reused:

• Keep inconclusive failures as INVESTIGATING, not "ENVIRONMENTAL (likely)" (contracts/pr-comment-template.md L123, quickstart.md L391).
• Compare per-entry changelog results, not the 6/54 aggregate (data-model.md L29–L32; plan.md L31–L33, L118, L153, L233).
  • Outcome: closed in docs: specs - land CI remediation, changelog agent and agent consolidation specs #3500 07cf08f282 and 0ea0b3e6c2. The root npm run validate:changelog script runs the repository-wide safety audit and reports no per-entry results, so it cannot establish an entry-level baseline; both branches now run .github/validation/changelog/bin/validate.js, which does. The stale 6/54 figure itself is tracked in the spec's own resolution note as no longer accurate.
• Match exact finding identity (location and content), not message or rule text (data-model.md L68–L69, L107–L108, L222–L224; plan.md L119–L123, L160, L167, L187).
• Different errors on both branches do not establish an environmental cause (data-model.md L148).
  • Outcome: closed in docs: specs - land CI remediation, changelog agent and agent consolidation specs #3500 0ea0b3e6c2: data-model.md now classifies a differing error as AMBIGUOUS and requires comparing each failure, matching what the quickstart matrix and flow already said. The two files previously gave three different answers for the same case.
• Make "ready to merge" conditional on per-category evidence (plan.md L146, L199).
• Compare against the failing run's base and head SHAs, not the current develop (quickstart.md L33).
• Compare failing test names, not grep -c "FAIL\|PASS" counts (quickstart.md L562).
• Attribute failures by evidence, not timing (spec.md L81).

Second CodeRabbit CLI pass on #3500 (2026-09-26)

coderabbit review --agent --base develop on 46b201dd8f (19 files) returned 10 further findings after the first pass's 10 were addressed. Most are the same defects already listed above, re-raised on the corrected prose, so they need no new decision. The genuinely new ones are listed first.

New — needs a decision

• Limit automatic labelling to non-exemption labels (spec.md L77, User Story 4). The scenario has validation apply changelog labels with no manual step, which implies the workflow may attach meta:no-changelog — the exemption label — on its own. An exemption must be an explicit human act; if the workflow can set it, the exemption is not an audit trail.
• Run the same entry validator on both branches (017 quickstart.md L72). The develop comparison writes .json from the feature-016 engine while the audit-branch comparison still writes .txt from the root validate:changelog safety audit, so the two sides are not comparable. Both must use .github/validation/changelog/bin/validate.js.
  • Outcome: closed in docs: specs - land CI remediation, changelog agent and agent consolidation specs #3500 0ea0b3e6c2: both branches use the same engine, the same --output text format and matching filenames, so the documented count and cross-branch diff compare like with like.
• Test the shipped workflow, not a local simulation (016 quickstart.md L319). The scenario reconstructs the gate's decision in shell, so it verifies the scenario's copy of the logic rather than changelog-unified.yml. Either drive the real workflow or state explicitly that the scenario documents intent and is not a gate test.

Already covered above — re-raised, no new decision

• 017 quickstart.md L424 inconclusive classification → "Keep inconclusive failures as INVESTIGATING"
• 017 pr-comment-template.md L51 category verdicts → same item
• 017 quickstart.md L397 wrong branch → "Compare against the failing run's base and head SHAs"
• 017 quickstart.md L565 lint loses the file type → "Compare failing test names, not grep -c counts"
• 017 plan.md L344 post without re-verification → "Make 'ready to merge' conditional on per-category evidence"

Withdrawn after verification (2026-09-26)

• Exclude CHANGELOG.md from the docs-only bypass — invalid, withdrawn. This item repeated a
  false claim. changelog-unified.yml tests changed.includes("CHANGELOG.md") and returns
  run_validation = true before the docs-only exemption is evaluated, so a changelog-only pull request
  is validated and never reaches the docs/**/*.md test. The gate's own suite already asserts this
  (runs validation when the root changelog changed). No workflow change is needed and none should be
  made. Tracked and corrected in docs: spec 016 FR-009 asserted a non-existent bypass in the shipped changelog gate #3584, fixed in fix: spec - correct a false claim about the shipped changelog gate #3583.

Tracked elsewhere

• scripts/automation/__tests__/performance/results-phase-2b.json timestamp/result mismatch — a generated fixture; see test: full Jest run writes artefacts into the tracked tree #3498 and PR fix: test - stop tests writing into the repository and guard against it (#3498) #3499, which stops tests writing into the repository. Not edited in docs: specs - land CI remediation, changelog agent and agent consolidation specs #3500.

Note on convergence

Two further CLI passes each returned 10 findings. The unresolved tail is about the prose of specifications for a system that is not built, so there is no implementation to check a claim against and each pass raises new hypotheticals. #3470 already carries the spec 016 design decisions and the stale-lock/OS-lock question that the recovery-mutex work touches. Continuing to review in place is not converging; these checklists are the right home for the remainder.