Repository navigation
fix(#7387): surface silent risk-assessment omissions - #7388
fullsend-ai-coder[bot] wants to merge 6 commits into
Conversation
Risk-assessment comments were missing on ~15% of review runs with no
log marker and no PR-visible diagnostic, so a miss could not be told
apart from a skip.
The pi Agent tool now emits grep-stable fullsend:persona:{invoke,
complete,error,reject,skip} lines for every named persona. Bootstrap
also logs skip when a persona file fails to register. When
REVIEW_RISK_ASSESSMENT_ENABLED is on (ADR 0089 default) and the review
result omits risk_assessment, fullsend post-review posts a sticky
"Risk assessment unavailable this run" comment instead of staying
silent.
The agents-repo post-review.sh still posts the scored comment when
the field is present; this CLI path covers the missing-field case
without a companion change there.
Note: pre-commit could not fetch remote hook repos (HTTP 403). Local
equivalents were run: gofmt, go vet, ADR lints, lychee, gitleaks,
trailing-whitespace/EOF.
Closes #7387
|
🤖 Review · Commit: |
|
🤖 Finished Review · ✅ Success · Started 10:05 PM UTC · Completed 10:26 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.64 |
Site previewPreview: https://92583653-site.fullsend-ai.workers.dev Commit: |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Risk Assessment: elevated (3/5) DetailsRe-review anchoring applies: Tier 1 signals (12 files, large blast radius, zero protected/dependency paths, bot author) are unchanged from the prior assessment characteristics, and the incremental delta since the prior SHA is small and additive (two files, mostly test coverage) rather than new functional surface, so the prior score of 3/elevated is preserved. Previous runRisk Assessment: elevated (3/5) DetailsTier 1 signals are not truly unchanged from the prior anchor: this round added new files (sticky.go/sticky_test.go) making a genuine functional change to FindMarkedComment, the single shared matching primitive underlying sticky-comment upsert used across postreview/post-comment flows, alongside a 33% growth in file/line count beyond the previously-assessed additive doc rewording. Blast radius remains large but is now docs-plus-shared-utility rather than mostly docs, warranting elevation to 3/elevated, tempered by strong accompanying test coverage, zero protected-path or dependency changes, and no reverts in history. Previous run (2)Risk Assessment: moderate (2/5) DetailsRe-review anchoring applies: Tier 1 signals (10 files, large blast radius via 820 lines, 0 protected paths, no dependency changes, bot author) are unchanged in kind from the prior assessment, and Tier 2 git history reconfirms the same known high-churn/multi-author docs hotspots already factored into the prior moderate score, with no reverts or hack/workaround sentiment found; the incremental diff since the last review (a new marker constant, a collision test, and doc rewording) is additive and low-risk, so the composite score is preserved at 2/moderate rather than adjusted. Previous run (3)Risk Assessment: moderate (2/5) DetailsTier 1 signals (file count, protected-path count, dependency changes, bot authorship) are unchanged from the prior assessment one commit ago, and git history confirms the same two docs files remain high-churn hotspots; this is a fix-agent iteration addressing prior review feedback with no new risk-relevant scope change, so the score is anchored to the prior moderate composite of 2. Previous run (4)Risk Assessment: moderate (2/5) DetailsTier 1 signals are unchanged from the prior review (10 files, no protected/security paths, no dependency changes, bot author); Tier 2 churn/fix-rate hotspots in the touched docs files and a well-matched, low-mismatch linked issue reproduce the same low-moderate composite, so the prior score of 2 is preserved per re-review anchoring. Previous run (5)Risk Assessment: moderate (2/5) DetailsWell-scoped, bot-authored observability fix directly implementing a linked, well-matched issue, with no protected or security-sensitive paths and decent test coverage, but two touched docs files are high-churn/high-fix-rate hotspots which lifts the git-history tier to a low-moderate composite score of 2. |
ReviewFindingsLow
Previous runReviewThis is a re-review. All three findings from the prior round are resolved: FindingsLow
Next steps:
Previous run (2)ReviewThis is a re-review. The two findings carried from the prior round are both resolved: the diagnostic marker no longer collides with the ADR 0089 score-card marker (verified via the new FindingsMedium
Low
Next steps:
Previous run (3)ReviewThis is a re-review. Six of the seven findings from the prior round are confirmed fixed: the shutdown() never-dispatched mislabeling, FindingsMedium
Next steps:
Previous run (4)ReviewThis is a re-review. Six of the seven prior findings are confirmed fixed (persona shutdown mislabeling, FindingsMedium
Next steps:
Previous run (5)ReviewFindingsMedium
Low
Next steps:
|
|
🤖 Finished Fix · ✅ Success · Started 10:28 PM UTC · Completed 10:47 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.07 |
Fix findings from the review agent on PR #7388 (silent risk-assessment omissions, #7387): - fullsend-agent.js: shutdown() decided "never-dispatched" from dispatchedPersonas, which is only populated at invoke/reject/skip time. A persona still queued behind maxConcurrent when shutdown() fired had none of those yet, so it was mislabeled never-dispatched, and its own skipNamed call then no-opped because the name was already recorded. Track "run() entered" separately (startedPersonas) so shutdown only labels personas whose run() was never called, and a queued call now gets its own accurate skip reason (e.g. aborted) instead. - postreview.go: parseReviewResult failed the whole JSON parse on any type mismatch in the optional risk_assessment field, discarding an otherwise-valid action/body/findings. Decode risk_assessment via json.RawMessage in a second pass so a malformed nested value logs a warning and leaves RiskAssessment nil instead of losing the result. - postreview.go: allowlist RiskAssessment.Level against the ADR 0089 enum before interpolating it into printer output. - pi_personas.go: format the discovery-skip persona name with %q (like reason) instead of %s, since that name comes from an unvalidated filename. - docs/agents/review.md: add REVIEW_RISK_ASSESSMENT_ENABLED to the Variables reference table. - docs/ADRs/0089: reword "instead of omitting the comment" (ambiguous about which comment) to "instead of staying silent about the missing assessment". Not addressed (disagreement recorded): the cross-repo-contract finding on whether REVIEW_RISK_ASSESSMENT_ENABLED propagates consistently between the sandbox orchestrator and the fullsend post-review process — that can only be verified against fullsend-ai/agents, which isn't checked out here. Addresses #7388
🔧 Fix agent — iteration 1 (bot-triggered)Fixed the logic error where shutdown() mislabeled a queued persona as never-dispatched (with a regression test), fixed parseReviewResult discarding a whole valid result on a malformed nested risk_assessment value, allowlisted RiskAssessment.Level before log interpolation, escaped the discovery-skip persona name with %q, added the missing REVIEW_RISK_ASSESSMENT_ENABLED doc row, and reworded the ambiguous ADR sentence. Recorded a disagreement on the cross-repo-contract finding, which requires coordination with fullsend-ai/agents that this sandbox can't perform. All new and existing tests pass (node --test: 57 pass/1 skip; go test ./internal/cli/... ./internal/runtime/... pass except two pre-existing, unrelated failures confirmed present before this fix's changes). Fixed (6):
Disagreed (1):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 10:49 PM UTC · Completed 11:12 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $8.34 |
|
🤖 Finished Fix · ✅ Success · Started 11:14 PM UTC · Completed 11:29 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.00 |
- Supersede the stale "Risk assessment unavailable this run" sticky diagnostic once a later run on the same PR does produce a risk_assessment, instead of leaving it describing a run that in fact succeeded. - Correct docs/agents/review.md: REVIEW_RISK_ASSESSMENT_ENABLED is read by fullsend post-review from the runner's own environment (env.runner), not the sandbox — unlike REVIEW_FINDING_SEVERITY_THRESHOLD, which is read in-sandbox. Drop the inaccurate "no separate configuration is needed" claim. - Unify the fullsend:persona:skip marker's name= quoting across all three emission sites (Go discovery-skip, Go registered-persona skip, JS logPersona) on one convention: collapse control characters and quote only when whitespace remains. Add a Go and a JS test that both match emitted lines against the same shared regex. Carries forward the cross-repo-contract finding on REVIEW_RISK_ASSESSMENT_ENABLED propagation as a recorded disagreement (out of scope for this repo; requires a tracked follow-up against fullsend-ai/agents, not a code change here). Addresses #7388
🔧 Fix agent — iteration 2 (bot-triggered)Fixed the three current-round findings: superseding the stale risk-assessment sticky once a later run succeeds, correcting docs/agents/review.md's env.runner vs env.sandbox guidance for REVIEW_RISK_ASSESSMENT_ENABLED, and unifying the fullsend:persona:skip name= quoting convention across the Go and JS emission sites with a shared-regex test on each side. Recorded a disagreement on the carried-forward cross-repo-contract finding, which requires action in a different repository and was already declined as out of scope in the prior iteration. Fixed (3):
Disagreed (1):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 11:30 PM UTC · Completed 11:52 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.23 |
|
🤖 Finished Fix · ✅ Success · Started 11:54 PM UTC · Completed 12:05 AM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.52 |
- Use a distinct marker (<!-- fullsend:risk-assessment-missing -->) for the missing-risk-assessment diagnostic sticky, separate from <!-- fullsend:risk-assessment --> which post-review.sh (in fullsend-ai/agents) uses to find and update the ADR 0089 score-card comment. sticky.FindMarkedComment matches via strings.Contains, so the shared marker could let this CLI select and overwrite the score card. Added a regression test asserting the two markers cannot collide as substrings, and a short ADR 0089 note documenting the split. Renamed the constant to missingRiskAssessmentMarker throughout. - Clarified docs/agents/review.md: REVIEW_RISK_ASSESSMENT_ENABLED is read independently by the sandbox orchestrator (gates whether the risk-assessment pre-pass is dispatched) and by fullsend post-review in env.runner (gates the missing-assessment diagnostic). Both must be set together to fully enable or disable the feature. Pre-commit could not fetch remote hook repositories (sandboxed network policy); ran the equivalent hooks directly instead (gofmt, go vet, lint-adr-status, lint-adr-numbers, lint-adr-frontmatter, lint-docs-links, lychee, trailing-whitespace/EOF/merge-conflict checks) — all pass. Addresses #7388
|
🤖 Finished Review · ✅ Success · Started 12:07 AM UTC · Completed 12:31 AM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $8.65 |
|
🤖 Finished Fix · ✅ Success · Started 12:33 AM UTC · Completed 12:57 AM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.56 |
- sticky.FindMarkedComment: match a marker as a body prefix instead of
an unanchored substring. Post always writes marker+"\n"+body (and
BuildUpdatedBody preserves that prefix across updates), so a
legitimate sticky comment's body always starts with its marker.
Contains-based matching let postMissingRiskAssessment /
clearMissingRiskAssessmentDiagnostic select (and overwrite) an
unrelated bot-authored comment that merely quoted another marker
string mid-body — plausible now that ADR 0089 and this PR document
the literal diagnostic marker text. Added regression tests in both
internal/sticky and internal/cli covering the exact collision
scenario (a review comment quoting the diagnostic marker).
- postreview.go: parseReviewResult now rejects a successfully-decoded
risk_assessment that lacks a usable score (1-5) and level (the ADR
0089 enum) — e.g. {} or an object missing those keys decodes into a
zero-value struct rather than leaving the pointer nil, which was
silently treated as a present assessment. Added
hasUsableRiskAssessment and tests for {}, a missing-fields object, an
out-of-range score, and an unknown level, plus an end-to-end
postMissingRiskAssessment test for the {} case.
- pi_personas.go / fullsend-agent.js: route the fields interpolated
into fullsend:persona:skip and fullsend:persona:reject marker lines
(persona name, skip/reject reason) through the same substitution
rules as runtime.sanitizeOutput before writing to stderr, not just a
\r\n\t collapse. A skipped persona's file-derived name (never passed
ValidSubagentKey) and reason text can carry "::"-delimited GHA
workflow-command syntax or %0A-encoded newlines that the old
collapse-only formatPersonaLogName let through unbroken. Mirrored the
same rules in the JS logPersona helper's sanitizePersonaLogField,
since subagent_type, reject reasons, model, and stopReason are
equally agent-controlled there. Extended the Go and JS test suites
with cases for "::error::"-style and %0A-encoded payloads in both the
name and reason fields.
Pre-commit could not fetch remote hook repositories (sandboxed network
policy); ran the equivalent hooks directly instead (gofmt, go vet,
gitleaks, trailing-whitespace/EOF/merge-conflict/CRLF checks) — all
pass. go test and node --test both pass for every changed package;
golangci-lint (installed ad hoc via `go run`) reports zero findings in
any changed file. Two pre-existing, unrelated test failures were
observed and left alone (TestPostScriptEnv_NoSchemaAppendedWhen* and
TestStartFetchService_*) — verified to fail identically on the
pre-fix commit and traced to this sandbox's own environment (a leaked
FULLSEND_OUTPUT_SCHEMA var and a network policy returning 403 for a
local health probe), not to any code this PR touches.
Addresses #7388
🔧 Fix agent — iteration 4 (bot-triggered)Addressed all three findings from the re-review: hardened sticky.FindMarkedComment to match by body prefix instead of unanchored substring (fixing the marker-collision risk for postMissingRiskAssessment/clearMissingRiskAssessmentDiagnostic and any other sticky caller), made parseReviewResult reject a risk_assessment object lacking a usable score/level instead of treating a zero-value struct as present, and routed persona-skip/reject marker-line fields (name and reason in Go, name/reason/model/stopReason in JS) through the same substitution rules as runtime.sanitizeOutput so GHA workflow-command syntax and encoded newlines can no longer reach the workflow log unbroken. Added targeted regression tests for each fix in both Go and JS. No findings were disagreed with — all three were verified against the code and fixed at the root cause each remediation pointed to. Fixed (3):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 12:59 AM UTC · Completed 1:15 AM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.54 |
|
🤖 Finished Fix · ✅ Success · Started 1:17 AM UTC · Completed 1:26 AM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $0.94 |
Allowlist ra.Level through sanitizeRiskLevel before interpolating it into the unusable-risk_assessment WARNING line in parseReviewResult, matching the allowlisting this PR already applies at other print sites. Adds a regression test asserting a level of "::error::pwned" appears in stderr only as "unknown". Note: pre-commit could not run (network-blocked hook-repo fetch); ran the local hooks directly instead (gofmt, go vet, lint-broken-symlinks). Addresses #7388
🔧 Fix agent — iteration 5 (bot-triggered)This re-review round surfaced one new low-severity finding (the rest of the review body is prior-round history already marked resolved). Fixed by routing ra.Level through the existing sanitizeRiskLevel allowlist before it reaches the unusable-risk_assessment WARNING line, and added a test covering a malicious level value. Fixed (1):
Tests: passedNext steps:
|
|
🤖 Finished Review · ✅ Success · Started 1:28 AM UTC · Completed 1:46 AM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.67 |
Superseded by updated review
|
This pull request has been automatically marked as stale because it has not had any activity in the last month. It will be closed in 2 weeks if no further activity occurs. Remove the |
Summary
Risk-assessment comments were missing on ~15% of review runs (e.g. PR #7385) with no log marker and no PR-visible diagnostic. This change makes those misses observable in workflow logs and on the PR.
Related Issue
#7387
Changes
fullsend:persona:{invoke,complete,error,reject,skip}lines for every named persona, includingrisk-assessment. Session shutdown logsskip reason=never-dispatchedfor registered personas the orchestrator never called.fullsend post-reviewparses optionalrisk_assessment. WhenREVIEW_RISK_ASSESSMENT_ENABLEDis on (ADR 0089 default) and the field is absent, it posts a sticky diagnostic: Risk assessment unavailable this run. Failures posting the diagnostic do not block the review verdict.The scored
Risk Assessment: N/5comment is still produced bypost-review.shin fullsend-ai/agents when the field is present. This PR covers the missing-field case in the CLI that script already invokes.Testing
node --test internal/runtime/pi_extension/fullsend-agent.test.mjs— 56 pass, 1 skip (persona lifecycle invoke/complete/error/reject/skip).riskAssessmentEnabled100%,postMissingRiskAssessment93.8%,parseReviewResult/sanitizeReviewResult100%,resolvePersonaModels98.2%.go vet ./internal/cli/ ./internal/runtime/— pass.Checklist
Closes #7387
Post-script verification
agent/7387-risk-assessment-observability)200a184607d39eeb6b2e8158f8ff698d19337ccb..HEAD)