Skip to content

feat(verify-pr): re-sync FULLSEND_OUTPUT_DIR sandbox dual-mode onto SKILL.md - #286

Merged
mrizzi merged 8 commits into
RHEcosystemAppEng:TC-5787-verify-pr-fullsend-v037from
mrizzi:TC-5812
Sep 3, 2026
Merged

mrizzi merged 8 commits into
RHEcosystemAppEng:TC-5787-verify-pr-fullsend-v037from
mrizzi:TC-5812

Conversation

@mrizzi

@mrizzi mrizzi commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Re-applies the sandbox dual-mode adaptation onto the current verify-pr SKILL.md, gated on the FULLSEND_OUTPUT_DIR environment variable. When the var is unset (marketplace/interactive users), behavior is unchanged. When set (fullsend sandbox), the skill runs without Jira/GitHub tokens: it reads pre-fetched task and PR data and accumulates every write as a structured action.

Changes

Purely additive to plugins/sdlc-workflow/skills/verify-pr/SKILL.md (228 insertions, no deletions):

  • Step 0.6 – Sandbox Mode Detection — detects FULLSEND_OUTPUT_DIR; in sandbox mode, no direct Jira writes / GitHub reads / PR comment posts, and initializes the { "report": {}, "actions": [] } accumulator.
  • Step 0.7 – Load Pre-Fetched Data — reads /sandbox/workspace/.pre-script/verify-pr-input.json (the TC-5810 prefetch contract), skips Steps 1–3 and all GitHub reads, with a fallback to direct APIs if the file is missing/invalid.
  • Inline sandbox-mode notes on each read/write step (Steps 2, 3, 4a, 5a, 6d, 6e, 7b, 9).
  • Step 9 – Sandbox Mode Output — populates the report object, appends a post_report action, and writes the complete result to $FULLSEND_OUTPUT_DIR/agent-result.json.

Contract alignment

Every emitted action type matches the verify-pr-result.schema.json enum: create_subtask, create_link, post_pr_reply, post_pr_comment, create_root_cause_task, post_comment, post_report. link_type values use the schema's Blocks/Related enum.

Verification

  • uvx skillsaw — 0 errors (Grade A); context-budget finding is warn-level (non-blocking; CI strict mode disabled).
  • claude plugin validate plugins/sdlc-workflow — validation passed.
  • No scripts/ changes → pytest not required.
  • Interactive path unchanged (all changes gated on FULLSEND_OUTPUT_DIR).

Implements TC-5812

🤖 Generated with Claude Code

Review feedback fixes (Sourcery-ai on PR #286)

Four follow-up sub-tasks address review comments on this PR:

  • TC-5980 — Step 0.7 now validates the prefetched verify-pr-input.json against verify-pr-input.schema.json (jsonschema), not just JSON syntax; malformed-but-parseable input is rejected.
  • TC-5981 — On bad/missing prefetch input, sandbox mode fails fast: it writes an {"error": ...} result (no credentialed fallback is possible tokenless) so the runner's output validation surfaces it as a hard failure. Interactive mode keeps warn-and-fall-back.
  • TC-5982 — Idempotency checks (Steps 6d/6f/7c) now run tokenlessly in the sandbox: pre-verify-pr.sh prefetches related-issue metadata (summary, labels, description, issuetype, comments) into verify-pr-input.json under a new optional idempotency key.
  • TC-5983 — execute_post_report now posts the Jira comment from report_adf (via adf_to_markdown), matching Step 9's contract; GitHub still gets report_md + its commit marker.
  • TC-6026 — Reconcile the optional idempotency bundle: the schema now requires idempotency.related_issues, so Step 0.7 rejects a prefetch omitting it before the tokenless Steps 6d/6f/7c dedup checks run (fail-fast, per TC-5980/TC-5981). pre_verify_pr.py always emits the bundle (empty list when the task has no relations); SKILL.md states the always-present invariant.
  • TC-6032 — Tighten the idempotency contract at the item level: idempotency.related_issues.items now requires the six fields the dedup checks consume (key, summary, labels, description, issuetype, comments), so a structurally incomplete prefetched entry fails Step 0.7 instead of being silently treated as non-matching. build_idempotency_bundle already emits all six; a test asserts an incomplete entry raises ValidationError.
  • TC-6033 — Step 0.6 now uses presence detection (${FULLSEND_OUTPUT_DIR+x}) and fails fast on an exported-but-empty value, instead of ${FULLSEND_OUTPUT_DIR:-not-set} which treated exported-but-empty as unset and fell back to the credentialed interactive path despite the sandbox gate being present.

Verification: python3 -m pytest plugins/sdlc-workflow/scripts/ -q (109 passed), uvx skillsaw (0 errors, Grade A), claude plugin validate plugins/sdlc-workflow (passed).

Summary by Sourcery

Enable verify-pr to run safely in tokenless sandboxes while retaining the existing interactive workflow.

New Features:

  • Add sandbox execution for verify-pr that consumes validated pre-fetched task and GitHub data and emits structured actions and reports without Jira or GitHub credentials.

Bug Fixes:

  • Fail fast on missing or invalid sandbox input instead of attempting an unavailable credentialed fallback.
  • Preserve tokenless idempotency checks by requiring and prefetching related-issue metadata.
  • Send Jira report comments from tracker-native report content rather than GitHub-specific markdown and commit markers.

Enhancements:

  • Align sandbox actions, link types, and report output with the verify-pr result schema while preserving interactive behavior.

Documentation:

  • Document sandbox detection, prefetch validation, tokenless processing, structured actions, and result output for verify-pr.

Tests:

  • Expand coverage for related-issue prefetching, schema validation, sandbox idempotency data, and tracker-specific report posting.

…KILL.md

Re-apply the sandbox dual-mode adaptation onto the current verify-pr
SKILL.md, gated on the FULLSEND_OUTPUT_DIR env var so interactive
(marketplace) behavior is unchanged when it is unset.

When set (fullsend sandbox): skip Jira/GitHub API calls, read pre-fetched
task and PR data from the mounted verify-pr-input.json, accumulate every
write as an action in { "report": {}, "actions": [] }, and write the
result to $FULLSEND_OUTPUT_DIR/agent-result.json. Adds Step 0.6 (Sandbox
Mode Detection), Step 0.7 (Load Pre-Fetched Data), inline sandbox-mode
notes on each read/write step, and a Step 9 Sandbox Mode Output block.

Every emitted action type matches the verify-pr-result.schema.json enum
(create_subtask, create_link, post_pr_reply, post_pr_comment,
create_root_cause_task, post_comment, post_report); link_type values use
the schema's Blocks/Related enum.

Implements TC-5812

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

This PR re-syncs verify-pr with the FULLSEND_OUTPUT_DIR sandbox contract: trusted prefetching supplies all task, PR, and idempotency reads; sandbox execution validates inputs, avoids credentials and network access, emits schema-aligned deferred actions, and writes a structured report, while interactive execution remains unchanged.

Sequence diagram for tokenless verify-pr sandbox flow

sequenceDiagram
    participant Runner as Trusted pre-script runner
    participant Sandbox as verify-pr sandbox
    participant Output as agent-result.json
    participant Post as post-script executor
    participant Jira as Jira
    participant GitHub as GitHub

    Runner->>Runner: pre-verify-pr.sh fetches task, PR, and related issues
    Runner->>Sandbox: provide verify-pr-input.json
    Sandbox->>Sandbox: validate input schema
    alt input valid
        Sandbox->>Sandbox: inspect prefetched task, GitHub, and idempotency data
        Sandbox->>Sandbox: accumulate create_subtask, create_link, and comment actions
        Sandbox->>Sandbox: build report and append post_report
        Sandbox->>Output: write report and actions
        Post->>Output: read result
        Post->>GitHub: execute post_report using report_md
        Post->>Jira: execute post_report using report_adf
        Post->>Jira: execute deferred Jira actions
        Post->>GitHub: execute deferred PR actions
    else input missing or invalid
        Sandbox->>Output: write structured error result
        Output-->>Post: validation failure
    end
Loading

Flow diagram for verify-pr dual-mode selection

flowchart TD
    Start[Start verify-pr] --> Mode{FULLSEND_OUTPUT_DIR set?}
    Mode -->|No| Interactive[Interactive mode]
    Interactive --> Direct[Read Jira and GitHub; execute writes directly]
    Mode -->|Yes| Sandbox[Sandbox mode]
    Sandbox --> Validate[Validate verify-pr-input.json]
    Validate -->|Invalid| Fail[Write error result and stop]
    Validate -->|Valid| Prefetched[Use prefetched task, PR, and idempotency data]
    Prefetched --> Actions[Accumulate schema-aligned actions]
    Actions --> Report[Write report and actions to agent-result.json]
Loading

File-Level Changes

Change Details Files
Add gated tokenless sandbox execution to the verify-pr skill while preserving interactive behavior.
  • Detect FULLSEND_OUTPUT_DIR and route reads from prefetched task/GitHub data.
  • Validate prefetched input against the schema and fail fast with a structured error when sandbox input is invalid.
  • Accumulate Jira and GitHub writes as schema-aligned actions and emit the final report to agent-result.json.
  • Use prefetched related-issue metadata for tokenless idempotency checks.
  • Document sandbox handling across verification, deduplication, task creation, comments, and reporting.
plugins/sdlc-workflow/skills/verify-pr/SKILL.md
plugins/sdlc-workflow/schemas/verify-pr-input.schema.json
Extend prefetching to provide the related-issue metadata required for sandbox idempotency checks.
  • Enumerate and fetch sub-task and linked-issue keys with the fields needed by deduplication logic.
  • Always emit an idempotency.related_issues bundle, including an empty list when no relations exist.
  • Add transformation and CLI support for embedding the bundle in verify-pr-input.json.
plugins/sdlc-workflow/scripts/pre-verify-pr.sh
plugins/sdlc-workflow/scripts/pre_verify_pr.py
Correct deferred report execution so Jira receives the tracker-native report body while GitHub retains its Markdown-specific content and marker.
  • Resolve report_adf references and render ADF to Markdown for Jira posting.
  • Continue using report_md plus the commit marker for GitHub comments.
  • Add coverage for native Jira rendering and reference resolution.
plugins/sdlc-workflow/scripts/execute-actions.py
plugins/sdlc-workflow/scripts/test_execute_actions.py
Expand regression coverage for idempotency prefetching, schema enforcement, and sandbox preparation.
  • Test related-key collection, metadata bundling, defaults, and CLI transformations.
  • Verify schema acceptance of complete bundles and rejection of missing idempotency data.
  • Guard the shell prefetch workflow and preserve existing script behavior.
plugins/sdlc-workflow/scripts/test_pre_verify_pr.py

Possibly linked issues

  • #TC-6026: PR directly implements TC-6026: schema validation requires the bundle, prefetch emits it, and tests cover missing and valid data.
  • #TC-5980: PR adds Step 0.7 schema validation and tests rejecting syntactically valid but schema-invalid pre-fetched input.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="plugins/sdlc-workflow/skills/verify-pr/SKILL.md" line_range="111" />
<code_context>
+(where the tokens live) and mounts it read-only into the sandbox. Read it:
+
+```bash
+cat /sandbox/workspace/.pre-script/verify-pr-input.json | python3 -m json.tool > /dev/null 2>&1 && echo "Pre-fetched data available" || echo "ERROR: Pre-fetched data missing or invalid"
+```
+
</code_context>
<issue_to_address>
**issue (bug_risk):** The pre-fetched input check validates only JSON syntax, not `verify-pr-input.schema.json`; a syntactically valid object with missing or malformed `task` or `github` fields is accepted and later field accesses fail or use incomplete data instead of taking the documented invalid-input path.

**Triggers:** When the mounted file contains valid JSON that violates the input schema.

**Suggested fix:** Validate the file against `verify-pr-input.schema.json` before treating it as available.
</issue_to_address>

### Comment 2
<location path="plugins/sdlc-workflow/skills/verify-pr/SKILL.md" line_range="128" />
<code_context>
+  Step 9's HEAD-SHA retrieval — using the bundle fields instead.
+
+If the file is missing or invalid, log a warning and fall back to Steps 1–3 and the
+direct GitHub reads (interactive behavior).
+
 ## Inputs
</code_context>
<issue_to_address>
**issue (broader_impact):** The documented fallback from a missing or invalid prefetch file returns to direct Jira and GitHub API behavior even though `FULLSEND_OUTPUT_DIR` explicitly denotes a sandbox with no tokens, no `gh` CLI, and no network egress; the fallback therefore fails in the very environment this mode is intended to support.

**Triggers:** When the pre-fetched input file is missing or invalid in sandbox mode.

**Suggested fix:** Fail clearly with a structured result, or have the trusted pre-script repair/report the prefetch failure; do not fall back to credentialed APIs inside the sandbox.
</issue_to_address>

### Comment 3
<location path="plugins/sdlc-workflow/skills/verify-pr/SKILL.md" line_range="590-591" />
<code_context>

 jira.create_issue_link(type="Blocks", inwardIssue=<sub-task-id>, outwardIssue=<parent-task-id>)

+**Sandbox mode:** Instead of calling `jira.create_issue` and `jira.create_issue_link`,
+append a `create_subtask` action and a `create_link` action. Use this pattern for **all**
+sub-task categories below (review feedback, CI failure, eval failure):
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Sandbox mode replaces sub-task and root-cause writes with actions but provides no sandbox path for the Jira reads required by the idempotency checks: the prefetch contains only shallow `issue_links`, not linked sub-task labels/descriptions or root-cause comments, and Step 7c still requires fetching linked-task comments. The agent therefore cannot perform the existing deduplication checks without attempting unavailable Jira reads and will either fail or emit duplicate tasks.

**Triggers:** When prior review-feedback or root-cause tasks already exist and the run reaches the corresponding idempotency check.

**Suggested fix:** Prefetch the linked-task metadata and root-cause comments, or add explicit sandbox read actions/data and use them for these checks.
</issue_to_address>

### Comment 4
<location path="plugins/sdlc-workflow/skills/verify-pr/SKILL.md" line_range="1215-1218" />
<code_context>
+### Sandbox Mode Output
+
+**Sandbox mode only:** Instead of posting to GitHub and Jira directly, populate the
+`report` object and append a single `post_report` action. The runner's `post_script`
+posts the GitHub PR comment (from `report_md`, applying the commit-scoped
+find-then-update-or-create path above) and the Jira comment (from `report_adf`) after
+the sandbox exits.
+
+Populate `report` (all fields required by
</code_context>
<issue_to_address>
**issue (bug_risk):** The sandbox contract says the runner posts the Jira comment from `report_adf`, but the actual `execute_post_report` implementation posts `report_md` to Jira and ignores `report_adf`; the deferred report therefore loses the required ADF representation and can lose the mandated Jira footnote/formatting.

**Triggers:** On every successful sandbox run that emits `post_report`.

**Suggested fix:** Make the runner consume `report_adf` for the Jira comment, or change the skill contract to emit and validate the representation the runner actually uses.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 4 findings to address first, and if the sandbox instructions or action mappings are wrong, the agent could create incorrect Jira subtasks/links or post incorrect GitHub and Jira comments that persist after this file is reverted. Those records and messages require manual cleanup; reverting only prevents future executions from using the behavior.

Blocking findings: plugins/sdlc-workflow/skills/verify-pr/SKILL.md:111, plugins/sdlc-workflow/skills/verify-pr/SKILL.md:128, plugins/sdlc-workflow/skills/verify-pr/SKILL.md:591, plugins/sdlc-workflow/skills/verify-pr/SKILL.md:1218


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@mrizzi

mrizzi commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

verify-pr report — TC-5812

Overall: ⚠️ WARN · commit e972d78 · PR #286

Check Verdict
Review Feedback ⚠️ WARN — 4 code-change requests → sub-tasks created
Root-Cause Investigation ✅ DONE — 1 root-cause task (TC-5984)
Scope Containment ✅ PASS
Diff Size ✅ PASS
Commit Traceability ✅ PASS
Sensitive Patterns ✅ PASS
CI Status ✅ PASS
Acceptance Criteria ✅ PASS
Test Quality N/A
Test Change Classification N/A
Verification Commands ✅ PASS

Review Feedback (WARN)

Four inline review comments from sourcery-ai[bot] were classified as code change requests; each has a blocking sub-task and a threaded reply:

Sub-task Comment Issue
TC-5980 3915109206 Step 0.7 reads pre-fetched data but never validates it against verify-pr-input.schema.json
TC-5981 3915109216 Step 0.7 documents an impossible sandbox fallback to direct GitHub reads
TC-5982 3915109227 Idempotency path documents Jira reads with no sandbox-mode equivalent
TC-5983 3915109239 Step 9 says the runner posts from report_adf, but execute-actions.py posts report_md

All four sub-tasks block TC-5812.

Root-Cause Investigation (DONE)

The four defects share one systemic root cause in the plan-feature phase: TC-5787's deliverable is a single contract split across four sibling artifacts (schema, producer, consumer, orchestration doc), but decomposition scoped each task's acceptance criteria to one artifact in isolation, so no task owned cross-artifact consistency. TC-5984 captures the fix: plan-feature must detect contract-split Features and require cross-artifact-consistency criteria (reads-have-a-source, inputs-validated-against-schema, documented-behaviors-feasible-under-the-trust-model, doc-vs-runner field agreement).

Other checks

Diff is purely additive to SKILL.md (228 insertions, sandbox-mode notes only, gated on FULLSEND_OUTPUT_DIR) — scope contained, no sensitive patterns, commits traceable, no source/test code changed (Test Quality / Test Change Classification N/A). No scripts/ changes ⇒ pytest not required; plugin validation passes.

Note: This report is informational. It does not merge the PR or transition the Jira issue — a human reviewer decides.


This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9.

mrizzi and others added 4 commits September 2, 2026 18:28
Step 0.7 treated verify-pr-input.json as usable if it merely parsed as
JSON. A syntactically valid but structurally incomplete prefetch (missing
the github bundle or task fields) passed the check and then failed deep
inside a later step. Replace the json.tool syntax check with jsonschema
validation against verify-pr-input.schema.json so a structurally
invalid-but-parseable prefetch is treated as invalid, mirroring the
existing validate-output-schema.sh pattern.

Implements TC-5980

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…aled fallback

Step 0.7 documented that a missing/invalid prefetch falls back to Steps
1-3 and direct Jira/GitHub reads. That "interactive" fallback cannot work
in sandbox mode (FULLSEND_OUTPUT_DIR set): the sandbox has no tokens, no
gh CLI, and no network egress, so credentialed calls fail obscurely or
hang. Split the failure handling: in sandbox mode write a structured
error to $FULLSEND_OUTPUT_DIR/agent-result.json and stop (the runner's
output validation surfaces it as a hard failure), while interactive mode
keeps the existing warn-and-fall-back behavior unchanged.

Implements TC-5981

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Steps 6d/6f/7c dedupe against the task's existing sub-tasks and linked
root-cause issues via Jira reads, but the sandbox has no Jira token, so
those idempotency checks had no data source and risked creating duplicate
sub-tasks/root-cause tasks on re-run. Prefetch the read data on the
trusted runner (split-trust): pre-verify-pr.sh enumerates the task's
related keys and fetches each (summary, labels, description, issuetype,
comments), and pre_verify_pr.py embeds them under a new `idempotency`
bundle in verify-pr-input.json. SKILL.md Steps 0.7/6d/6f/7c now read
`idempotency.related_issues` in sandbox mode instead of calling Jira; the
input schema documents the new (optional) bundle. No token or egress is
added to the sandbox.

Implements TC-5982

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
execute_post_report posted the GitHub-only report_md (with its embedded
commit marker) to Jira and never read report_adf, contradicting SKILL.md
Step 9's documented contract. Render the Jira comment body from report_adf
via adf_to_markdown so the tracker-native report (and its Comment Footnote)
is used; GitHub keeps report_md plus the commit-scoped marker.

Implements TC-5983

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mrizzi

mrizzi commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

@sourcery-ai review

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="plugins/sdlc-workflow/skills/verify-pr/SKILL.md" line_range="127-128" />
<code_context>
+  the already-checked-out PR-head tree. **Skip every GitHub read step** — Step 3
+  (checkout), Step 4a (review/comment fetches), Step 5a (diff/stat/commits), and
+  Step 9's HEAD-SHA retrieval — using the bundle fields instead.
+- Read the `idempotency.related_issues` array (each entry has `key`, `summary`,
+  `labels`, `description`, `issuetype`, `comments`) — the task's existing sub-tasks
+  and linked issues, prefetched on the runner. **Use it for every idempotency read**
+  (Steps 6d, 6f, 7c) instead of calling Jira; the sandbox has no token. The array is
+  empty when the task has no sub-tasks or linked issues yet.
+
+If the validation above fails (file missing, not valid JSON, or not conforming to the
</code_context>
<issue_to_address>
**issue (broader_impact):** Sandbox input validation accepts a structurally valid prefetch that omits the optional `idempotency` bundle, but the sandbox instructions then unconditionally rely on `idempotency.related_issues` for Steps 6d, 6f, and 7c. Those idempotency checks have no tokenless data source in that case and the run proceeds without deduplication, creating duplicate sub-tasks or root-cause tasks on reruns.

**Triggers:** When an older or manually supplied input bundle contains the required task/GitHub fields but omits `idempotency`.

**Suggested fix:** Require `idempotency.related_issues` in sandbox input validation, or treat a missing bundle as invalid and fail fast instead of continuing without idempotency data.

```suggestion
    validate(instance=instance, schema=schema)
    if (
        not isinstance(instance, dict)
        or not isinstance(instance.get("idempotency"), dict)
        or not isinstance(instance["idempotency"].get("related_issues"), list)
    ):
        raise ValidationError(
            "required idempotency.related_issues array is missing"
        )
    print("Pre-fetched data available")
```
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 1 finding to address first, and if the sandbox-mode branching or prefetched idempotency data is wrong, the runner could create duplicate Jira tasks or links and post incorrect Jira/GitHub comments without live human intervention; those records and messages outlive a revert and require cleanup. Reverting restores the prior behavior but does not remove actions already executed.

Blocking findings: plugins/sdlc-workflow/skills/verify-pr/SKILL.md:128


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@mrizzi

mrizzi commented Sep 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Verification Report for TC-5812 (commit df76f1b)

Check Result Details
Review Feedback WARN 7 classifiable items enumerated (5 inline threads + 2 sourcery review bodies); all classified. This run added transparency replies for the 2 review bodies (5090904620, 5093004840) — aggregated restatements of the inline findings, deduplicated to existing sub-tasks. Sub-tasks: TC-5980–5983 (original 4) + TC-6026 (optional idempotency bundle vs unconditional consumers). No new sub-tasks this run.
Root-Cause Investigation N/A No new sub-tasks created this run (all review feedback deduplicated to existing sub-tasks); systemic root cause remains tracked by TC-5984
Scope Containment WARN 6 files changed beyond the sole task file (SKILL.md); all are review-feedback fixes for TC-5980–5983, justified
Diff Size PASS 643 insertions / 12 deletions across 7 files
Commit Traceability WARN 1/5 commits reference TC-5812; the other 4 reference child sub-tasks TC-5980–5983; all 5 carry both Assisted-by and Co-Authored-By trailers
Sensitive Patterns PASS No secrets, tokens, or credentials introduced
CI Status PASS All 5 checks passing
Acceptance Criteria PASS All acceptance criteria met
Test Quality WARN Repetitive test pair (test_build_idempotency_bundle_extracts_fields_and_comments / _handles_missing_fields) and 4 missing docstrings in test_pre_verify_pr.py; Eval Quality: N/A
Test Change Classification ADDITIVE Only new tests added (test_execute_actions.py +39, test_pre_verify_pr.py +174)
Verification Commands PASS python3 -m pytest plugins/sdlc-workflow/scripts/ -q → 107 passed

Overall: WARN

All sourcery feedback is now fully enumerated and classified: the 5 inline threads and both review-body summaries. Every finding is tracked by an existing sub-task (TC-5980–5983, TC-6026) — no duplicates created. Domain sub-agent verdicts are unchanged from the prior verification at this same commit (df76f1b). The sole open merge blocker is TC-6026; the systemic root cause is tracked by TC-5984. Scope Containment, Commit Traceability, and Test Quality warnings are informational or justified. No FAILs.


This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9.

@mrizzi

mrizzi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

[sdlc-workflow/verify-pr] Re: @sourcery-ai[bot] review (5090904620) — Classified as code change request (4 findings). This review body aggregates the four inline comments already tracked by sub-tasks: TC-5980 (SKILL.md:111 — validate prefetch against schema), TC-5981 (:128 — no credentialed fallback in sandbox), TC-5982 (:591 — sandbox path for idempotency Jira reads), and TC-5983 (:1218 — report_adf/report_md contract). Existing sub-tasks cover this feedback — no new sub-task created (deduplicated).

@mrizzi

mrizzi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

[sdlc-workflow/verify-pr] Re: @sourcery-ai[bot] review (5093004840) — Classified as code change request — this review body aggregates the inline comment at SKILL.md:127-128 already tracked by sub-task TC-6026 (schema makes the idempotency bundle optional while Steps 0.7/6d/6f/7c consume idempotency.related_issues unconditionally). Existing sub-task covers this feedback — no new sub-task created (deduplicated).

The TC-5982 fix added an optional top-level `idempotency` object to
verify-pr-input.schema.json, but the sandbox SKILL.md steps read
`idempotency.related_issues` unconditionally for the Steps 6d/6f/7c dedup
checks. A prefetch that passed Step 0.7 validation while omitting the bundle
left those tokenless checks with no data source, silently skipping dedup and
creating duplicate sub-tasks / root-cause tasks on reruns.

Option 1 (require it): the producer always runs on the trusted runner with a
Jira token, so it can always emit the bundle (empty related_issues when the
task has no relations). Make the schema require `idempotency` and its
`related_issues`, so Step 0.7 rejects a prefetch lacking it before any
consuming step runs — consistent with the TC-5980/TC-5981 fail-fast contract.
transform_to_input now always emits the bundle; SKILL.md states the invariant.

Implements TC-6026

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Assisted-by: Claude Code
@mrizzi

mrizzi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

@sourcery-ai review

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="plugins/sdlc-workflow/skills/verify-pr/SKILL.md" line_range="127-128" />
<code_context>
+- Read the `idempotency.related_issues` array (each entry has `key`, `summary`,
</code_context>
<issue_to_address>
**issue (bug_risk):** The input schema requires the `idempotency.related_issues` array itself but does not require `key`, `labels`, `description`, `issuetype`, or `comments` on each array item. A structurally incomplete entry therefore passes Step 0.7 and the sandbox dedup checks silently treat it as non-matching, allowing duplicate subtasks or root-cause tasks to be created.

**Triggers:** When a prefetch contains a related-issue object with missing metadata fields.

**Suggested fix:** Require the fields consumed by Steps 6d, 6f, and 7c in each `related_issues` item, or reject incomplete entries before execution.

```suggestion
    validate(instance=instance, schema=schema)
    required_related_fields = {"key", "labels", "description", "issuetype", "comments"}
    for index, related_issue in enumerate(instance["idempotency"]["related_issues"]):
        missing_fields = required_related_fields - related_issue.keys()
        if missing_fields:
            missing = ", ".join(sorted(missing_fields))
            raise ValidationError(
                f"related_issues[{index}] is missing required fields: {missing}"
            )
    print("Pre-fetched data available")
```
</issue_to_address>

### Comment 2
<location path="plugins/sdlc-workflow/skills/verify-pr/SKILL.md" line_range="66" />
<code_context>
+
+Check whether the `FULLSEND_OUTPUT_DIR` environment variable is set:
+
+```bash
+echo ${FULLSEND_OUTPUT_DIR:-not-set}
+```
+
</code_context>
<issue_to_address>
**issue (bug_risk):** The detection example uses `${FULLSEND_OUTPUT_DIR:-not-set}`, which treats an exported-but-empty `FULLSEND_OUTPUT_DIR` as unset. That causes the skill to select interactive mode and attempt credentialed Jira/GitHub operations even though the sandbox mode gate is present in the environment.

**Triggers:** When the runner exports `FULLSEND_OUTPUT_DIR` with an empty value instead of omitting it.

**Suggested fix:** Detect presence with `${FULLSEND_OUTPUT_DIR+x}` and separately reject an empty output directory, or require a non-empty value before entering sandbox mode.

```suggestion
if [ "${FULLSEND_OUTPUT_DIR+x}" = x ]; then
  if [ -z "$FULLSEND_OUTPUT_DIR" ]; then
    echo "FULLSEND_OUTPUT_DIR is set but empty"
    exit 1
  fi
  echo "$FULLSEND_OUTPUT_DIR"
else
  echo not-set
fi
```
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 2 findings to address first, and if the sandbox-mode workflow or prefetched idempotency data is wrong, it can create duplicate Jira sub-tasks and links or post incorrect Jira and GitHub comments after the runner executes the actions. Reverting stops future runs but does not automatically remove records, links, or externally posted messages already created.

Blocking findings: plugins/sdlc-workflow/skills/verify-pr/SKILL.md:128, plugins/sdlc-workflow/skills/verify-pr/SKILL.md:66


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@mrizzi

mrizzi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

[sdlc-workflow/verify-pr] Re: @sourcery-ai[bot] review (5101905003) — Classified as code change request (2 findings).

  • Comment 1 (schema idempotency.related_issues items lack required per-item fields) → TC-6032
  • Comment 2 (Step 0.6 ${FULLSEND_OUTPUT_DIR:-not-set} treats exported-but-empty as unset) → TC-6033

Both Blocks TC-5812. The review body restates inline comments 3924536285 and 3924536305, which are tracked by the sub-tasks above.

@mrizzi

mrizzi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

verify-pr report — TC-5812 (commit 5bb07cc)

Overall: ⚠️ WARN — no FAIL. Two new blocking sub-tasks opened from this commit's review.

Check Result Details
Review Feedback ⚠️ WARN 2 new sourcery bug_risk code change requests on commit 5bb07cc → TC-6032, TC-6033 (both Blocks TC-5812)
Scope Containment ⚠️ WARN Sole parent-task file SKILL.md present; 6 extra files are justified child sub-task fixes (TC-5980–5983, TC-6026) on the same PR branch
Diff Size ✅ PASS 697 insertions / 14 deletions across 7 files — proportionate to parent + 5 review sub-tasks
Commit Traceability ✅ PASS All 6 commits reference TC-5812 or a direct sub-task via Implements; both trailers present on each
Sensitive Patterns ✅ PASS No secrets/credentials in any added line
CI Status ✅ PASS All 5 checks pass (Script Unit Tests 3.11–3.14, Sourcery review)
Acceptance Criteria ⚠️ WARN Dual-mode implemented, gated on FULLSEND_OUTPUT_DIR, action types all schema-enumerated, trailers present — but the two latent gaps (TC-6032/TC-6033) touch the "gated on FULLSEND_OUTPUT_DIR" guarantee
Verification Commands ✅ PASS Static gate (a) matches (8 FULLSEND_OUTPUT_DIR refs); pytest 108 passed. Gate (b) exit-1 is a benign false positive — its regex matches pre-existing ADF node types (rule/paragraph/text/link) in the Comment-Footnote example, not action types; all real action types are schema-enumerated
Test Quality ⚠️ WARN Repetitive build_idempotency_bundle test pair (parameterization candidate) + 3 new tests missing docstrings in test_pre_verify_pr.py
Test Change Classification ✅ ADDITIVE +12 tests, 0 removed, ~45 assertions added, one strengthened; no coverage weakened

Root-Cause Investigation: SKIPPED (idempotent). TC-6032 (schema does not require per-item idempotency.related_issues fields) is the same cross-artifact contract-consistency seam already tracked by root-cause TC-5984; TC-6033 (Step 0.6 empty-env detection) is a localized robustness bug, not a systemic plan-feature gap. No new root-cause task.

New merge blockers (this run):

  • TC-6032 — require per-item fields in idempotency.related_issues so an incomplete prefetch entry fails Step 0.7 instead of silently skipping dedup (item-level follow-on to TC-6026). ← sourcery 3924536285
  • TC-6033 — Step 0.6 must treat an exported-but-empty FULLSEND_OUTPUT_DIR as a hard error, not interactive mode. ← sourcery 3924536305

Classification replies: inline 3924866359 (→TC-6032), 3924866594 (→TC-6033); review-body 5526289494 (→both).

verify-pr did not merge the PR and did not transition the Jira issue.

mrizzi and others added 2 commits September 3, 2026 16:08
The idempotency.related_issues array is required (TC-6026) but its items
declared properties without a required list, so a prefetched related-issue
entry missing key/summary/labels/description/issuetype/comments passed Step
0.7 schema validation. The tokenless-sandbox dedup checks (Steps 6d/6f/7c)
then silently treated the incomplete entry as non-matching, letting duplicate
sub-tasks and root-cause tasks be created on reruns — the item-level analogue
of the array-level gap TC-6026 closed.

Add a per-item required list of the six fields the dedup checks consume.
build_idempotency_bundle already emits all six unconditionally, so the
producer stays schema-valid. Add a test asserting an incomplete entry raises
ValidationError.

TC-6032

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step 0.6 detected sandbox mode with `echo ${FULLSEND_OUTPUT_DIR:-not-set}`.
The `:-` operator treats an exported-but-empty value the same as unset, so a
runner exporting the gate with an empty value would select the credentialed
interactive path despite the sandbox gate being present — the opposite of the
intended tokenless behavior.

Use presence detection (`${FULLSEND_OUTPUT_DIR+x}`) and reject an empty value
with a non-zero exit. Sandbox mode is selected whenever the variable is present
and non-empty; interactive mode only when it is genuinely unset.

TC-6033

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mrizzi

mrizzi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

@sourcery-ai review

@sourcery-ai

sourcery-ai Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Sorry @mrizzi, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 21 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@mrizzi

mrizzi commented Sep 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Verification Report for TC-5812 (commit 7ca7e8c)

Check Result Details
Review Feedback PASS All 7 inline threads + 3 review body items classified in prior runs; all code change requests addressed via sub-tasks (TC-5980-5983, TC-6026, TC-6032, TC-6033)
Root-Cause Investigation N/A No new sub-tasks created this run
Scope Containment WARN 6 out-of-scope files (schema, scripts, tests) beyond the 1 task-specified file; all are supporting infrastructure for the SKILL.md changes
Diff Size WARN 751 lines changed across 7 files for a 1-file task; disproportionate but explained by supporting infrastructure and review-driven fixes
Commit Traceability WARN 1 of 8 commits references TC-5812 directly; remaining 7 reference sub-task IDs (TC-5980, TC-5981, TC-5982, TC-5983, TC-6026, TC-6032, TC-6033)
Sensitive Patterns PASS No secrets, credentials, or API keys detected in 765 added lines
CI Status WARN Cannot be fetched in sandbox mode (no gh CLI)
Acceptance Criteria PASS Dual-mode added, interactive path unchanged, all emitted action types match schema enum
Test Quality WARN Repetitive Test Detection: WARN (2 schema-validation tests share identical structure); Test Documentation: PASS; Eval Quality: N/A
Test Change Classification ADDITIVE 14 new test functions added; existing test assertions tightened; no tests removed or weakened
Verification Commands PASS FULLSEND_OUTPUT_DIR grep confirms 11 references; orphan-action-type check passes

Overall: WARN

All acceptance criteria are satisfied and verification commands pass. Warnings are informational:

  • Scope Containment / Diff Size: 6 additional files (schema, pre-script, tests) are supporting infrastructure required by the SKILL.md sandbox dual-mode feature and driven by reviewer feedback across 7 fix commits.
  • Commit Traceability: The initial commit references TC-5812; subsequent fix commits reference their respective sub-task IDs (TC-5980-5983, TC-6026, TC-6032, TC-6033), maintaining traceability through the task hierarchy.
  • CI Status: Cannot be determined from sandbox; no gh CLI available.
  • Test Quality: Two schema-validation tests could be parameterized (borderline: each guards a distinct contract).

This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9.

Previous run

Verification Report for TC-5812 (commit 7ca7e8c)

Check Result Details
Review Feedback PASS All 7 inline threads + 3 review body items classified in prior runs; all code change requests addressed via sub-tasks (TC-5980-5983, TC-6026, TC-6032, TC-6033)
Root-Cause Investigation N/A No new sub-tasks created this run
Scope Containment WARN 6 out-of-scope files (schema, scripts, tests) beyond the 1 task-specified file; all are supporting infrastructure for the SKILL.md changes
Diff Size WARN 751 lines changed across 7 files for a 1-file task; disproportionate but explained by supporting infrastructure and review-driven fixes
Commit Traceability WARN 1 of 8 commits references TC-5812 directly; remaining 7 reference sub-task IDs (TC-5980, TC-5981, TC-5982, TC-5983, TC-6026, TC-6032, TC-6033)
Sensitive Patterns PASS No secrets, credentials, or API keys detected in 765 added lines
CI Status WARN Cannot be fetched in sandbox mode (no gh CLI)
Acceptance Criteria PASS Dual-mode added, interactive path unchanged, all emitted action types match schema enum
Test Quality WARN Repetitive Test Detection: WARN (2 schema-validation tests share identical structure); Test Documentation: PASS; Eval Quality: N/A
Test Change Classification ADDITIVE 14 new test functions added; existing test assertions tightened; no tests removed or weakened
Verification Commands PASS FULLSEND_OUTPUT_DIR grep confirms 11 references; orphan-action-type check passes

Overall: WARN

All acceptance criteria are satisfied and verification commands pass. Warnings are informational:

  • Scope Containment / Diff Size: 6 additional files (schema, pre-script, tests) are supporting infrastructure required by the SKILL.md sandbox dual-mode feature and driven by reviewer feedback across 7 fix commits.
  • Commit Traceability: The initial commit references TC-5812; subsequent fix commits reference their respective sub-task IDs (TC-5980-5983, TC-6026, TC-6032, TC-6033), maintaining traceability through the task hierarchy.
  • CI Status: Cannot be determined from sandbox; no gh CLI available.
  • Test Quality: Two schema-validation tests could be parameterized (borderline: each guards a distinct contract).

This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9.

Previous run (2)

🔍 verify-pr report — ⚠️ WARN

Task: TC-5812 · Commit: 7ca7e8cc · plugin v0.13.9

Check Verdict Notes
Review Feedback ✅ PASS All 10 classifiable items (7 inline threads + 3 review bodies) already classified on prior runs; no new items on this HEAD. The two most recent code-change-requests — TC-6032 (per-item schema required) and TC-6033 (empty-FULLSEND_OUTPUT_DIR fail-fast) — are delivered in this commit. No new sub-tasks, no new replies.
Scope Containment ⚠️ WARN Task-declared file skills/verify-pr/SKILL.md present; 6 extra files (verify-pr-input.schema.json, pre-verify-pr.sh, pre_verify_pr.py, execute-actions.py, test_pre_verify_pr.py, test_execute_actions.py) all trace to review-feedback sub-tasks TC-5980..TC-6033. Justified extras, no missing task files.
Diff Size ✅ PASS +751 / −14 across 7 files — proportionate for a multi-sub-task review-feedback PR (doc + schema + scripts + matching tests).
Commit Traceability ✅ PASS All 8 commits reference a Jira ID (parent TC-5812 or child sub-task) and carry both Assisted-by + Co-Authored-By trailers.
Sensitive Patterns ✅ PASS No secrets/keys/credentials in any added line across 7 files. Near-matches (env var names, fabricated SHA/issue-key fixtures, template placeholders) correctly excluded.
CI Status ✅ PASS All 5 checks pass: Script Unit Tests (3.11–3.14) + Sourcery review.
Acceptance Criteria ✅ PASS Dual-mode gated on FULLSEND_OUTPUT_DIR via presence detection ${FULLSEND_OUTPUT_DIR+x} with exported-but-empty fail-fast; interactive path untouched when unset; all 7 emitted action types in the result-schema enum; both commit trailers present.
Verification Commands ✅ PASS Step 0.6 + output block present; orphan-check clean (only benign ADF-node false positives); pytest 109 passed.
Test Quality ⚠️ WARN (informational) Repetitive-test pairs in test_pre_verify_pr.py (idempotency-bundle field tests; schema-rejection tests) — parameterization candidates. 3 new tests lack docstrings (test_related_keys_empty_when_no_relations, test_build_idempotency_bundle_empty, test_transform_with_idempotency_attaches_key).
Test Change Classification 🟢 ADDITIVE (informational) Both modified test files only add/tighten coverage — +14 tests, negative schema-rejection cases, stronger report_adf assertions; no removals, relaxed matchers, skips, or mock-scope narrowing.
Root-Cause Investigation N/A (informational) No new sub-tasks created this run — nothing to investigate.

Overall: ⚠️ WARN — driven solely by the justified Scope Containment WARN (review-feedback files); Test Quality WARN is informational. No FAIL, and no open merge blockers (TC-6032 and TC-6033, the prior sole blockers, are fixed in this commit).

verify-pr did not modify code, merge, or transition the parent task.


This report was AI-generated by sdlc-workflow/verify-pr v0.13.9.

@mrizzi
mrizzi merged commit e696d00 into RHEcosystemAppEng:TC-5787-verify-pr-fullsend-v037 Sep 3, 2026
5 checks passed
@mrizzi
mrizzi deleted the TC-5812 branch September 3, 2026 15:43
@mrizzi

mrizzi commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator Author

Verification Report for TC-5812 (commit 7ca7e8c)

Check Result Details
Review Feedback PASS All 10 items (7 inline + 3 review bodies) classified in prior runs; all replies posted
Root-Cause Investigation N/A No new sub-tasks created this run
Scope Containment WARN All task files present; 6 additional supporting files outside task spec
Diff Size WARN 751+/14- across 7 files for a 1-file task
Commit Traceability FAIL No commits reference TC-5812 in their messages
Sensitive Patterns PASS No sensitive patterns in 751 added lines
CI Status WARN Cannot verify from sandbox (no gh CLI)
Acceptance Criteria PASS Dual-mode gated on FULLSEND_OUTPUT_DIR; all action types schema-enumerated; trailers present
Test Quality WARN Repetitive: PASS - Test Doc: WARN (3/14 missing docstrings) - Eval Quality: N/A
Test Change Classification ADDITIVE Both test files new to repo; 14 functions added, 0 removed
Verification Commands PASS Both verification commands pass (grep + schema orphan check)

Overall: FAIL

Commit Traceability is FAIL: None of the 8 commits on the PR branch reference TC-5812 in their commit messages. Commits use conventional commit format (feat(verify-pr): / fix(verify-pr):) with descriptive summaries but omit the Jira task ID.

Scope Containment and Diff Size are WARN: The task specifies only SKILL.md, but the PR also modifies 6 supporting files (schema, prefetch scripts, tests). These are plausibly related but exceed the task description scope.

CI Status is WARN: Cannot be verified from the sandbox environment (no gh CLI available).

Test Quality is WARN: 3 of 14 new test functions in test_pre_verify_pr.py lack docstrings (test_related_keys_empty_when_no_relations, test_build_idempotency_bundle_empty, test_transform_with_idempotency_attaches_key).


This comment was AI-generated by sdlc-workflow/verify-pr v0.13.9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant