Skip to content

fix(scaffold): stop decoding the contents API 404 body in reconcile-repos - #7620

Merged
waynesun09 merged 3 commits into
mainfrom
fix-reconcile-404-body
Sep 23, 2026
Merged

waynesun09 merged 3 commits into
mainfrom
fix-reconcile-404-body

Conversation

@waynesun09

@waynesun09 waynesun09 commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Fixes #7619. Fixes #7622.

What changed

A new helper, fetch_contents_field, reads one field from the contents API:

  • It uses gh's output only when the call succeeds.
  • A 404 counts as absent: it prints nothing and returns 0.
  • Any other failure prints ::warning::Failed to read <path> for <repo> and returns 1. The caller then counts that repository as failed and moves on. It never guesses whether the repository is enrolled.

It replaces --jq .content 2>/dev/null || true in two places:

  • The enroll shim check. A repository with no shim now takes the enroll path again.
  • The unenroll sha lookup. A missing file is "already removed", so no DELETE is sent with an error body as the sha.

Why

On a 404, gh prints the JSON error body on stdout, and || true kept it as the file content. #7584 added a second decode of that value under pipefail, which turned a warning into exit 1. Every e2e run now fails in the repo-maintenance step on the fresh e2e-lock repository. That includes merge-queue runs.

Tests

The gh stubs now behave like real gh:

  • A failed call prints its body raw on stdout.
  • The missing-file cases return a 404 body.
  • Every stub drains --input stdin. Five stubs ignored it, and jq ... | gh api --input - could die of SIGPIPE under pipefail. That made main's unchanged suite pass only 1 run in 4 on GNU/Linux.

New Test 7 covers five cases:

  1. A 404 takes the enroll path.
  2. A 500 warns and counts as failed, with no write.
  3. Unenroll with a 404 is already removed, with no DELETE.
  4. Unenroll with a 500 counts as failed, with no DELETE.
  5. The script exits 1 with the right count.

Main's unchanged script fails these tests. The fixed script passes 24/24, both in the GNU container and on macOS. Each of four mutations is killed: any failure counts as absent, a 404 counts as a failure, either call site reverted.

Scope

reconcile-repos.sh is part of the deprecated per-org installation mode (ADR 0044). This is a maintenance fix: the e2e admin test still exercises that path, and the defect blocks every e2e run.

A second commit applies the same helper to the unenroll default-branch probe, which the review agent flagged. Only a 404 there means "already unenrolled".

Also fixes #7622: e2e tests this PR's scripts

The e2e CLI now stamps the commit under test as the scaffold's upstream ref. It does this only when that commit can be fetched from fullsend-ai/fullsend; otherwise it falls back to main. The stamp goes into a build-time-only variable, upstreamRefOverride, set through -ldflags alone, with no flag and no environment variable. The rendered repo-maintenance.yml then checks out this PR's scripts/, not main's. Release builds and behaviour-test builds are unchanged.

Out of scope

The same --jq … || true pattern is still used for default_branch and tree.sha. Those values fail loudly further on, instead of being misread.

…epos

On a failed call, gh api prints the JSON error body on stdout. The
reconcile script read the shim with `--jq .content 2>/dev/null ||
true`, so a repo with no shim got the 404 body as its "content". The
script then base64-decoded that body and treated the missing shim as
a stale one.

#7584 added a second decode of the same value. That decode stops the
script under pipefail, so it failed every e2e run: the fresh
e2e-lock repo has no shim. The defect predates #7584, which only
made the decode fatal.

Add fetch_contents_field. It trusts gh output only on success. A 404
(status "404" in the body, the same check delete_branch uses) means
the file is absent: empty output, return 0. Any other failure warns
with the repo and path, and the caller counts the repo as failed and
moves on. Use it for the shim check in enroll and for the removal
branch's file SHA in unenroll, where an error body could have been
sent as the DELETE sha.

Tests: every gh stub now behaves like real gh. A failed call prints
its body unfiltered on stdout, --silent prints nothing, and --input
stdin is drained. The stubs that ignored stdin made
`jq ... | gh api --input -` fail at random with SIGPIPE on Linux; 3
of 4 runs of main's suite failed that way in a GNU/Linux container.
New Test 7 covers a 404 taking the enroll path, a 500 failing the
repo with no write, and unenroll with the file missing or unreadable
on the removal branch sending no DELETE.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Handle contents API 404s safely during repository reconciliation

🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes

Grey Divider

AI Description

• Treat contents API 404 responses as absent files instead of encoded content.
• Fail repositories safely on other contents API errors without issuing writes.
• Align GitHub CLI stubs with real output and stdin behavior.
Diagram

graph TD
  R["Reconcile Repo"] --> F["Fetch Contents Field"] --> G["GitHub Contents API"] --> D{"API Result"}
  D -->|Success| V["Use Field Value"]
  D -->|404| A["Treat File Absent"]
  D -->|Other Error| E["Warn and Fail"]
Loading
High-Level Assessment

The centralized helper is the appropriate approach because both call sites require identical classification and must never consume failed-command output as data. Inline checks would duplicate sensitive shell logic, while parsing gh api --include response headers would couple behavior to CLI formatting rather than the stable JSON status envelope.

Files changed (2) +260 / -22

Bug fix (1) +32 / -2
reconcile-repos.shClassify contents API failures before consuming response fields +32/-2

Classify contents API failures before consuming response fields

• Adds 'fetch_contents_field' to return requested fields only after successful API calls, treat 404 responses as absent files, and report other failures. Uses the helper for enrollment content reads and unenrollment SHA reads so error bodies cannot be decoded or submitted as deletion SHAs.

internal/scaffold/fullsend-repo/scripts/reconcile-repos.sh

Tests (1) +228 / -20
reconcile-repos-test.shModel real gh failures and cover contents API status handling +228/-20

Model real gh failures and cover contents API status handling

• Updates GitHub CLI stubs to emit raw error bodies, honor '--silent', and drain '--input' streams to prevent SIGPIPE failures under 'pipefail'. Adds regression coverage for enrollment and unenrollment behavior across 404 and 500 responses, including warnings, write suppression, failure counts, and continued reconciliation.

internal/scaffold/fullsend-repo/scripts/reconcile-repos-test.sh

@qodo-code-review

qodo-code-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Deprecated install impact goes unflagged ✗ Dismissed 📘 Rule violation § Compliance
Description
fetch_contents_field and its two call sites modify repository reconciliation for the deprecated
per-organization installation path, but the PR description does not identify that scope or cite ADR
0044. The omission applies whenever this fix is reviewed or released, so reviewers cannot
distinguish maintenance of the deprecated scaffold from supported installation work.
Code

internal/scaffold/fullsend-repo/scripts/reconcile-repos.sh[R304-307]

+fetch_contents_field() {
+  local repo="$1"
+  local path="$2"
+  local filter="$3"
Relevance

●●● Strong

Explicit compliance rule requires deprecation scope and ADR citation; similar PR-description
guidance was accepted.

PR-#1903

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Compliance rule 2795055 requires every change touching deprecated per-organization installation
content to call out that impact and reference ADR 0044 in the PR description. The changed helper
belongs to the repository-enrollment reconciliation script invoked by the per-organization
maintenance workflow, while ADR 0044 identifies this reconciliation machinery as deprecated; the
supplied PR description contains neither the required deprecation notice nor an ADR 0044 reference.

Rule 2795055: Flag and avoid changes to deprecated per-org installation mode content (ADR 0044)
internal/scaffold/fullsend-repo/scripts/reconcile-repos.sh[2-13]
internal/scaffold/fullsend-repo/scripts/reconcile-repos.sh[297-318]
internal/scaffold/fullsend-repo/.github/workflows/repo-maintenance.yml[67-73]
docs/ADRs/0044-deprecate-per-org-installation-mode.md[47-56]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
This PR modifies reconciliation logic belonging to the deprecated per-organization installation mode without explicitly identifying that scope or referencing ADR 0044 in the PR description.

## Fix Focus Areas
- internal/scaffold/fullsend-repo/scripts/reconcile-repos.sh[297-318]

## Recommended Fix
Update the PR description to state that the fix affects deprecated per-organization installation functionality and include an explicit reference or link to ADR 0044. No production-code change is required.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 79 rules
✅ Cross-repo context — repo relationships
  Explored: repo: fullsend-ai/.fullsend (sha: 7c163cad)
Review mode: ⚖️ Balanced: This changes runtime reconciliation behavior for API failures and deletion/enrollment decisions, with substantial shell-test/mock changes across multiple paths, so it carries genuine correctness and operational risk but is not clearly dense enough to justify redundant extended review.

Grey Divider

Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:56 PM UTC · Completed 3:12 PM UTC

Commit: 1d2c183 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.01

@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.14286% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/e2etest/buildargs.go 75.75% 5 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Sep 23, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Expanded bug fix with 43% test ratio across 7 files remains at moderate risk due to recent regression history in reconcile-repos scripts and absence of a runtime feature flag, balanced by zero protected paths and well-scoped issue alignment.

Previous run

Risk Assessment: moderate (2/5)

Details

Expanded bug fix with 43% test ratio across 7 files remains at moderate risk due to recent regression history in reconcile-repos scripts and absence of a runtime feature flag, balanced by zero protected paths and well-scoped issue alignment.

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Small scoped bug fix with 50% test ratio and no protected paths, but elevated to moderate risk due to high recent churn and regression history in reconcile-repos scripts without a feature flag.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Looks good to me

Previous run

Looks good to me

Previous run (2)

Looks good to me

Previous run (3)

Review

Findings

Low

  • [fail-open] internal/scaffold/fullsend-repo/scripts/reconcile-repos.sh:675 — Residual pre-existing gap, not the reconcile-repos.sh treats the contents API 404 body as file content; e2e fails since #7584 #7619 body-decode bug and not part of this PR's diff. The unenroll default-branch existence probe still uses gh api ... --silent 2>/dev/null and treats any non-zero exit (403/429/5xx/network, not just 404) as "already unenrolled" (SKIPPED), so a transient error can leave the shim in place on a repo marked enabled: false in config.yaml. The new fetch_contents_field helper and Test 7 only cover the enroll .content check and the unenroll ?ref= SHA lookup — this skip-gate is never exercised by the new tests. Not a blocker for this targeted fix.
    Remediation: In a follow-up, classify the default-branch existence GET the same way as fetch_contents_field (404 → already unenrolled; any other failure → warning, FAILED+=1, continue), and add a Test 7-style case for a disabled repo whose default-branch contents call returns 500.

fullsend-ai-review[bot]

This comment was marked as outdated.

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Sep 23, 2026
Unenroll checked for the shim on the default branch with
`gh api ... --silent 2>/dev/null`. Any failure, including a 403,
429 or 5xx, read as "already unenrolled", so a transient error left
the shim in place on a disabled repo.

Use fetch_contents_field there as well. A 404 is still already
unenrolled. Any other failure warns, counts the repo as failed and
moves on. Test 7 gains a disabled repo whose default-branch lookup
returns 500.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:19 PM UTC · Completed 3:34 PM UTC

Commit: 2619dde · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.02

@fullsend-ai-review fullsend-ai-review Bot removed the risk/moderate PR risk: moderate label Sep 23, 2026
@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review September 23, 2026 15:34

Superseded by updated review

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:43 PM UTC · Completed 6:59 PM UTC

Commit: 36524ff · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.38

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Sep 23, 2026
fullsend-ai-review[bot]

This comment was marked as outdated.

The rendered repo-maintenance.yml checks out
internal/scaffold/fullsend-repo/scripts/ from fullsend-ai/fullsend at
the CLI's upstream ref and copies it over the vendored copy. The e2e
CLI is a dev build, so that ref falls back to main. Every PR and
merge-group e2e therefore ran main's scripts: a PR that changes them
was never tested before merge, and a broken main failed every PR's
e2e, including the PR that fixed it.

Add upstreamRefOverride, a build-time-only variable. resolveUpstreamRef
returns it with no tag when it is set. There is no flag or env var, so
release and user builds are unchanged, and resolveAgentsRef ignores it.

BuildCLIBinary stamps it with HEAD of the module root, under three
conditions: the root is the top of a git checkout, HEAD is a full SHA,
and that commit can be fetched from fullsend-ai/fullsend. The last
check does the same fetch the rendered workflow makes, into a
throwaway repo. It holds for a pushed branch, a fork PR head (through
refs/pull) and a merge-group commit. A local commit that was never
pushed stays unstamped and falls back to main, as before.

GITHUB_SHA is not used because on pull_request_target it is the base
commit. BuildModuleBinary, used by the behaviour tests and external
consumers, stays unstamped.

The e2e install then pins fullsend-ai/fullsend references in the
scaffold (the scripts checkout and the mint-token action) to the
commit under test. This is the same ref-with-no-tag shape that
`github setup --fullsend-ref` already produces.

Fixes #7622

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:08 PM UTC · Completed 7:24 PM UTC

Commit: 74edde9 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.53

@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review September 23, 2026 19:24

Superseded by updated review

@ralphbean ralphbean left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in slack, this won't be relevant soon due to the org mode removal, but if it unblocks CI now, that's good. It can be removed with org mode when that happens.

@waynesun09
waynesun09 added this pull request to the merge queue Sep 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026
@waynesun09
waynesun09 added this pull request to the merge queue Sep 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026
@waynesun09
waynesun09 added this pull request to the merge queue Sep 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026
@waynesun09
waynesun09 added this pull request to the merge queue Sep 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026
@waynesun09
waynesun09 added this pull request to the merge queue Sep 23, 2026
Merged via the queue into main with commit 390bd2b Sep 23, 2026
43 of 44 checks passed
@waynesun09
waynesun09 deleted the fix-reconcile-404-body branch September 23, 2026 22:23
@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 10:25 PM UTC · Completed 10:34 PM UTC

Commit: 74edde9 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.55

@fullsend-ai-retro

Copy link
Copy Markdown

PR #7620 (human-authored, waynesun09) fixed issue #7619's contents-API-404 decoding bug and, over a ~7.5h lifecycle, folded in a second fix for sibling issue #7622. The pipeline performed well: the first review pass (run 35877472872) caught a genuine fail-open gap in the unenroll probe, and the author fixed it within ~5 minutes via a follow-up commit — a clear example of review agent value delivered, not just rubber-stamping. The two E2E failures on the first two commits were correctly attributable to the still-open #7622 defect (not flakiness); E2E passed once the human's third commit fixed the ref-pinning issue. Final agent review approved, a human reviewer (ralphbean) approved with a scoping caveat (this code path is slated for removal per ADR 0044), and CI was clean at merge.

Two investigative threads surfaced only corroborating evidence for issues already tracked elsewhere, so no new proposals were filed for either:

  1. Duplicate agent work: sibling issue e2e runs main's scaffold scripts, not the PR's (upstream ref falls back to main) #7622 also triggered a full triage→code→review pipeline producing PR ci(#7622): pin e2e scaffold refs to the commit under test #7624, which reached an APPROVED review but was closed as a duplicate ~1.5h later once waynesun09 bundled the same fix into fix(scaffold): stop decoding the contents API 404 body in reconcile-repos #7620 himself (his stated reason: ci(#7622): pin e2e scaffold refs to the commit under test #7624 lacked the reconcile-repos.sh treats the contents API 404 body as file content; e2e fails since #7584 #7619 dependency and couldn't pass e2e on its own). The retro that already ran on ci(#7622): pin e2e scaffold refs to the commit under test #7624 (run 35908537065) reviewed this exact case, confirmed triage's dispatch was correct at the time (no fix(scaffold): stop decoding the contents API 404 body in reconcile-repos #7620 coverage of e2e runs main's scaffold scripts, not the PR's (upstream ref falls back to main) #7622 yet existed when ci(#7622): pin e2e scaffold refs to the commit under test #7624 was dispatched), quantified the ~$29 in discarded agent spend, and deliberately declined to file a new proposal — deferring to the existing human/agent-coordination cluster (Triage agent: check related issues for active PRs before labeling ready-to-code #2000, Review agent should detect existing open PRs for the same linked issue #1360, Review agent should check for duplicate/superseding PRs before approving #1313, Code agent dispatch should warn when issue already has open agent PRs #6852, Code agent should act on triage-agent coordination recommendations about related open PRs #3897, consolidated under Consolidate 22+ overlapping "skip retro/agent dispatch" issues into a single design issue #5817). This PR's timeline adds no new mechanism beyond what that retro already captured.
  2. Review-comment 422 fallback: the fail-open finding was demoted from an inline comment to a file-level comment (the flagged line was outside the diff hunk), and that file-level POST itself hit an HTTP 422, falling back to embedding the finding as prose in the review body. This reproduces the exact three-tier fallback-degradation gap already tracked in post-review: file-level comment fallback itself fails with 422, findings silently degrade to prose-only #7346 (filed from PR docs(adr): serialize agent runs and coalesce subsequent events #6909), including the specific diagnostic gap post-review: file-level comment fallback itself fails with 422, findings silently degrade to prose-only #7346 asks to fix: this run's logs also lack the "API error detail" line needed to see GitHub's actual validation message. This run is additional evidence the fix in post-review: file-level comment fallback itself fails with 422, findings silently degrade to prose-only #7346 is still outstanding and worth prioritizing, but it doesn't warrant a separate issue.

Overall this was a well-executed workflow with no new systemic gaps identified specific to this PR.

This branch was previously deployed

1 inactive deployment
dev — 74edde9e Deployed Sep 23, 2026 by waynesun09 via behaviour #13732
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge risk/moderate PR risk: moderate

Projects

None yet

2 participants