Skip to content

fix(cli): remove the OpenShell first-poll settle wait - #8500

Closed
waynesun09 wants to merge 2 commits into
mainfrom
fullsend-3809-settle-removal
Closed

waynesun09 wants to merge 2 commits into
mainfrom
fullsend-3809-settle-removal

Conversation

@waynesun09

@waynesun09 waynesun09 commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

Follow-up to the OpenShell 0.1.4 bump (#8299).

  • Remove the first-poll settle wait. This deletes openShellFirstPollSettle, internal/cli/openshell_settle.go and its test, and the wait before the GitHub pre-flight check in run.go.
  • Bump the GitLab runner VM fallback pins. hack/gitlab-runner-vm/{create-gcp-vm,create-openshift-vm,setup}.sh still defaulted to 0.1.2 when the pin file is absent; they now default to 0.1.4.
  • Correct the runner-VM docs. The README and setup.sh said a stale VM pin "cannot stick". In fact executor/gateway.sh (run from prepare.sh) fails a job whose image reports a different OpenShell version than the host pin. They now say that, and that setup.sh must be re-run after a bump.

Testing

go build ./..., go vet and go test ./internal/cli/ pass. pre-commit passes (gofmt, go vet, shellcheck, gitlint).

I ran this branch as a real local fullsend run on an OpenShell v0.1.4 gateway (podman), with --forge github and the dummy runtime. It ran four times with no wait, and the pre-flight passed on the first attempt each time:

## run 1  ✓ Sandbox created (15.5s)  ✓ GitHub API reachable from sandbox (2.8s)  Validation: passed
## run 2  ✓ Sandbox created (5.7s)   ✓ GitHub API reachable from sandbox (0.8s)  Validation: passed
## run 3  ✓ Sandbox created (3.0s)   ✓ GitHub API reachable from sandbox (0.6s)  Validation: passed
## run 4  ✓ Sandbox created (3.1s)   ✓ GitHub API reachable from sandbox (0.7s)  Validation: passed

No run printed terminated, an EOF or a retry.

The pre-flight finishes before the supervisor's first poll at about 10 s, so I also ran back-to-back requests from inside the sandbox across that window, starting as soon as .env existed:

Ready seen: fs-sch-b7de4951b29e
.env present at +2s
requests=84 failures=0 over 25s        # gh api /rate_limit, back to back
loop covered +2s..+27s after Ready

Stop/start on the #8299 head: the first gh api calls straight after Ready (no wait) all exited 0.

@waynesun09
waynesun09 requested a review from a team October 16, 2026 17:10
@waynesun09

Copy link
Copy Markdown
Member Author

/fs-fix-stop

@github-actions

Copy link
Copy Markdown

Fix agent disabled for this PR. Remove the fullsend-no-fix label or use /fs-fix to re-engage.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Remove obsolete OpenShell settle wait and align runner VM pins

🐞 Bug fix ⚙️ Configuration changes 📝 Documentation 🕐 10-20 Minutes

Grey Divider

AI Description

• Remove the 12-second settle wait so GitHub pre-flight starts without an obsolete workaround.
• Align GitLab runner VM fallback versions with the OpenShell 0.1.4 pin.
• Clarify that runner VMs need setup rerun after a pin bump to accept new job images.
Diagram

graph TD
  Pin["OpenShell pin"] --> Setup["VM setup"] --> Guard["Version guard"]
  Ready["Sandbox ready"] --> Check["GitHub preflight"] --> Agent["Agent run"]
Loading
High-Level Assessment

Removing the workaround is appropriate now that pinned OpenShell 0.1.4 includes the upstream first-poll fix. Keeping a shortened or conditional wait would retain complexity for the older release rather than addressing the pinned version.

Files changed (5) +7 / -17

Bug fix (1) +1 / -11
run.goStart GitHub pre-flight without first-poll delay +1/-11

Start GitHub pre-flight without first-poll delay

• Removes the sandbox-ready timestamp and call to the deleted settle-wait helper. The GitHub connectivity check now follows the preceding setup steps without an extra wait.

internal/cli/run.go

Documentation (1) +1 / -1
README.mdCorrect runner VM version-pin guidance +1/-1

Correct runner VM version-pin guidance

• Explains that a job image requiring a different OpenShell version can fail against the host pin, rather than automatically upgrading it. Directs operators to rerun setup.sh after a pin bump before using newly built runner images.

hack/gitlab-runner-vm/README.md

Other (3) +5 / -5
create-gcp-vm.shSet GCP VM fallback to OpenShell 0.1.4 +1/-1

Set GCP VM fallback to OpenShell 0.1.4

• Changes the default used when the shared OpenShell pin file is absent from 0.1.2 to 0.1.4.

hack/gitlab-runner-vm/create-gcp-vm.sh

create-openshift-vm.shSet OpenShift VM fallback to OpenShell 0.1.4 +1/-1

Set OpenShift VM fallback to OpenShell 0.1.4

• Changes the default used when the shared OpenShell pin file is absent from 0.1.2 to 0.1.4.

hack/gitlab-runner-vm/create-openshift-vm.sh

setup.shAlign setup fallback and version guidance +3/-3

Align setup fallback and version guidance

• Defaults to OpenShell 0.1.4 when the pin file is absent. Corrects the comment about per-job version handling and instructs operators to rerun setup after a bump.

hack/gitlab-runner-vm/setup.sh

@qodo-code-review

qodo-code-review Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Run docs omit the removed startup wait ✗ Dismissed
Description
The runAgent path removes the OpenShell first-poll wait before the GitHub pre-flight check without
updating the documented fullsend run behavior. Every run documentation location listed for this
command group remains unchanged, so users cannot see that startup no longer includes the wait or its
former status message.
Code

internal/cli/run.go[R2420-2423]

+	// 9b. Pre-flight GitHub API connectivity check.
	// Validates that the sandbox can reach api.github.com through the proxy
	// before starting the agent. Without this, agents that depend on gh CLI
	// burn their entire timeout on doomed API calls. See #2143.
Relevance

●●● Strong

Recent CLI changes consistently accepted requests to update user-facing documentation when behavior
or help text changes.

PR-#5763
PR-#8287

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The changed runAgent flow removes the wait immediately before the pre-flight check, changing
fullsend run startup behavior. The documentation guide requires the run command's CLI reference
and guides to be reviewed and updated when its behavior changes.

Rule 2806787: Update CLI-related docs according to docs/contributing/documentation.md when changing CLI surface
internal/cli/run.go[2420-2423]
docs/contributing/documentation.md[97-105]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The `fullsend run` startup sequence no longer waits for the OpenShell first settings poll, but the required run documentation touchpoints were not updated.

## Fix Focus Areas
- docs/cli/run.md[1-30]
- docs/guides/user/running-agents-locally.md[47-57]
- docs/guides/user/building-custom-agents.md[1-20]
- docs/guides/dev/cli-internals.md[1-20]

## Recommended Fix
Review the listed `fullsend run` documentation locations and add or update concise wording where startup sequencing, pre-flight behavior, or user-visible status output is described. Ensure the documentation no longer implies that the removed OpenShell first-poll settle wait occurs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Updated runner images fail on stale VMs ✗ Dismissed
Description
The new README instruction says rerunning setup.sh after a version bump refreshes the executor
pin, but the script only copies the pin file already present on the VM. If that VM still has its
original provisioning files, the executor retains the old pin and rejects jobs built with the new
OpenShell version.
Code

hack/gitlab-runner-vm/README.md[315]

+| `OPENSHELL_VERSION` | no | from `.github/scripts/openshell-version.sh` | OpenShell version used at **VM provision** (Renovate-tracked). Per-job `prepare.sh` reads the job image's `openshell --version` and installs it only when it equals this host pin; a job image on any other version fails the job ("this host only trusts the Renovate-pinned …"). After an OpenShell bump, re-run `setup.sh` (it refreshes `~/gitlab-runner-executor/.github/scripts/openshell-version.sh`) before the VM runs runner images built from the bump. |
Relevance

●●● Strong

Accepted precedent requires rejecting stale VM OpenShell pins and documenting trusted-version
behavior.

PR-#7244
PR-#7253

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Provisioning stages the repository pin onto the VM; setup subsequently sources that VM-local file
and copies it into the executor without fetching an update. The executor rejects a job image
reporting a version different from its trusted pin.

hack/gitlab-runner-vm/create-gcp-vm.sh[818-835]
hack/gitlab-runner-vm/setup.sh[48-61]
hack/gitlab-runner-vm/setup.sh[818-830]
hack/gitlab-runner-vm/executor/gateway.sh[425-445]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Rerunning `setup.sh` alone does not obtain a bumped OpenShell pin. On an existing VM, the documented procedure leaves the executor trusting the old version and newer job images are rejected.

## Fix Focus Areas
- hack/gitlab-runner-vm/README.md[315-315]
- hack/gitlab-runner-vm/setup.sh[39-42]

## Recommended Fix
Instruct operators to transfer the updated pin file to the VM before rerunning `setup.sh`, or describe the supported reprovisioning procedure. Make the setup script comment consistent with that prerequisite.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 89 rules
✅ Skills: 5 invoked
  writing-user-docs
  writing-adrs
  check-patch-coverage
  cutting-releases
  adr-corner
✅ Cross-repo context — repo relationships
Review mode: Auto: ⚖️ Balanced: Runtime networking behavior and runner version pins carry meaningful operational risk.

Grey Divider

Tip of the day
💡 Did you know, you can pick a display mode instead of tweaking every Display option by hand

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread internal/cli/run.go Resolved
Comment thread hack/gitlab-runner-vm/README.md Resolved
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:12 PM UTC · Completed 5:21 PM UTC

Commit: 73666c2 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.38

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Oct 16, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Moderate risk change removing a polling settle wait across 7 files with clean metadata signals, balanced by elevated churn and change coupling in CLI and runner scripts.

Previous run

Risk Assessment: moderate (2/5)

Details

Moderate risk change removing a polling settle wait across 7 files with clean metadata signals, balanced by elevated churn and change coupling in CLI and runner scripts.

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Review

Findings

Low

  • [pattern-inconsistency] internal/cli/run.go:2420 — The PR relabels step 9b-2 as // 9b. Pre-flight GitHub API connectivity check., which duplicates the earlier // 9b. Pre-agent security scan (sandbox-internal, Path B). at line 2392 in the same function. Before this PR the labels were 9b, 9b-1 and 9b-2, so there was no collision.
    Remediation: Use a distinct label such as 9b-bis (matching the existing 9e-bis convention).
Previous run

Review

Findings

Low

  • [cancellation-regression] internal/cli/run.go:2427 — Removing waitForOpenShellFirstPoll also removes the only ctx.Err() checkpoint between sandbox creation and the GitHub connectivity pre-flight. A run cancelled during bootstrap now goes straight into checkSandboxGitHubConnectivity, which is not context-aware (fixed sandbox.Exec timeouts and time.Sleep backoff). Everything earlier in that stretch (uploads, security scan) is also ctx-unaware, and the check takes only a few seconds on a healthy sandbox, so the impact is small.
    Remediation (optional): add an if err := ctx.Err(); err != nil { ... } guard before the pre-flight, with a small test.
  • [intent-tier-mismatch] PR title — fix is questionable under COMMITS.md. Removing an obsolete startup workaround fixes no defect; it removes a latency cost, which fits perf or refactor.
    Remediation: consider perf(cli): remove the OpenShell first-poll settle wait.

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Oct 16, 2026
@codecov

codecov Bot commented Oct 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

OpenShell 0.1.3 and later include NVIDIA/OpenShell#3819, so the supervisor's
first settings poll no longer reloads policy and terminates connections in
flight (NVIDIA/OpenShell#3809). With the pin at 0.1.4 (#8299), the 12 s wait
before the GitHub pre-flight check is no longer needed. Local runs still on
OpenShell 0.1.2 should upgrade to the pinned version.

Signed-off-by: Wayne Sun <gsun@redhat.com>
Assisted-by: Claude
The Renovate bump updates only .github/scripts/openshell-version.sh, so the
fallback defaults in the VM provisioning scripts still said 0.1.2. Keep them in
step with the pin file.

The README and setup.sh also said a stale VM pin cannot stick, but
executor/gateway.sh (called from prepare.sh) rejects a job image whose OpenShell
version differs from the host pin. Say that, and that a VM must be
re-provisioned with create-*-vm.sh --resume from an updated checkout after a
bump.

Assisted-by: Claude (fix), Claude (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🤖 Review · ❌ Terminated · Started 11:59 PM UTC · Ended 12:07 AM UTC

Commit: a2197f6 · View workflow run →

@waynesun09

Copy link
Copy Markdown
Member Author

Validation on OpenShell 0.1.4 (head a2197f696)

This is a real local fullsend run on an OpenShell v0.1.4 gateway (podman), with --forge github and the dummy runtime, using a harness whose schema is cached in a git-ignored .fullsend-cache/.

Two runs, with no settle wait. The pre-flight passed on the first attempt each time:

## run 1  ✓ Sandbox created (10.5s)  ✓ GitHub API reachable from sandbox (1.6s)  Validation: passed
## run 2  ✓ Sandbox created (3.2s)   ✓ GitHub API reachable from sandbox (0.5s)  Validation: passed

No run printed terminated, an EOF or a retry. Earlier, the pre-rebase head had four more runs, all clean (see the PR description).

The supervisor's first settings poll is the moment NVIDIA/OpenShell#3809 used to cut connections. To cover it, I ran back-to-back requests from inside the sandbox, starting as soon as .env existed:

Ready seen: fs-sch-24f973ff7bad
.env present at +3s
requests=88 failures=0 over 25s        # gh api /rate_limit, back to back
loop covered +3s..+28s after Ready
  ✓ GitHub API reachable from sandbox (0.5s)
    Validation: passed
# in sandbox:
FULLSEND_OUTPUT_SCHEMA=/sandbox/workspace/.fullsend/output-schema.json
PASS: output validated against schema

Comment thread internal/cli/run.go Resolved
@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review October 17, 2026 00:07

Superseded by updated review

@fullsend-ai-review

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 11:59 PM UTC · Completed 12:07 AM UTC

Commit: a2197f6 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.74

@waynesun09
waynesun09 enabled auto-merge October 17, 2026 00:18
@waynesun09

Copy link
Copy Markdown
Member Author

@fullsend-ai/core this needs a second-maintainer review: the author cannot approve their own PR. CI is green, all threads are resolved, the 0.1.4 validation is above, and auto-merge is armed. It removes the #3809 settle wait now that the pin is 0.1.4 (#8299).

@waynesun09

Copy link
Copy Markdown
Member Author

Closing in favour of #8531: the same change will come as a bot-authored PR, so it can be approved for v0.48.1.

@waynesun09 waynesun09 closed this Oct 17, 2026
auto-merge was automatically disabled October 17, 2026 13:44

Pull request was closed

@waynesun09
waynesun09 deleted the fullsend-3809-settle-removal branch October 17, 2026 18:17

This branch was previously deployed

1 inactive deployment
dev — a2197f69 Deployed Oct 16, 2026 by waynesun09 via playback #16912
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fullsend-no-fix Skip bot-triggered fix agent runs ready-for-merge All reviewers approved — ready to merge risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant