Repository navigation
fix(gc): protect sibling gateway images (Fixes #12582) - #12608
Conversation
GC scans host-wide Docker images but previously protected only the selected gateway's registry. Read every validated gateway registry before classifying orphans, and stop before deletion when that inventory cannot be safely read. Keep GC in a focused maintenance action module to stay within the existing source dependency budget. Preserve the public portable-runtime guard. Add regressions for sibling images without containers, unsafe state, shared tags, missing roots, dry-run, confirmation, and Docker deletion failures. All 74 focused tests pass. An isolated Docker check preserves both registered images from ports 8080 and 8090 and removes only the genuine orphan. Fixes #12582 Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (7)
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review. 📝 WalkthroughWalkthroughGarbage collection now checks image registrations across gateway registries and rechecks ownership before deletion. Sandbox creation and registry recovery use a non-waiting portable host fence. ChangesGateway-aware image garbage collection
Non-waiting portable host fence
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Bug fix · Severity of issue fixed: Medium Sequence Diagram(s)sequenceDiagram
participant CLI as NemoClaw CLI
participant GC as garbageCollectImagesWithoutPortableAuthority
participant Docker
participant Registries as Gateway registries
participant Locks as Registry locks
CLI->>GC: Invoke image garbage collection
GC->>Docker: List sandbox images
GC->>Registries: Read registered image tags
GC->>Docker: Refresh image list after confirmation
GC->>Registries: Refresh registered image tags
GC->>Locks: Lock existing registry files
GC->>Registries: Recheck image ownership
GC->>Docker: Remove remaining orphan images
Suggested reviewers: Merge Risk: ⚪ Minimal · up to The inspected cleanup path protects registered images during deletion, and no actionable merge-blocking issue remains after normal checks. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 31.25% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 16 functions across 17 files. (2 skipped: 2 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall line coverage in commit f7c138c in the Show a line coverage summary of the most impacted files.
TypeScript / code-coverage/cliThe overall line coverage in commit f7c138c in the Show a line coverage summary of the most impacted files.
Updated |
Re-read every host gateway registry after confirmation and hold registry locks while removing orphaned images. Preserve images registered while the prompt was open and stop if a registry becomes unsafe to read. Cover late registration and unsafe state in the GC tests, and explain host-wide image protection in the command reference. Refs #12582 Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-12608.docs.buildwithfern.com/nemoclaw |
Mock the new lock dependency so the maintenance suite can load with its existing credential mock. The dedicated GC suite still exercises real registry locks. All 76 focused tests pass. Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
Ready for review at
CI, the Advisor and CodeRabbit are green on this head, all commits are verified and no review threads are open. |
rsliter
left a comment
There was a problem hiding this comment.
Requesting changes for one fail-closed deletion-safety gap. The accepted #12582 contract requires malformed gateway inventory to stop before deletion. The current imageTag check accepts empty and whitespace-only strings. Orphan classification then treats those values as protecting no real image, so gc can still pass a registered image to docker rmi. Please reject unusable string values before classification and add empty and whitespace-only regressions that prove dockerRmi is not called. The latest PR commit also conflicts with current main, so refresh the branch and rerun the required checks after the repair.
Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
Follow-up on the fail-closed deletion-safety feedback: commit |
|
PR Review Advisor finished for commit Request review only when Require no Advisor blockers is green. |
|
The requested fail-closed guard is on the current PR head: non-string, empty, and whitespace-only sibling image tags are rejected before classification, with regression cases asserting Docker image removal is not called. The focused maintenance GC suite passes 20/20 on this head. The branch is synced with main and the current hosted checks are green. |
rsliter
left a comment
There was a problem hiding this comment.
Re-reviewed commit f7c138c. The prior blocker is resolved: registry image tags now reject empty, whitespace-only, and non-string values. The accepted #12582 scope is implemented with fail-closed host registry discovery, a second post-confirmation scan, host and per-registry locking around deletion, and coverage for sibling gateways, concurrent registration, malformed state, and genuine orphan removal. The complete diff and all nine security categories are clear. The trusted gate check reports all 77 current checks green, no conflicts, no automated-review blockers, and verified commits.
## Outcome The independent E2E growth check can accept the approved budget for #12767 after this prerequisite policy merges into `main`. ## Reason Independent enforcement reads exceptions from the trusted base. A candidate-only entry cannot authorize its own budget change. ### Related issues Refs #7475. Prerequisite for #12767. ## Changes Add one exception bound to PR #12767 and both budget SHA-256 hashes. Preserve all existing exception entries. Remove the temporary entry after #12767 merges. The approved repair moves restoration validation into one builder within the scanned live source set. It preserves every restoration predicate, including plugin validation, and retains the seven-assertion reduction from #12608. The budget changes from `53c51c4229d387633165771730f8f1a8373f84e9f3303b2144280027a56e3441` to `1653f4c6c0292d2853727c39ddb36aa55361f749df3db32aa3847c3d4a7f30cb`. ## Verification - Existing growth parser and guardrail tests: 65 passed. - Production policy evaluator with the actual budget files: approved transition passes; wrong PR, changed base bytes, changed candidate bytes, and candidate-only policy are rejected. - CLI type check: passed with an 8 GiB Node heap. Default setup exhausted its default heap. - The diff changes only the exception policy and contains no secrets, API keys, or credentials. - Normal commit hooks: passed, including growth guardrails, repository checks, secret scan, and commitlint. - Final publication validation and CLI TypeScript: passed; plugin and JavaScript configuration checks legitimately skipped. All three published commits are GitHub Verified, including `3c4688d37d17f3a8e3aa96a933bb073924430ce2`. The previous candidate passed growth enforcement and CodeRabbit. Its CI failed before tests in the inherited Ubuntu mirrorlist installer. This candidate integrates the landed fix from #12837 and resolves the independent policy-entry conflict while preserving both entries. [CI run 37809262635](https://github.com/NVIDIA/NemoClaw/actions/runs/37809262635) passed for the integrated candidate, including all twelve CLI shards. Independent growth enforcement passed. No live E2E is required for this policy-only diff; all nine specialists concurred. ## Review notes [Rebecca approved this bounded relocation, budget transition, and prerequisite PR](#12767 (comment)). Her maintainer permission and the record were verified. This authorization is not a PR review approval or an Advisor waiver. CodeRabbit completed its exact-head review with no actionable findings. All nine Advisor specialists reported clear, and the blocker gate passed in [Advisor run 37811789238](https://github.com/NVIDIA/NemoClaw/actions/runs/37811789238). The policy PR changes no executable, live test, or budget file. Its parser continues to enforce PR identity and both hashes. The main repair remains local until this dependency lands. Local Docker and CLI exposure are not required for these policy checks. --- Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated internal validation records for an approved exception, including its review conditions, validity requirements, and verification details. No changes to restoration behavior or assertion counts are reflected in these records. * **Documentation** * Recorded that a previously approved assertion reduction remains in effect and noted when the exception is expected to be removed. These updates do not change any user-facing functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
## Outcome Refresh the exception for #12382 so its reviewed runtime assertions can pass the independent growth check after #12608 reduced main's gateway assertions. ## Reason #12608 merged before #12718 and changed the base budget bytes. The exception adopted by #12718 therefore matches neither the new base budget nor #12382's prepared integration. ### Related issues Refs #12382, #12718, #12608. ## Changes Replace only the two budget digests for #12382 and add a scoped explanation. Preserve every other exception and all enforcement code. | Budget | SHA-256 | |---|---| | Base | `53c51c4229d387633165771730f8f1a8373f84e9f3303b2144280027a56e3441` | | Runtime candidate | `8162408384d7a6b97ef0062042c73d4bc9486f54d2dfa6c0c5732e42d1060e2a` | The runtime transition retains native admin approval before survival/rebuild mutations and the plugin-install success check. It adds one unique assertion point and one generated probe condition over the reduced main budget. It adds no live scenario or matrix lane. ## Verification - Canonical `e2eAssertionBudgetGrowthViolations` consumer: accepts the proposed trusted-base transition; rejects a different PR, changed base bytes, changed candidate bytes, and candidate-only policy. - Assertion census: the prepared runtime source fits the proposed budget. - All 58 growth parser tests passed after resolving the shared-policy conflict. Existing census evidence remains valid because the budgets and runtime source did not change. - Signed commit and normal publication hooks passed, including plugin and checked-JavaScript type checks. - Reviewed the complete one-file diff. No secrets, API keys, or credentials. ## Review notes The branch incorporates #12837, which repairs the inherited Ubuntu mirrorlist installation failure. The branch also incorporates #12833 and preserves its #12767 exception. The #12382 budget digests are unchanged. The policy comment now cites the maintainer acceptance recorded below; no exception value or enforcement code changed. Signed commit `71535b2` passed normal commit and publication hooks. Fresh CI and Advisor checks for this comment-only update are pending. [Senthil approved the PR](#12840 (review)), confirming that the recorded acceptance resolves his earlier change request. Maintainer Prekshi Vyas (@prekshivyas) [accepted the exact replacement budget pair](#12840 (comment)), including one additional unique assertion point and one generated probe condition. Independent enforcement is not waived: adoption on main remains required before #12382 can use this exception. The approval records the exact replacement pair and its bounded assertion increase. --- Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated internal CI records with the approved exception’s refreshed verification values. This does not change end-user functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Outcome Renew the exact assertion-budget exception for PR #12769 after main reduced the concurrent-gateway assertion inventory. The exception continues to permit only the approved four-point increase for same-name OpenClaw model-change recreation. ## Reason The original prerequisite #12776 matched the previous whole-file budget hashes. Main's seven-assertion reduction in #12608 changed those hashes, so the exact-match guard correctly rejected the updated candidate. The changed-model regression contract has not expanded. ### Related issues Refs #12667. Prerequisite for #12769. ## Changes Update only PR #12769's base and candidate budget hashes and its explanation. Preserve all other exceptions, all assertion counts, and the exact PR identity and digest checks. Remove this exception after #12769 merges. ## Verification - Existing growth-guardrail parser tests: 58 passed, including exact-transition acceptance and rejection cases. - Independently computed hashes match canonical main `888e95d0ad01db264575181a620923e568c343c7` (budget unchanged from `c518e121d23c616da1b083c44c9eb504fb419ebd`) and PR candidate `a377d75342b1db1c6cbe230300373c14822ecd0a`; all sibling exception entries are unchanged. - `git diff --check` passed. The diff contains no secrets, API keys, or credentials. - Publication validation and CLI, plugin, and JavaScript configuration TypeScript hooks passed during the guarded normal push. --- Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated automated validation records to reflect a revised approved limit while preserving previously accepted adjustments. The records also clarify the conditions associated with the approval. These changes affect validation tracking only; they do not change product functionality or user-facing behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Outcome Record the approved two-assertion exception for Slice 4 PR #12777 in trusted base policy. Once this prerequisite merges, independent enforcement can evaluate that exact budget transition without accepting policy supplied by the feature branch. ## Reason The independent growth check reads exceptions from main. The candidate-only record in #12777 cannot satisfy that check. ### Related issues Refs #12558. Prerequisite for #12777. ## Changes Add one PR-number-bound entry to `ci/e2e-assertion-growth-exceptions.json`. No test budget, enforcement code, workflow, product behavior or other exception changes. The [recorded approval](#12777 (comment)) authorizes exactly two live assertions for native host-local success and sibling isolation. Main #12608 subsequently removed seven unrelated assertions. This entry preserves that reduction and records the mechanically recalculated transition for signed Slice 4 candidate `3f967908b29892c28f57aa4f6457532f4811c6cb`: - Base budget SHA-256: `53c51c4229d387633165771730f8f1a8373f84e9f3303b2144280027a56e3441`. - Candidate budget SHA-256: `8bdce2cee059bb402f1f504e267f99cc39f1f7bf98928ffe0fb1bc95674dfa80`. - Only `inference-routing.test.ts` changes relative to current main: `[36,51,48,89,15]` to `[38,53,50,91,15]`. Direct assertion points increase from 1959 to 1961; unique points from 3203 to 3205. A different PR or budget digest cannot use this entry. Remove it after #12777 merges. ## Verification - `npx vitest run --project integration test/automation/pull-requests/growth-guardrails.test.ts test/automation/pull-requests/growth-guardrail-parsers.test.ts`: 65 tests pass. - Local pure-check rehearsal: current trusted policy reports eight expected growth violations; the proposed exact exception on the trusted side reports zero. No GitHub status was changed. - Compared both budget files and their complete structural difference. Only the two approved inference-routing assertions and corresponding aggregate counters increase. - Normal signed-commit hooks, including growth enforcement and secret scanning, pass. The diff contains no secrets, API keys or credentials. ## Review notes The required upstream APT setup fix from #12837 is integrated through main `c518e121d23c616da1b083c44c9eb504fb419ebd`. The exception conflict preserves every existing main entry and the unchanged #12777 digests. All 65 focused tests and normal signed-commit hooks pass on merge commit `b96598a2037a7e8347f8a8492624ff6c59ecb567`; latest-head CI remains required after publication. This is a workflow-authored transcription and updated disposition of Rebecca Sliter's existing approval, not a new independent review or a passing CI result. Current #12777 growth failures remain failures until trusted policy is available and enforcement passes. This prerequisite requires its own CI and automated review. Repository-local setup and compilation completed. The contributor doctor reports an unavailable local Docker daemon and a CLI resolving to another installation; neither is used by this policy-only validation. No runtime or CLI exposure was changed. --- Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated the automated test-baseline allowance for the approved inference-routing assertions. The allowance applies only when the recorded base and head digests match, and continues to account for the reduction in assertions from the prior baseline. Independent validation continues to use the trusted baseline policy. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Outcome
nemoclaw gcpreserves images registered to another gateway port, including images with no container. With alice on 8080 and bob on 8090, cleanup from either port now removes only the unregistered image. An unreadable or invalid gateway inventory stops cleanup before deletion. GC checks the registries again after confirmation, so an image registered while the prompt was open is preserved.Reason
GC lists Docker images across the host but previously checked only the selected gateway's registry. That made bob's registered image look orphaned when cleanup ran from alice's gateway. Docker could reject the deletion while a container used the image, or delete it when no container remained.
Related issues
Fixes #12582.
Changes
src/lib/actions/maintenance/gc.tsso the existing maintenance entry point stays within its dependency budget. Keep its portable-runtime guard at the public entry point.Verification
Test repair
de387589b7d30240748617190d18692161d15ac1isolates the registry-lock dependency inmaintenance.test.ts. Before this repair, that suite failed during import because messaging hooks requestedgetCredentialfrom its prompt-only mock. After the repair, all 76 tests passed acrosssrc/lib/actions/maintenance-gc.test.ts,src/lib/actions/maintenance.test.ts,src/lib/domain/maintenance/images.test.ts, andsrc/lib/state/gateway-registry.test.ts, usingnode_modules/.bin/vitest run --project cliwith those four paths. The dedicated GC suite still exercises real registry locks. Focused Oxlint, Oxfmt, andgit diff --checkpassed.Normal commit and pre-push hooks passed, including publication validation and CLI typechecking. GitHub reports all three commits as Verified, and the branch, PR, and pull refs match
de387589b7d30240748617190d18692161d15ac1. Hosted CI and Advisor must evaluate the new commit. CodeRabbit has skipped review while this PR is draft. No new live gateway or sandbox E2E run was performed for this test-only repair.Previous published repair:
c1aa5a4b1367238b903f6035090cd62839af5007. Normal signed commit hooks and pre-push publication validation passed, including CLI typechecking. GitHub reports the repair as Verified; the PR, branch, and pull refs match.Repair validation: the GC suite passed all 18 tests. Focused Oxlint and
git diff --checkpassed.npm run docscompleted with zero errors and two warnings. New tests cover registration during confirmation, unsafe sibling state, and registry-lock ownership and release.The following evidence was recorded for the original implementation:
npx vitest run --project cli src/lib/actions/maintenance-gc.test.ts src/lib/actions/maintenance.test.ts src/lib/domain/maintenance/images.test.ts src/lib/state/gateway-registry.test.tspassed all 74 tests.NODE_OPTIONS=--max-old-space-size=8192 npm run test:changed, run with a temporary private home, passed 190 affected tests and 7 growth checks. The first run failed 16 unchanged CLI adapter cases because the workspace could not create a lock in the account home.NODE_OPTIONS=--max-old-space-size=8192 npm run typecheck:cliandNODE_OPTIONS=--max-old-space-size=8192 npm run build:clipassed. The initial setup type check exhausted the default Node heap.npm run test:projects:check, andgit diff --checkpassed.21413c42e01ddaca6bd5e9843d3b2f5bff864e62as Verified.Review notes
The accepted cleanup boundary requires complete validated registry inventory before deletion. An independent review of commit
21413c42e01ddaca6bd5e9843d3b2f5bff864e62and this description found no actionable correctness, security, or writing issues. The follow-up repair refreshes the inventory after confirmation and holds the existing registry locks during deletion. A separate review checked the repair, its public entry point, lock order, refusal behavior, and lock release.The Docker check did not start live OpenShell gateways or sandbox containers. Cross-platform CI and automated PR review must evaluate the repair commit; this PR stays draft.
AI-assisted repair and text: Codex.
Signed-off-by: Deepak Jain deepujain@gmail.com
Summary by CodeRabbit