Repository navigation
fix(e2e): select Nemotron 3.5 Super VL for public NVIDIA targets - #13583
Merged
Merged
Conversation
The OpenClaw and Hermes native NVIDIA switch targets and the Linux AMD64 Pi qualification target selected nvidia/nemotron-3-super-120b-a12b. The public NVIDIA endpoint now returns HTTP 404 for that model, so the switch was rejected and Pi onboarding stopped during endpoint validation. Select nvidia/nemotron-3.5-super-vl-120b-a12b in the catalogue entries, which take precedence over the job environment, and derive the switch helper, public inference adapter, and Pi expectation from the product default model. Signed-off-by: San Dang <sdang@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
📒 Files selected for processing: 13
✨ Finishing Touches
Comment |
Contributor
Code coverage overviewLanguages: TypeScript TypeScript / code-coverage/plugin
View coverage of the 10 most impacted files.
TypeScript / code-coverage/cli
View coverage of the 10 most impacted files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. Updated |
Cover the Pi target's catalogue model and managed inference evidence in its mapped fast test, as required by the E2E mock/live parity check. Signed-off-by: San Dang <sdang@nvidia.com>
The public endpoint answers nvidia/nemotron-3.5-super-vl-120b-a12b requests with the response model private/nvidia/nemotron-3.5-super-120b-a12b. Live run 39548034650 observed this for Hermes, OpenClaw, and Pi. Accept only that exact serving name for the Super VL request; every other model still requires an exact response model match. Signed-off-by: San Dang <sdang@nvidia.com>
Pi qualification now runs against nvidia/nemotron-3.5-super-vl-120b-a12b. Update the Pi support reference. Record the response-model evidence for the Super VL serving name, and test that the alias does not apply to other requested models. Signed-off-by: San Dang <sdang@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-13583.docs.buildwithfern.com/nemoclaw |
Contributor
|
PR Review Advisor finished for commit Request review only when Require no Advisor blockers is green. |
hunglp6d
approved these changes
Oct 21, 2026
sandl99
marked this pull request as ready for review
October 21, 2026 10:03
deepujain
added a commit
that referenced
this pull request
Oct 21, 2026
Include the public NVIDIA model selection fix from #13583 so the inference-switch E2E uses Super VL instead of the retired model that returns HTTP 404. Preserve the private IPv6 policy repair and existing live assertions. Signed-off-by: Deepak Jain <deepujain@gmail.com>
prekshivyas
pushed a commit
that referenced
this pull request
Oct 21, 2026
## Outcome Express onboarding can abort with HTTP 403 when a Docker host alias resolves to a private IPv6 address such as `fdc4:f303:9324::254`. New local-inference profiles admit that address range while existing sandboxes keep their recorded permissions until the user reselects the local provider. ## Reason The host Ollama service and authentication proxy were healthy, but the generated provider policy admitted only private IPv4 addresses. A distinct immutable profile identity lets new selections use IPv6 without changing permissions attached to existing sandboxes. ### Related issues Fixes #13544. ## Changes - Add `fc00::/7` for `host.openshell.internal` and `host.docker.internal`. Preserve the selected host, port, paths, methods, executable permissions, and credential reference. Literal private IPv4 endpoints retain their `/32` restriction. - Preserve recorded IPv4-only profiles for verification, resume, rebuild, and cleanup. OpenClaw and Hermes reselection use the existing ownership transaction, including rollback on publication failure. - Keep `TC-INF-19` for a prepared WSL2 Docker Desktop host. It requires IPv6-only ULA resolution, authenticated sandbox chat, and explicit policy denials for public, link-local, and loopback IPv6. Lower-layer tests cover resolution classification and fixture behavior. - Keep the native-Docker WSL lane on its existing full E2E suite, with revision verification. Its Ubuntu Docker installation does not configure IPv6-only host aliases, so it cannot run the Docker Desktop proof. Shell-execution regression tests protect both full-suite success and failure propagation. - Clarify private IPv4 and IPv6 host-alias permissions in the troubleshooting guide. Include main's model-selection fix from #13583 so the E2E catalogue uses Super VL and its reviewed response-model mapping. ## Verification - The new workflow regression ran against `a5133fa` before repair. Both native-Docker cases reproduced the defect: full-suite statuses 0 and 17 became 42 when the script invoked the unsupported IPv6 scenario. The same cases pass after repair and preserve their original statuses. - With Node 24.18.1 and `NEMOCLAW_TEST_TIMEOUT=60000`, this command passed all 82 tests across three suites: ```bash node node_modules/vitest/vitest.mjs run --maxWorkers=1 --project integration test/automation/e2e/platform-vitest-main-workflow.test.ts --project e2e-support test/e2e/support/native-local-ipv6-evidence.test.ts test/e2e/support/onboard-timeout-contract.test.ts ``` - Historical policy reproduction: both host-alias regression cases failed before the original fix because `fdc4:f303:9324::254` was excluded. At `3743603`, nine focused suites passed all 391 tests. Policy and selection code are unchanged by this workflow repair. - Historical E2E on `a5133fa` passed. The [Advisor run on that head](https://github.com/NVIDIA/NemoClaw/actions/runs/39642884716) had no P0/P1 findings, but `Require no Advisor blockers` failed on six unresolved WSL E2E recommendations. Those results do not validate this updated commit. Fresh CI, image checks, automated reviews, and applicable E2E are pending. - WSL2 ARM Docker Desktop qualification has not run. No supported manual PR selector supplies that environment. The native-Docker WSL lane and Ubuntu catalogue runs do not establish that qualification. - On `32f7d712bd10a6ea7a476961acecd8b43a5f3690`, normal commit hooks, pre-push publication validation, and CLI TypeScript checks passed. The diff contains no secrets, API keys, credentials, or unrelated dependency changes. ## Review notes Decision: accept the existing host-alias policy repair. Reason: the reported private IPv6 bridge falls outside the generated profile. Placement: profile generation and identity, existing selection owners, provider guidance, and the existing IPv6 live test. Validation plan: profile and selection tests, workflow and fixture contracts, publication checks, and WSL2 ARM Docker Desktop qualification. Existing permissions change only through explicit reselection. The workflow review identified a valid mismatch between the native-Docker runner and the required IPv6-only network. The repair removes that incompatible invocation and its extra timeout and artifact handling. Every live IPv6 assertion remains intact. This preserves the existing workflow trust boundary and full-suite behavior. A new trusted qualification lane remains separate work; this decision neither clears the Advisor gate nor waives the missing live evidence. Self-review of `32f7d712bd10a6ea7a476961acecd8b43a5f3690` covers the workflow repair, executable regression cases, test guidance, and timeout comment. Independent re-review is pending. The sensitive native-local policy, identity, selection, and rollback implementation is unchanged by this repair. The existing eight added live assertion points retain their recorded [growth approval](#13582 (comment)). No live assertion or growth record changes here. DGX Station validation is not applicable. --- Signed-off-by: Deepak Jain <deepujain@gmail.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **New Features** - Native local connections now support private IPv6 unique-local addresses when resolved through approved host aliases. - IPv4-only and dual-stack profiles are distinguished while preserving existing permissions and connection receipts. - Native local provider switching now correctly handles profile changes and restores the previous connection if persistence fails. - **Bug Fixes** - Public, link-local, loopback, and unsupported literal IPv6 endpoints remain blocked. - Private IPv4 literal endpoints remain restricted to exact host addresses. - **Documentation** - Added troubleshooting guidance for Docker Desktop WSL IPv6 resolution and native-local endpoint restrictions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Deepak Jain <deepujain@gmail.com>
wscurran
added
area: e2e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outcome
The OpenClaw and Hermes native NVIDIA inference-switch E2E targets and the Linux AMD64 Pi qualification target now select the public model
nvidia/nemotron-3.5-super-vl-120b-a12bathttps://integrate.api.nvidia.com/v1. Before this change, they selectednvidia/nemotron-3-super-120b-a12b, which the public endpoint rejects with HTTP 404.Reason
E2E run 39541140084 on
c1dda570failed all three targets:nemoclaw inference setwas rejected because the sandbox invocation probe returned HTTP 404 with the NVCF "Function not found for account" response. The previous selection stayed in place.The product default (
DEFAULT_CLOUD_MODEL) moved to Super VL in #13542; the E2E selectors did not.runCatalogueTargetassigns each catalogue entry's environment over the job environment, so the catalogue values decide the model. Changing a helper fallback alone would not change it.Changes
tools/e2e/target-catalogue.mts: setNEMOCLAW_SWITCH_MODELforopenclaw-inference-switchandhermes-inference-switch, andNEMOCLAW_MODELforpi-agent-qualification-amd64, to Super VL.test/e2e/live/public-nvidia-switch-provider.ts,test/e2e/fixtures/inference-adapter.ts, andtest/e2e/live/pi-agent-qualification.test.ts: useDEFAULT_CLOUD_MODELfor the switch fallback, the public adapter default, and the Pi model expectation."model": "private/nvidia/nemotron-3.5-super-120b-a12b". Every Hermes, OpenClaw, and Pi request in runs 39548034650 and 39551307701 returned this value. The public/v1/modelscatalog has no selectable model with that name, and it lists Super VL as the only Nemotron 3.5 Super model.acceptedSwitchResponseModels()accepts that exact serving name only for the Super VL request. Every other requested model, including the old Super ID, still requires an exact match. The OpenClaw gatewaymodel.runcheck stays exact. In OpenClaw 2026.9.2,agentMeta.modelreports the requested model, and the observed value was Super VL.models.jsonevidence in the mapped Pi fast test; adapter precedence over an ambientNEMOCLAW_MODELwith explicit-override preservation; positive and negative alias cases.docs/reference/pi-support.mdx: name Super VL as the qualification model.Old-model strings remain where they test older-model compatibility, act as mock identities, or belong to targets outside this scope. Text, tool, lifecycle, credential, and policy assertions are otherwise unchanged. The Pi receipt records no model, and Pi image sources did not change, so the receipt stays valid.
Verification
e2e-supportsuites (inference-adapter,inference-switch-workflow-boundary,public-nvidia-switch-provider,openclaw-inference-switch-helpers,hermes-inference-switch-command-shape,pi-agent-qualification-events): passed.npm run test:changedon macOS: changed-file failures were limited to host-only gaps that also fail on the unchanged base (Bash 3mapfile, missing Pythonyaml) and one load timeout that passed alone.npm run docs: passed. Hooks passed, including growth guardrails, E2E mock/live parity, publication validation, and TypeScript checks.05c6defc: all checks pass. PR Review Advisor on05c6defc: all nine specialists clear.targets=openclaw-inference-switch,hermes-inference-switch,pi-agent-qualification-amd64, which also selects the two Anthropic sibling shards):a979ca89: Pi passed. OpenClaw and Hermes switched to Super VL, then failed the response-model check described above.d7adb5fe: OpenClaw, Hermes, and Pi passed with Super VL on the public endpoint. Cleanup passed.05c6defc(final candidate): Hermes and Pi passed in attempt 1. OpenClaw passed in attempt 3: it switched tonvidia-prod/ Super VL athttps://integrate.api.nvidia.com/v1, sandbox inference returned PONG, and the gatewaymodel.runreported Super VL with text PONG. Cleanup passed. The two earlier OpenClaw attempts failed duringinference setfor reasons outside this change. In attempt 1, the provider returned HTTP 500 to the sandbox probe and the previous selection was preserved. In attempt 2, the first probe got no HTTP status. NemoClaw retried successfully, but the command passed the test's 120-second deadline during the gateway restart. Run 39225787037 hit the same deadline on Oct 19 with the old model.Review notes
tools/e2e/target-catalogue.mtsmatches the sensitive patterntools/e2e/**. Self-review covered the complete diff through05c6defc1ce166c9fa0253196ad1b70fccaa58e2. No independent pre-publication review exists. This draft awaits review.Maintainer decision requested: accept the Super VL serving-name alias as response-model evidence. Possible follow-up outside this PR: the 120-second
inference setdeadline in the OpenClaw switch test is shorter than the product's bounded sandbox-probe retry window.Signed-off-by: San Dang sdang@nvidia.com