[HarryMoss]

Summary

This adds a dedicated Kubernetes E2E lane that starts a real OTLP collector and checks traces from a successful sandbox operation. Missing gateway or compute-driver spans fail the lane clearly. It completes the collector-validation work remaining after #5348.

Related Issue

Closes #3767

Changes

• Add a tracing task and CI matrix row using the existing gateway harness.
• Check expected operations, resource attributes, valid IDs and shared gateway-to-driver validation context within a bounded deadline.
• Recheck the saved receipt against collector data so a zero shell status cannot hide a skipped probe.
• Preserve partial logs and reclaim owned resources on success, failure, timeout and handled cancellation.
• Support normal single-file and merged-file KUBECONFIG selection, fix empty-array handling on macOS Bash and update the test guidance.

Release Notes

Adds an explicit collector-backed Kubernetes tracing validation lane.

Testing

• 21 checker/lifecycle tests and affected Python lint/format/type, Markdown and diff checks pass on source tree d4a19edd0b60374f3a6ea355e7a8dad1fedb352c.
• Client-only configuration regression covers single-file and merged-file kubeconfigs without contacting a server.
• Unit and E2E coverage added, including missing receipts, incorrect attribution/context, skipped probes and process cleanup.

Real Linux ARM64 Kubernetes/Collector positive, missing-export and SIGTERM cleanup controls passed on the preceding tree 2047c903b723005e5276884d62d45b76fc516e16, using base-pinned CLI/images from af8182f7b. The final follow-up changes only configuration selection, its regression and CI documentation; those cluster controls were not rerun or relabelled for this revision. Workflow, Helm and Bash checks remain applicable to their unchanged files. Standalone ShellCheck retains the base's one existing SC2034 warning.

Temporary resources were removed. No full workspace CI or backend-retention pass is claimed. Please apply area:build, topic:observability and test:e2e so the new collector lane runs in NVIDIA CI. Maintainer and workflow code-owner approval remain pending.

Checklist

• Follows Conventional Commits.
• Commits are signed off for DCO.
• Test and development guidance updated.

Architecture documentation changes are not applicable to this test-infrastructure contribution.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[HarryMoss]

Drafted with Codex; reviewed and approved by @HarryMoss.

Could a maintainer please apply area:build, topic:observability and test:e2e to this PR, and approve it for NVIDIA CI?

The readiness check currently fails because there is no area label. Branch Checks and Helm Lint are waiting for /ok to test, and the E2E status currently means the label was not applied, so the collector lane has not run. DCO, vouch and issue acceptance have passed.

Thanks.

[krishicks]

Written by Claude Code, reviewed by @krishicks

I had another version of this lane locally while working on #5348 and forgot to push it. It's now up as #5517. It covers the same part of #3767 in a different way:

• Jaeger as the collector, pulled from mirror.gcr.io and pinned by digest. The test queries Jaeger's /api/v3 API, so it doesn't parse collector stdout and doesn't need to run as root.
• A Rust test in e2e/rust, run through the existing with-kube-gateway.sh harness and its cleanup, about 380 lines in total.
• The default gateway service name and the default sandbox image.

I added some checks from this PR: each span needs a non-error status and a service.version, and the gateway CreateSandbox span gets the same resource checks as the driver spans. #5517 doesn't include your unit-tested negative controls or the receipt re-check against captured collector data.

What do you think of #5517? If it covers what you were after, I'd like to merge it and close this one. If you think something from this PR should carry over, tell me what.

[HarryMoss]

Thanks Kris. I've compared #5517 at e207ddd with #5354. The Jaeger query API and existing Rust harness cover the collector-backed validation goal for #3767. Thanks for carrying over the status and resource checks.

A few additional checks may be worth considering:

• Match the probe. Provisioning is matched to the unique sandbox, but the public CreateSandbox lookup currently accepts any matching request and only requires a nonempty gateway name. Tying that request to the expected gateway and probe would strengthen it. The synchronous request/driver-validation join in test(ci): verify Kubernetes trace receipt at a collector #5354 is one option; asynchronous provisioning needn't share the public request's root.
• Missing, wrong and delayed evidence. Small Rust fixtures could cover missing operations, incorrect attribution, broken context and error status. Polling for the complete provision/client span set would also allow for spans arriving separately. Bounding the HTTP waits within the export deadline would keep a stalled query from falling back to the five-minute nextest timeout.
• Confirm the test ran. A check that the named Rust test actually executed would preserve the receipt guarantee. The small Bash 3.2 empty-array guards in test(ci): verify Kubernetes trace receipt at a collector #5354 may also be worth carrying over independently: an isolated reproduction returned zero after an unbound-array error, before the probe ran.
  Saving query/fixture diagnostics before teardown and retaining a focused setup-failure/cancellation control could also help. The Jaeger API and proof of test execution can serve the receipt-checking purpose without retaining the old stdout parser or receipt format.

Those are the main things I noticed. Some may fit #5517 and others a follow-up—happy to go with your judgement on that.