Skip to content

(fix): Reject conflicting DRA kubelet-plugin healthcheck ports when ComputeDomains are enabled - #3029

Open
jramkissoon wants to merge 2 commits into
NVIDIA:mainfrom
jramkissoon:issue/#2979
Open

jramkissoon wants to merge 2 commits into
NVIDIA:mainfrom
jramkissoon:issue/#2979

Conversation

@jramkissoon

@jramkissoon jramkissoon commented Oct 8, 2026 •

Copy link
Copy Markdown

Description

Fixes #2979

With draDriver.computeDomains.enabled: true the kubelet-plugin DaemonSet runs
two containers, gpus and compute-domains, each binding a gRPC health service
in the shared pod network namespace. Configured on the same port, the second
bind fails with EADDRINUSE and that container crash-loops; which one loses is
a startup race, so the symptom is hard to trace back to the GPUCluster spec.

The DRA driver state now validates the two healthcheck ports before rendering.
The check runs on the resolved ports, so setting only one field to the other
container's default (51516 / 51515) is caught too, and a disabled health
service never collides. A collision is reported through the existing
getManifestObjects error path: the GPUCluster gets a ReconcileFailed
condition naming both fields and the port, and the DaemonSet is left untouched.

Validation of the containers' metrics endpoints (HTTP_ENDPOINT), raised
during review, is deferred to a follow-up PR.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

Unit tests: TestDRADriverHealthcheckPortCollision covers equal explicit
ports, either port set to the other container's default, distinct ports, the
shipped defaults, computeDomains disabled, and one or both health services
disabled. The error cases fail against the pre-fix validator.

Manual testing on a kind cluster with mock NVML (nvml-mock via the run-ai
fake-gpu-operator), so both kubelet-plugin containers start and reach their
health bind:

  • Release operator v26.7.1 with computeDomains.kubeletPlugin.healthcheck.port: 51516:
    both containers render on 51516 and compute-domains crash-loops with
    failed to listen on :51516: listen tcp :51516: bind: address already in use.
  • Operator built from this PR: distinct ports 52000 / 52001 render with the
    probes on them, both containers ready with no restarts across two liveness
    periods, GPUCluster ready; the colliding spec is rejected with
    ReconcileFailed: ... must differ when computeDomains is enabled; both resolve to 51516
    while the DaemonSet and both containers stay unchanged; restoring distinct
    ports returns the GPUCluster to ready.

Test bed and captured evidence (not part of this PR):
https://github.com/jramkissoon/nvidia-gpu-operator/tree/gpu-operator-issue-%232979-rebased-26.7.2/deploy/issue-test-bed

@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@jramkissoon
jramkissoon requested a review from a team as a code owner October 8, 2026 12:31
@jramkissoon jramkissoon changed the title Reject conflicting DRA kubelet-plugin healthcheck ports when ComputeDomains are enabled (fix): Reject conflicting DRA kubelet-plugin healthcheck ports when ComputeDomains are enabled Oct 8, 2026
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Walkthrough

Walkthrough

The DRA driver defines metrics ports for the GPU and compute-domains containers. It checks resolved healthcheck ports against enabled listener ports and returns a wrapped configuration error when positive ports conflict. Manifest generation performs this check before rendering. Tests cover collisions, disabled services, and non-colliding configurations.

Priority: ⚪ Not assessed

Merge Risk: 🟡 Moderate · up to b4275

Configurations with custom listener environment values can still render a DaemonSet whose listeners conflict. Reject those overrides or validate their effective values before merging, and add the missing collision tests.

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

@kvalliyurnatt kvalliyurnatt self-assigned this Oct 13, 2026
Comment thread internal/state/dra_driver.go Resolved
@kvalliyurnatt

Copy link
Copy Markdown
Contributor

@jramkissoon are you still planning to work on this ?

@jramkissoon

Copy link
Copy Markdown
Author

@kvalliyurnatt Yes, still on it.

jramkissoon added a commit to jramkissoon/nvidia-gpu-operator that referenced this pull request Oct 22, 2026
Review on NVIDIA#3029 pointed out that the kubelet-plugin skips its metrics
listener when METRICS_PATH is empty, so a healthcheck on the freed port
must be accepted. Prove that on the cluster:

  - compute-domains-metrics-disabled empties METRICS_PATH in both
    containers and moves their healthchecks onto 8080 / 8081. It must be
    accepted and both containers must stay ready, which also shows the
    user env entry wins over the template's METRICS_PATH=/metrics.
  - gpus-metrics-port-collision puts the gpus healthcheck on 8080 with
    metrics on. It must still be rejected with the DaemonSet untouched.

scripts/metrics-path-proof.sh runs both plus recovery (make
metrics-path-proof). post-fix-images.sh takes TAG so a before and an
after build from a dirty tree get distinct tags and the operator rolls.

Issue NVIDIA#2979

Signed-off-by: jramkissoon <julianramkissoon@icloud.com>
devin-ai-integration[bot]

This comment was marked as resolved.

jramkissoon added a commit to jramkissoon/nvidia-gpu-operator that referenced this pull request Oct 22, 2026
Devin review on NVIDIA#3029: rejecting HTTP_ENDPOINT in kubeletPlugin.env breaks
GPUClusters that moved the metrics listener, a configuration that works on
released operators.

compute-domains-http-endpoint-override moves the gpus metrics endpoint to
:9090. scripts/http-endpoint-check.sh (make http-endpoint-check) deploys it
and reports whether the operator accepts it, with metrics served on 9090
and not on 8080, or rejects it. EXPECT selects the passing outcome.

  v26.7.1:          accepted, GET :9090/metrics 200, :8080 closed
  issue/NVIDIA#2979 head: rejected, "must not set HTTP_ENDPOINT"

Issue NVIDIA#2979

Signed-off-by: jramkissoon <julianramkissoon@icloud.com>
Comment thread internal/state/dra_driver.go Resolved
Comment thread internal/state/dra_driver.go Resolved
Comment thread internal/state/dra_driver_test.go Resolved
jramkissoon added a commit to jramkissoon/nvidia-gpu-operator that referenced this pull request Oct 26, 2026
…t bed

Review comment by kvalliyurnatt on PR NVIDIA#3029, internal/state/dra_driver.go
line 152 (NVIDIA#3029 (comment)):

  "This is only checking for ports, if the ip addresses are different but
  port is same, this would falsely reject a valid configuration."

validateKubeletPluginPorts keeps only the port of each HTTP_ENDPOINT, so two
metrics listeners that share a port but bind different addresses are reported
as a collision although both binds succeed inside the pod.

Two overlays and one check script reproduce the condition on whichever
operator is deployed:

- compute-domains-metrics-distinct-hosts: gpus HTTP_ENDPOINT 127.0.0.1:9090,
  compute-domains HTTP_ENDPOINT 127.0.0.2:9090. The comment's case.
- gpus-metrics-host-vs-healthcheck: gpus HTTP_ENDPOINT 127.0.0.1:51515
  against the compute-domains health service on its default :51515. The
  counter-case: the health services bind the wildcard address, and Linux
  refuses a specific-address bind on a port with a wildcard listener.
- scripts/distinct-hosts-check.sh deploys an overlay and reports accepted,
  rejected or collided. Accepted requires the render, no restarts over two
  liveness periods, the metrics server start line of each container on its
  configured address, and HTTP 200 on every address, dialed from inside the
  gpus container with bash /dev/tcp since port-forward only reaches
  127.0.0.1. Collided means the render went through but a container lost
  its bind and restarted. EXPECT selects the outcome that exits 0.

Results on a kind cluster with mock NVML (proof/distinct-hosts-check-*):

  PR build 98f910e (3a8c81a + test bed), distinct hosts:
    rejected: "the gpus container metrics endpoint (HTTP_ENDPOINT
    "127.0.0.1:9090") and the compute-domains container metrics endpoint
    (HTTP_ENDPOINT "127.0.0.2:9090") must differ; both resolve to 9090".
    DaemonSet left on :8080/:8081.
  Release v26.7.1, distinct hosts:
    accepted: both containers ready with 0 restarts over 65s, GET /metrics
    200 on 127.0.0.1:9090 and on 127.0.0.2:9090, each container logging
    "Starting metrics HTTP server" on its own endpoint.
  PR build, gpus metrics 127.0.0.1:51515:
    rejected: "both resolve to 51515".
  Release v26.7.1, gpus metrics 127.0.0.1:51515:
    collided: compute-domains crash-looped with "failed to listen on :51515:
    listen tcp :51515: bind: address already in use" while gpus held the
    specific address.

This confirms the comment for metrics-against-metrics: a port-only
comparison rejects a configuration the kernel accepts. It also bounds the
fix: a wildcard listener on either side still collides with every address
on that port, so a host-aware validator must keep rejecting the second
overlay. The health services always bind the wildcard address.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
jramkissoon added a commit to jramkissoon/nvidia-gpu-operator that referenced this pull request Oct 26, 2026
…est bed

Review comment by kvalliyurnatt on PR NVIDIA#3029, internal/state/dra_driver.go
line 94 (NVIDIA#3029 (comment)):

  "With METRICS_PORT=9090 followed by HTTP_ENDPOINT=:$(METRICS_PORT),
  Kubernetes supplies the plugin with :9090. The validator instead parses
  the literal expression and rejects it as nonnumeric, breaking previously
  valid configurations. I don't think just checking for nonnumeric is the
  right way to fix this."

The kubelet expands $(NAME) in an env value against variables defined
earlier in the same container. The template renders its own entries,
HEALTHCHECK_PORT among them, before kubeletPlugin.env, and the user's
entries follow in order, so both kinds of reference resolve.

Two overlays reproduce the condition on whichever operator is deployed:

- gpus-http-endpoint-expansion: METRICS_PORT=9090 then
  HTTP_ENDPOINT=:$(METRICS_PORT). The comment's case.
- gpus-http-endpoint-expansion-collision: HTTP_ENDPOINT=:$(HEALTHCHECK_PORT),
  which expands to the gpus health port, 51516. A real collision that only
  an expansion-aware validator can reject for the right reason.

scripts/distinct-hosts-check.sh becomes scripts/endpoint-check.sh, since
it now covers both HTTP_ENDPOINT review comments. It distinguishes the
rendered env value from the address the plugin binds after expansion,
prints the rejection reason, and in the accepted path prints the plugin's
startup flag dump, which shows the value the kubelet handed over. The
distinct-hosts-check target keeps working; env-expansion-check is new.

Results on a kind cluster with mock NVML (proof/endpoint-check-*):

  PR build a7f9ec1, :$(METRICS_PORT):
    rejected: HTTP_ENDPOINT ":$(METRICS_PORT)" must use a numeric port.
  PR build a7f9ec1, :$(HEALTHCHECK_PORT):
    rejected: HTTP_ENDPOINT ":$(HEALTHCHECK_PORT)" must use a numeric port.
    The right outcome for the wrong reason.
  Release v26.7.1, :$(METRICS_PORT):
    accepted: the DaemonSet carries the literal, the plugin's flag dump
    shows httpEndpoint ":9090", the metrics server starts on :9090 and
    GET /metrics answers 200; both containers ready with 0 restarts.
  Release v26.7.1, :$(HEALTHCHECK_PORT):
    collided: the gpus metrics server took 51516 and the health service
    crash-looped with "failed to listen for healthcheck service at :51516:
    bind: address already in use".

This confirms the comment: a configuration the kubelet resolves into a
working one is rejected for its literal text. The collision overlay shows
why the fix should expand references the way the kubelet does rather than
skip what it cannot parse: skipping would let a provable collision through.

The operator image component points at the build of the previous commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
jramkissoon added a commit to jramkissoon/nvidia-gpu-operator that referenced this pull request Oct 26, 2026
Review comment by kvalliyurnatt on PR NVIDIA#3029,
internal/state/dra_driver_test.go line 485
(NVIDIA#3029 (comment)):

  "I think `:http` is valid and go resolves it to `:80`"

The plugin passes HTTP_ENDPOINT straight to net.Listen, which accepts a
service name for the port. The validator parses the port with strconv and
rejects the name as nonnumeric; the test at line 485 asserts that
rejection.

Two overlays reproduce the condition on whichever operator is deployed:

- gpus-http-endpoint-named-port: HTTP_ENDPOINT=:http. The comment's case.
- gpus-http-endpoint-named-port-collision: HTTP_ENDPOINT=:http with the
  gpus healthcheck on 80, so both listeners of one container want the
  port. A real collision that only a validator resolving the name can
  reject for the right reason.

endpoint-check.sh gains the two cases and a dial address separate from
the plugin's: the plugin logs and reports ":http" as written, so the log
and flag checks compare against that, while the in-pod HTTP check dials
80 numerically (bash's /dev/tcp needs a services table to resolve the
name and the distroless image has none). named-port-check is new.

Results on a kind cluster with mock NVML (proof/endpoint-check-*):

  PR build e451f72, :http:
    rejected: HTTP_ENDPOINT ":http" must use a numeric port.
  PR build e451f72, :http with healthcheck 80:
    rejected: HTTP_ENDPOINT ":http" must use a numeric port.
    The right outcome for the wrong reason.
  Release v26.7.1, :http:
    accepted: the DaemonSet carries ":http", the plugin's flag dump shows
    httpEndpoint ":http", the metrics server starts on ":http" and
    GET :80/metrics answers 200; both containers ready with 0 restarts.
  Release v26.7.1, :http with healthcheck 80:
    collided: the gpus metrics server took 80 and the health service
    crash-looped with "failed to listen for healthcheck service at :80:
    bind: address already in use".

This confirms the comment: ":http" is a working listen address on
released operators and the validator rejects it. The collision overlay
shows the fix should resolve service names as net.Listen does, so a name
that lands on a health port is still caught.

The gpus container is privileged and on the pod network, so port 80 is
bindable and nothing on the node competes for it.

The operator image component points at the build of the previous commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

jramkissoon added a commit to jramkissoon/nvidia-gpu-operator that referenced this pull request Oct 26, 2026
…st bed

Devin review on PR NVIDIA#3029, internal/state/dra_driver.go line 159
(NVIDIA#3029 (comment)):

  "When HTTP_ENDPOINT uses $(NODE_NAME) and a healthcheck port,
  metricsAddress skips the entire listener. On nodes whose name resolves
  locally, the health listener collides and the plugin crash-loops."

The template sets NODE_NAME from a field reference ahead of
kubeletPlugin.env, so the kubelet expands $(NODE_NAME) although the
validator cannot know its value. The validator skips any endpoint that
still holds a reference, which also discards a port it does know.

One overlay reproduces the condition on whichever operator is deployed:

- gpus-http-endpoint-node-host: HTTP_ENDPOINT=$(NODE_NAME):51516, the gpus
  health port. A metrics server on 51516 can never coexist with the health
  service on :51516, whatever the host resolves to.

endpoint-check.sh gains the case. The bind-error search now reads the
current log before the previous one, since in CrashLoopBackOff the latest
failure is the current log, and the collided path prints the plugin's
flag dump as evidence of the expansion. node-host-check is new; its
default expectation is rejected, EXPECT=collided for an unfixed build.

Result on a kind cluster with mock NVML (proof/endpoint-check-20261026T070727):

  Build 2f39ddb (all three earlier fixes), $(NODE_NAME):51516:
    collided. The spec is accepted and rendered; the plugin's flag dump
    shows httpEndpoint "issue-2979-worker:51516"; the name resolves to
    the node's Docker address, outside the pod, and the gpus container
    crash-loops with "listen tcp 172.18.0.2:51516: bind: cannot assign
    requested address".

In this cluster the host resolves to a non-local address, so the metrics
bind fails outright; a host resolving locally would lose to the health
service with "address already in use" instead. Either way the spec cannot
run, and the port alone proves it: the fix should keep a known port when
only the host is unresolved and compare it against wildcard listeners.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jramkissoon

Copy link
Copy Markdown
Author

Pushed the fixes for the review: f19b7c5 (host-aware comparison),
3db452c ($(VAR) expansion), 6f1991d (service names), and for the
follow-up findings b53248c (a known port with an unresolved host)
and 0eec9ff (IP canonicalisation, port range). Each thread has the
details. In the referenced fork the test bed's full run passes on the merged build, 11 of 11:
the original issue, the METRICS_PATH and HTTP_ENDPOINT follow-ups, and
the accepted and collision case for every comment:
jramkissoon@faa268171

@kvalliyurnatt

Copy link
Copy Markdown
Contributor

@jramkissoon thanks for addressing the review comments, by trying to solve the metrics endpoint issue I think this has gotten pretty complex quickly due to the edge cases that we have discovered along the way, I am wondering if we can keep this PR down to the original scope of the issue which I think is the first two commits that you have ? we can revisit the metrics endpoint issue in a separate PR. wdyt ?

@jramkissoon

jramkissoon commented Oct 26, 2026 •

Copy link
Copy Markdown
Author

Hello @kvalliyurnatt

I do agree with the scope-creep, as currently this PR contains a larger volume of changes than I assume was initially expected.

Two questions:

  • Should a fresh PR be opened for the this issue containing only the required commits?
  • Should that be rebased from main since version 26.7.2 was released during the work on this issue?

@kvalliyurnatt

Copy link
Copy Markdown
Contributor

Hello @kvalliyurnatt

I do agree with the scope-creep, as currently this PR contains a larger volume of changes that I assume was initially expected.

Two questions:

  • Should a fresh PR be opened for the this issue containing only the required commits?
  • Should that be rebased from main since version 26.7.2 was released during the work on this issue?

I would just make changes to this PR to have only the first two commits, remove the rest. Then squash and sign the first two commits. Also rebase the branch in this PR with main.

devin-ai-integration[bot]

This comment was marked as resolved.

@jramkissoon

Copy link
Copy Markdown
Author

@kvalliyurnatt

Done: the PR is now the single commit 66a73d8, the original validator
and its tests squashed and rebased on main. The metrics-endpoint work is
set aside for a follow-up PR.

Local test bed used for scoped development work:
https://github.com/jramkissoon/nvidia-gpu-operator/tree/gpu-operator-issue-%232979-rebased-26.7.2/deploy/issue-test-bed

Please review and confirm if changes to resolve devin-ai's review are required. If not, then this is complete per scope requirements.

@kvalliyurnatt

Copy link
Copy Markdown
Contributor

@jramkissoon can you use a GPG key to sign your commit.

@kvalliyurnatt kvalliyurnatt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update the PR description as well.

Comment thread internal/state/dra_driver_test.go Resolved
When ComputeDomains are enabled, the kubelet-plugin DaemonSet runs two
containers that each bind a gRPC health service in the shared pod
network namespace. With both configured on one port the second bind
fails with EADDRINUSE and that container crash-loops; which one loses
is a startup race.

Validate the ports in the DRA driver state before rendering. The check
runs on the resolved ports, so setting only one field to the other
container's default is caught too, and a disabled health service is
skipped. The error names both fields and the port and surfaces through
the existing getManifestObjects path as a ReconcileFailed condition,
leaving the DaemonSet untouched.

Fixes NVIDIA#2979

Signed-off-by: jramkissoon <julianramkissoon@icloud.com>
@jramkissoon

Copy link
Copy Markdown
Author

@kvalliyurnatt Ready for re-review

@kvalliyurnatt

Copy link
Copy Markdown
Contributor

/ok to test 32e2a59

@jramkissoon

Copy link
Copy Markdown
Author

@kvalliyurnatt Thanks for the approval

Comment thread internal/state/dra_driver.go Resolved
kubeletPlugin.env is appended after the HEALTHCHECK_PORT entry the
template renders from healthcheck.port, and the last duplicate wins in
the container. A user entry therefore moves the health service behind
the back of the port validation: with ComputeDomains enabled,
gpus.kubeletPlugin.env HEALTHCHECK_PORT=51515 passes the check yet both
containers bind 51515. Even without a collision the override cannot
work, since the probes keep the spec port.

Reject HEALTHCHECK_PORT in kubeletPlugin.env for the gpus container,
and for the compute-domains container when it is rendered, pointing the
user at the healthcheck.port field.

Fixes NVIDIA#2979

(cherry picked from commit 709e7d3)
Signed-off-by: jramkissoon <julianramkissoon@icloud.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reject conflicting healthcheck ports in the DRA kubelet-plugin daemonset

3 participants