Skip to content

Render daemonsets.podSecurityContext on GPUCluster DaemonSets - #3222

Open
dundysm wants to merge 1 commit into
NVIDIA:mainfrom
dundysm:fix/gpucluster-pod-security-context
Open

dundysm wants to merge 1 commit into
NVIDIA:mainfrom
dundysm:fix/gpucluster-pod-security-context

Conversation

@dundysm

@dundysm dundysm commented Nov 11, 2026 •

Copy link
Copy Markdown

Issue

Fixes #3203

Description

GPUCluster accepts spec.daemonsets.podSecurityContext, but the DRA driver, DCGM, DCGM exporter, and validator DaemonSets never copied it onto the pod. ClusterPolicy already applies that field through applyCommonDaemonsetConfig.

These four DaemonSets now render the context the same way the driver operand does. Container securityContext is unchanged, including privileged: true and explicit runAsUser values.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets); no API or generated-asset changes
  • Go mod artifacts in-sync (make validate-modules); no dependency changes
  • Test cases are added for new code paths

Testing

  • go build ./...
  • go test ./internal/state/ ./internal/render/
  • go test ./controllers/ -run 'TestClusterPolicyDaemonsetsPodSecurityContext|TestApplyCommonDaemonsetConfig|TestTransformDCGM$|TestTransformDCGMExporter|TestTransformValidator$'
  • go vet ./internal/state/ ./controllers/ ./internal/render/
  • gofmt -l on the changed Go files is clean

TestGPUClusterDaemonsetsPodSecurityContext renders each GPUCluster DaemonSet with and without the context. TestClusterPolicyDaemonsetsPodSecurityContext is the control on the ClusterPolicy DCGM, DCGM exporter, and validator assets. Not applied to a live GPU cluster.


Devin Review

GPUCluster accepts spec.daemonsets.podSecurityContext but does not copy it
onto the DRA driver, DCGM, DCGM exporter, or validator pod specs. Render the
context the same way the driver operand does, and leave container privileged
and runAsUser settings in place.

Fixes NVIDIA#3203

Signed-off-by: Dundy Pasupuleti <dundysm@gmail.com>
@dundysm
dundysm requested a review from a team as a code owner November 11, 2026 05:25
@dundysm

dundysm commented Nov 11, 2026

Copy link
Copy Markdown
Author

@rahulait ready for review whenever you get a chance; dra driver, dcgm, dcgm exporter and validator now pick up daemonsets.podSecurityContext, with clusterpolicy render tests as the control.

@copy-pr-bot

copy-pr-bot Bot commented Nov 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

The four GPUCluster DaemonSet templates now conditionally render spec.daemonsets.podSecurityContext as the pod-level security context. Rendering tests check configured and unset values and verify container security contexts. Controller tests also check pod security context handling through preProcessDaemonSet, including behavior when the policy is unset.

Merge Risk: Moderate · up to 76ae4

A configured non-root UID can prevent the DRA GPU plugin from starting on affected nodes. Resolve or explicitly accept that startup risk before merging.

Security architecture risk: Low · up to 76ae4

The change honors existing configuration without adding a new actor or removing container-level security settings. Its scope includes privileged GPU components, so previously configured pod defaults may affect their execution during rollout. Deployment permissions and live admission behavior remain unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The configurable scope is the common GPUCluster operand set, not an individual tenant pod. A subject permitted to modify GPUCluster can now affect pod security defaults across these four DaemonSets and their selected nodes. The API declares GPUCluster cluster-scoped and singleton-named; generated enforcement and deployed writer permissions were not verified.

Trust Boundaries and Controls

  • observed — The runtime configuration path remains the GPUCluster resource read by the reconciler. The routed Test functions and security-context helpers reside in test files, not newly exposed production entrypoints. The supplied value uses the existing Kubernetes PodSecurityContext type.

Resilience and Maintainability Implications

  • observed — The existing application path attaches controller ownership before creating or updating resources. Failed state synchronization prevents aggregate readiness and causes reconciliation to retry, rather than treating a partially applied operand set as ready.

Hardening Proposals

  • proposed — Before upgrading an installation with an existing pod security context, validate its compatibility with all four operands, especially runAsNonRoot settings alongside required UID 0 containers. Exercise configuration removal during rollout to confirm the deployed environment converges as expected.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Additional comments: 1

internal/state/gpucluster_render_test.go: 1

172-184: Trivial · Maintainability & Code Quality · Low value

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

The required map omits the dra-validator state.

The dra-validator state is rendered but has no container expectations. For it, the test checks only the pod-level context and that container contexts are unchanged. That is acceptable because the validator template sets no container securityContext. A later change that adds one would not be caught.

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between b7d2c34 and 76ae442.

Files selected for processing: 6

  • controllers/daemonset_pod_security_context_test.go
  • internal/state/gpucluster_render_test.go
  • manifests/state-dcgm-exporter/0700_daemonset.yaml
  • manifests/state-dcgm/0500_daemonset.yaml
  • manifests/state-dra-driver/0500_daemonset.yaml
  • manifests/state-dra-validation/0500_daemonset.yaml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment on lines +85 to +88
{{- if .Daemonsets.PodSecurityContext }}
securityContext:
{{- .Daemonsets.PodSecurityContext | yaml | nindent 8 }}
{{- end }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Stability & Availability · Heavy lift

Supported by static analysis

Script executed:

git diff b7d2c34dc112d879b5fee84e694d84a601c02041 76ae44278b4baecb6605bd26c6e7bc0abf47b85b -- manifests/state-dra-driver/0500_daemonset.yaml
cat -n manifests/state-dra-driver/0500_daemonset.yaml
rg -n -F -e 'compute-domains' -e 'mount --bind' -e 'podSecurityContext' -- manifests/state-dra-driver internal | sed -n '1,220p'

Repository: NVIDIA/gpu-operator

Length of output: 19513


Script executed:

rg -n -F -e 'DRA_DRIVER_IMAGE' -e 'MASK_NVIDIA_DRIVER_PARAMS' -e 'PodSecurityContext' -- internal/state api/nvidia/v1alpha1 deployments/gpu-operator | sed -n '1,200p'
cat -n internal/state/gpucluster_render_test.go | sed -n '135,220p'
rg -n -F -e 'dra-driver-nvidia-gpu' -e 'DRA_DRIVER_IMAGE' -- deployments/gpu-operator/values.yaml bundle/manifests/gpu-operator-certified.clusterserviceversion.yaml .github | sed -n '1,100p'

Repository: NVIDIA/gpu-operator

Length of output: 5891


Web query:

Kubernetes privileged container non root UID all capabilities CAP_DAC_OVERRIDE CAP_SYS_ADMIN Kubernetes security context

Result:

Sources consulted:
- Linux kernel security constraints for Pods and containers | Kubernetes — https://kubernetes.io/docs/concepts/security/linux-kernel-security-constraints/
- Pod securityContext Internals for the CKS Exam: Linux Capability Sets, Runtime Default Capabilities, Why Added Capabilities Vanish for Non-Root Users, allowPrivilegeEscalation & no_new_privs, privileged Containers, fsGroup, fsGroupChangePolicy, supplementalGroupsPolicy, Sysctls, procMoun…[truncated]
- Configure a Security Context for a Pod or Container | Kubernetes — https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
- lib/IO/K8s/Api/Core/V1/SecurityContext.pm - metacpan.org — https://web.do.metacpan.org/release/GETTY/IO-K8s-1.100/source/lib/IO/K8s/Api/Core/V1/SecurityContext.pm
- SecurityContext class - io.k8s.api.core.v1 library - Dart API — https://pub.dev/documentation/kubernetes/latest/io.k8s.api.core.v1/SecurityContext-class.html

Web query:

github kubernetes-sigs dra-driver-nvidia-gpu v0.5.1 gpu-kubelet-plugin non root kubelet registration socket permissions

Result:

Sources consulted:
- Releases · kubernetes-sigs/dra-driver-nvidia-gpu · GitHub — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/releases
- kubernetes-sigs/dra-driver-nvidia-gpu v0.5.1 on GitHub — https://newreleases.io/project/github/kubernetes-sigs/dra-driver-nvidia-gpu/release/v0.5.1
- cmd/gpu-kubelet-plugin/driver.go — https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/8effb048f94b3f18338a6f93527352cda82ee385/cmd/gpu-kubelet-plugin/driver.go
- cmd/gpu-kubelet-plugin/driver.go — https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/main/cmd/gpu-kubelet-plugin/driver.go
- cmd/gpu-kubelet-plugin/main.go — https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/main/cmd/gpu-kubelet-plugin/main.go

Web query:

containerd privileged non root exec ambient capabilities CAP_SYS_ADMIN bash kubernetes runc UID 1000 privileged mount

Result:

Sources consulted:
- Chapter 7: Security Boundaries - Containerd From The Bottom Up — https://thecontainerdbook.com/chapters/part-2/07-security
- Kubernetes Linux Capabilities [Add, Drop and Examples] — https://www.golinuxcloud.com/kubernetes-securitycontext-capabilities/
- Docker and Container Capabilities | Luke Hebert — https://lukehebert.com/post/docker-and-container-capabilities/
- libcontainer/SPEC.md at main · opencontainers/runc — https://github.com/opencontainers/runc/blob/main/libcontainer/SPEC.md
- Capabilities and Privilege Management | open-beagle/runc | DeepWiki — https://deepwiki.com/open-beagle/runc/6.3-capabilities-and-privilege-management

Web query:

site:github.com/kubernetes-sigs/dra-driver-nvidia-gpu "runAsUser" "privileged" "non-root" kubelet plugin

Result:

Sources consulted:
- site/content/docs/reference/helm-values.md — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/main/site/content/docs/reference/helm-values.md
- deployments/helm/dra-driver-nvidia-gpu/values.yaml — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/main/deployments/helm/dra-driver-nvidia-gpu/values.yaml
- deployments/helm/dra-driver-nvidia-gpu/values.yaml — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/v0.4.1/deployments/helm/dra-driver-nvidia-gpu/values.yaml
- 616fe03 Add SCC to service accounts on OpenShift — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/commit/616fe03385145ca5dfeb8db5482d32e19b0e0b4e
- Add MPS config option to configure nvidia-cuda-mps-control to run in multi user mode — GitHub pull request 1153 in kubernetes-sigs/dra-driver-nvidia-gpu (link omitted to avoid creating a cross-reference)

Script executed:

cat -n deployments/gpu-operator/values.yaml | sed -n '605,640p'
cat -n internal/state/dra_driver.go | sed -n '130,195p'
rg -n -F -e 'KubeletPlugin' -e 'DaemonsetsSpec' -- api/nvidia/v1 api/nvidia/v1alpha1 | sed -n '1,140p'
rg -n -F -- 'MASK_NVIDIA_DRIVER_PARAMS' . --glob '!vendor/**' || test "$?" -eq 1

Repository: NVIDIA/gpu-operator

Length of output: 6531


Web query:

containerd CRI privileged runAsUser non root ambient capabilities 2.3 exec all capabilities Kubernetes

Result:

Sources consulted:
- Kubernetes should configure the ambient capability set — GitHub issue 56374 in kubernetes/kubernetes (link omitted to avoid creating a cross-reference)
- How to Restrict Container Capabilities Using securityContext in Kubernetes — https://oneuptime.com/blog/post/2026-02-09-restrict-container-capabilities-security-context/view
- Chapter 7: Security Boundaries - Containerd From The Bottom Up — https://thecontainerdbook.com/chapters/part-2/07-security
- Capabilities - Building a Container from Scratch in Rust — https://brianshih1.github.io/mini-container/capabilities.html
- containerd should set ambient capabilities — GitHub issue 5644 in containerd/containerd (link omitted to avoid creating a cross-reference)

Script executed:

python3 - <<'PY'
import urllib.request
urls = [
'https://raw.githubusercontent.com/containerd/containerd/v2.3.5/internal/cri/server/container_create.go',
'https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/cmd/gpu-kubelet-plugin/driver.go',
'https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/deployments/helm/dra-driver-nvidia-gpu/templates/kubeletplugin.yaml',
]
for url in urls:
    print('\n###', url)
    try:
        text = urllib.request.urlopen(url, timeout=12).read().decode()
        lines = text.splitlines()
        needles = ('WithoutAmbientCaps', 'WithAllCurrentCapabilities', 'Privileged', 'kubeletplugin.Start(', 'Mkdir', 'runAsUser', 'podSecurityContext')
        hits = [i for i, line in enumerate(lines) if any(n in line for n in needles)]
        for i in hits:
            print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-4), min(len(lines),i+7))))
            print('---')
        if not hits:
            print('No selected symbols found')
    except Exception as exc:
        print(type(exc).__name__, str(exc))
PY

Repository: NVIDIA/gpu-operator

Length of output: 8306


Script executed:

python3 - <<'PY'
import json, urllib.request
base='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/'
def read(url):
    return urllib.request.urlopen(url, timeout=15).read().decode()
try:
    tree=json.loads(read('https://api.github.com/repos/kubernetes-sigs/dra-driver-nvidia-gpu/git/trees/v0.5.1?recursive=1'))['tree']
    paths=[x['path'] for x in tree if x['type']=='blob' and ((x['path'].startswith('cmd/gpu-kubelet-plugin/') and x['path'].endswith('.go')) or x['path'].endswith('Dockerfile') or x['path']=='go.mod')]
    print('Candidate files:', ', '.join(paths))
    for path in paths:
        if path.endswith('Dockerfile') and 'gpu' not in path and path != 'Dockerfile':
            continue
        try:
            lines=read(base+path).splitlines()
        except Exception as e:
            print(path, 'read failed:', e)
            continue
        needles=('func NewDeviceState(', 'os.Mkdir', 'MkdirAll(', 'k8s.io/dynamic-resource-allocation', 'USER ', 'FROM ', 'kubeletplugin.Start(', 'NewDriver(ctx')
        hits=[i for i,l in enumerate(lines) if any(n in l for n in needles)]
        if hits:
            print('\n###', base+path)
            for i in hits:
                print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-5),min(len(lines),i+12))))
                print('---')
except Exception as e:
    print('Inspection failed:', type(e).__name__, e)
PY

Repository: NVIDIA/gpu-operator

Length of output: 30528


Script executed:

python3 - <<'PY'
import urllib.request
base='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/'
for path, ranges in {
'cmd/gpu-kubelet-plugin/main.go':[(44,80),(110,150),(280,370),(375,430)],
'deployments/container/Dockerfile':[(1,200)],
}.items():
    print('\n###',base+path)
    try:
        lines=urllib.request.urlopen(base+path,timeout=15).read().decode().splitlines()
        for a,b in ranges:
            print('\n'.join(f'{i+1}: {lines[i]}' for i in range(a-1,min(b,len(lines)))))
    except Exception as e:
        print(type(e).__name__,e)
PY
cat -n tests/holodeck.yaml | grep -n -C 2 'containerd'
cat -n manifests/state-dra-driver/0500_daemonset.yaml | sed -n '192,205p;227,245p;363,370p'

Repository: NVIDIA/gpu-operator

Length of output: 18786


Script executed:

python3 - <<'PY'
import urllib.request
url='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/cmd/gpu-kubelet-plugin/main.go'
lines=urllib.request.urlopen(url,timeout=15).read().decode().splitlines()
for a,b in [(78,88),(190,220),(245,280)]:
 print(f'\n### {url}:{a}-{b}')
 print('\n'.join(f'{i+1}: {lines[i]}' for i in range(a-1,min(b,len(lines)))))
for i,l in enumerate(lines):
 if 'func main(' in l or 'RunPlugin(' in l or 'app.Run' in l:
  print(f'\n### {url}:{max(1,i-5)}-{min(len(lines),i+12)}')
  print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-6),min(len(lines),i+12))))
PY

Repository: NVIDIA/gpu-operator

Length of output: 5688


Web query:

man7 capabilities execve nonroot ambient file capabilities effective permitted Linux calculation

Result:

Sources consulted:
- capabilities(7) - Linux manual page — https://man7.org/linux/man-pages/man7/capabilities.7.html
- Linux capabilities: what the five sets actually do — RodHat — https://rodhat.com/tips/2026-09-20-linux-capabilities-set-model/
- How to execve a process, retaining capabilities in spite of missing filesystem-based capabilities? — https://stackoverflow.com/questions/14636359/how-to-execve-a-process-retaining-capabilities-in-spite-of-missing-filesystem-b
- Beyond Container Capabilities: Understanding Linux Capability Sets | utam0k — https://www.utam0k.jp/en/blog/2025/12/14/linux-capability-sets/
- capabilities: Ambient capabilities [LWN.net] — https://lwn.net/Articles/636533/

Script executed:

python3 - <<'PY'
from urllib.request import urlopen
url='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/cmd/gpu-kubelet-plugin/main.go'
lines=urlopen(url,timeout=15).read().decode().splitlines()
for i,line in enumerate(lines):
 if 'KUBELET_PLUGINS_DIRECTORY_PATH' in line:
  print(url)
  print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-5),min(len(lines),i+3))))
PY

Repository: NVIDIA/gpu-operator

Length of output: 642


Ensure the GPU plugin can start with a non-root pod UID.

With runAsUser: 1000 on containerd 2.3.5, the plugin can fail to create its directory under a root-owned kubelet plugins directory. The gpus container then exits and can crash-loop. Privileged mode does not preserve ambient capabilities across the non-root executable.

The render test does not detect this startup failure.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this only changes anything when daemonsets.podSecurityContext is set, and that matches clusterpolicy today: applyCommonDaemonsetConfig copies the same field onto every operand daemonset, including the device plugin that also writes under /var/lib/kubelet. the init container here already pins runAsUser: 0, and i left the plugin containers as they are so the change stays a straight port of the clusterpolicy behavior. @rahulait if you'd prefer the kubelet plugin containers to pin runAsUser: 0 as well so a non-root pod uid can't reach them, i can add that here.

@kvalliyurnatt

Copy link
Copy Markdown
Contributor

@dundysm thank you for your contribution. can you ensure the commits are signed with a valid GPG key

@dundysm
dundysm force-pushed the fix/gpucluster-pod-security-context branch from 76ae442 to f555964 Compare November 17, 2026 11:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[gpu-op] GPUCluster silently ignores daemonsets.podSecurityContext

2 participants