Skip to content

gpu-config [13/13]: deploy the GPU configuration manager from GPUCluster - #3259

Open
henry118 wants to merge 13 commits into
NVIDIA:mainfrom
henry118:gpu-config/13-gpucluster
Open

henry118 wants to merge 13 commits into
NVIDIA:mainfrom
henry118:gpu-config/13-gpucluster

Conversation

@henry118

@henry118 henry118 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Part of #3246 (13/13). Depends on: #3247 to #3258. Review only the last commit.

Adds spec.gpuConfigManager to GPUCluster, the DRA-stack manager
DaemonSet, and the gpu-config init container in the DRA kubelet plugin.
Kubelet plugin pods roll once on upgrade.


Devin Review

@henry118
henry118 requested a review from a team as a code owner November 17, 2026 19:09
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

Adds a GPU configuration manager for container and DRA stacks. The change defines configuration documents and GPU matching, applies power, clock, and ECC settings through NVML, and publishes node status labels. It adds a validator gate, Kubernetes clients and ConfigMap watches, state persistence, and operator APIs, templates, RBAC, and DaemonSets. The manager is disabled by default in the Helm values. The build includes the manager executable and enables CGO for that command.

Merge Risk: Moderate · up to 25ec5

This change adds a node-level GPU settings manager and an init gate that holds GPU allocators until settings are decided for each driver load. Several decision inputs can be stale or misread: cached DaemonSet and label reads, pod lists bounded by an old node version, and failed init containers counted as success. As a result, GPU settings can be silently skipped for a driver load, or applied while workloads are starting. A malformed env entry can also block the DRA plugin rollout. The feature is off by default, but these issues should be fixed before it is relied on.

Security architecture risk: Minimal · up to 25ec5

The inspected configuration and startup paths preserve driver-load identity, serialize configuration with allocator release, and hold startup after incomplete application. No introduced security bypass or material recovery failure was established. The privileged permissions and non-automatic rollback behavior remain important operational constraints.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — A compromised manager service-account credential could exercise cluster-wide node patch, pod listing, claim reads, and ResourceSlice listing; scheduling and NODE_NAME are not authorization restrictions. This is a new credential holder, but cluster-wide node mutation already exists in the base DRA plugin and MIG manager. The inspected normal publisher writes only three GPU-configuration status labels on its configured node.

Security Findings and Attack Paths

  • inferred — Write access to a selected ConfigMap becomes authority over supported GPU settings on the nodes selecting it, potentially an entire enabled fleet for the cluster document. Invalid configuration can hold allocator startup. This follows the exposed configuration contract; no inspected path converts document content into commands, Kubernetes credentials, or an arbitrary node-patch target.

Trust Boundaries and Controls

  • observed — Document watches are restricted in code to selected names in one namespace and must initially synchronize. Parsing rejects unsupported schema and unknown settings; assignment overlap and supported device ranges are checked before writes. Configuration data does not choose the publisher's node identity.

Resilience and Maintainability Implications

  • observed — The protocol checks decisions again under the shared lock, rechecks driver identity after opening device handles, and checks identity again before gate release. Prior gate passes and allocator or GPU-pod evidence can force a no-write skip. Gate API-read failures treat the manager as expected rather than bypassing the wait; intentional no-manager and bounded no-module exits record their passes.

Hardening Proposals

  • proposed — Consider mediating status publication or enforcing a dedicated admission policy so manager credentials cannot modify arbitrary node fields. Treat selected ConfigMap write access as hardware-administration authority. These are defense-in-depth proposals, not verified bypass findings.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.displayName="Enable hostNetwork for NVIDIA GPU Configuration Manager"
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.x-descriptors="urn:alm:descriptor:com.tectonic.ui:booleanSwitch"
HostNetwork *bool `json:"hostNetwork,omitempty"`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the use cases for hostNetwork with the config manager?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there was concerns with IP exhaustion with DS based design. This is a way to workaround it.

@henry118
henry118 force-pushed the gpu-config/13-gpucluster branch from 114d1b7 to 25ec543 Compare November 18, 2026 23:11

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

Minor · Removing the output directory can break the GPU configuration lock. · main.go:530

cmd/nvidia-validator/main.go:530
Minor · Stability & Availability · Quick win

Removing the output directory can break the GPU configuration lock.

--cleanup-all calls os.RemoveAll(outputDirFlag) before any lock is taken. The GPU configuration store locks .gpu-config.lock by pathname and holds flock on the opened file. If cleanup runs while an applier or gate holds that lock, the holder keeps the unlinked file. A later lock attempt creates a new file at the same path and acquires it. Both processes can then hold the exclusive lock and write .gpu-config-state.

Nothing in the repository prevents an operator from running --cleanup-all (or setting CLEANUP_ALL) on a node while GPU configuration is in progress. The cleanup path must not remove the lock or state files while a holder can use them. Acquire the state lock before removal, or exclude the lock and state files from removal.

Other comments: 10

api/nvidia/v1/clusterpolicy_types.go: 1

1833-1834: Minor · Functional Correctness · Quick win

Document when the manager reads the pool label.

If a node moves from training to inference after the manager starts but before its first decision, the manager still uses the training ConfigMap. It sets up the watches from the startup label. The later Node read does not change those watches. State that the manager reads the pool label at startup and that relabeling does not change the selected pool until the manager restarts. (raw.githubusercontent.com)

internal/gpuconfig/config/document.go: 2

177-179: Minor · Data Integrity & Integration · Quick win

Reject entry names that lose their identity during JSON conversion.

YAML v2 decodes a !!binary key into a Go string, so the string-type check accepts it. If that key contains an invalid UTF-8 byte and another entry uses U+FFFD, JSON conversion gives both entries the same key. One entry disappears before Resolve can report an overlap. Reject invalid UTF-8 in both explicit and merged entry names before accepting the document. (raw.githubusercontent.com)


126-127: Minor · Functional Correctness · Quick win

Recognize merged schema versions in the fallback.

If <<: {version: v2} supplies the version and a new field contains a null map key, strict conversion fails. The fallback decodes into yamlv2.MapSlice, which drops the merged version. The manager reports a schema error instead of telling the operator that it does not support v2. Use a merge-aware lookup in the fallback and add a regression case for this input. (raw.githubusercontent.com)

internal/gpuconfig/statefile/file.go: 1

210-210: Minor · Data Integrity & Integration · Quick win

Preserve the apply decision when writing a gate pass.

If the file contains "apply":null followed by "Apply":{"module":7,"value":"applied"}, the reader sees an applied decision for load 7. A gate-only write sorts the keys in the opposite order, so null wins. The file still parses, but the next applier poll no longer sees the load as decided. Decode known state keys by exact spelling, or normalize aliases without changing the effective value of unrelated records. (raw.githubusercontent.com)

internal/gpuconfig/config/match_test.go: 1

81-85: Minor · Maintainability & Code Quality · Quick win

Test a match on the second product-name pattern.

Every product-name selector in these fixtures has one pattern. An implementation that checks only the first pattern would pass. Add a case such as productName: ["NVIDIA L40*", "NVIDIA H100*"] against the H100 fixture and assert that the entry's power limit is selected.

As per coding guidelines, “New behavior needs appropriate test coverage.”

Source: Coding guidelines

internal/gpuconfig/kube/documents_test.go: 2

130-134: Minor · Functional Correctness · Quick win

Assert that the edited document becomes visible before deletion.

The test changes only ResourceVersion. It passes if an update triggers onChange but leaves Snapshot() serving the old data. Change config.yaml and wait for Snapshot() to return the new value before sending the delete.

As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions


172-172: Minor · Functional Correctness · Quick win

Test name scoping on the watch path.

This test calls only ListWithContextFunc. It would still pass if WatchFuncWithContext lost its name selector and started receiving other nodes’ ConfigMap events. Exercise the watch path and assert that it excludes an event for other. (raw.githubusercontent.com)

As per coding guidelines, “New behavior needs appropriate test coverage.”

Source: Coding guidelines

internal/gpuconfig/decision/decide.go: 1

358-358: Minor · Stability & Availability · Quick win

Preserve the retryable load-ID error after an NVML open failure.

If NVML initialization fails and the follow-up load-ID read also fails, openFailed discards the ErrTransient error. The applier then exits on the raw NVML error instead of retrying on the next poll. Return the transient error with the NVML error as context. Keep the fatal NVML result when the load-ID read succeeds and confirms that the load did not change. (raw.githubusercontent.com)

internal/gpuconfig/decision/decide_test.go: 1

528-530: Minor · Functional Correctness · Quick win

Test the idle path with the state lock held.

This case passes if Decide acquires the state lock before checking for an existing decision, because nothing holds the lock. Hold the lock independently and verify, with a bounded wait, that Decide returns Idle before the holder releases it.

As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

manifests/state-dra-driver/0500_daemonset.yaml: 1

150-153: Minor · Functional Correctness · Quick win

Reject or skip spec.gpuConfigManager.env entries with an empty name.

dedupeEnv keeps every entry, including one with an empty name. The DRA DaemonSet template then renders it as - name: with a null name. The Kubernetes API rejects a container env entry that has no name. The DaemonSet apply then fails for that reconcile, and the DRA kubelet plugin update does not roll out.

The duplicate-name case is already handled: dedupeEnv keeps one entry per name.

The fix must ensure that an empty-named entry cannot reach the rendered env list. Either validation rejects it, or the entry is dropped before rendering. The choice between these two is open, so this comment does not prescribe one.

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 9e39c65 and 25ec543.

Files ignored due to path filters: 51

  • api/nvidia/v1/zz_generated.deepcopy.go is excluded by !**/zz_generated.*.go
  • api/nvidia/v1alpha1/zz_generated.deepcopy.go is excluded by !**/zz_generated.*.go
  • bundle/manifests/nvidia.com_clusterpolicies.yaml is excluded by !bundle/manifests/nvidia.com_*.yaml
  • bundle/manifests/nvidia.com_gpuclusters.yaml is excluded by !bundle/manifests/nvidia.com_*.yaml
  • config/crd/bases/nvidia.com_clusterpolicies.yaml is excluded by !config/crd/bases/**
  • config/crd/bases/nvidia.com_gpuclusters.yaml is excluded by !config/crd/bases/**
  • deployments/gpu-operator/crds/nvidia.com_clusterpolicies.yaml is excluded by !deployments/gpu-operator/crds/**
  • deployments/gpu-operator/crds/nvidia.com_gpuclusters.yaml is excluded by !deployments/gpu-operator/crds/**
  • go.sum is excluded by !**/*.sum, !go.sum
  • internal/state/testdata/golden/gpucluster-dra-driver-full-spec.yaml is excluded by !**/testdata/**
  • internal/state/testdata/golden/gpucluster-dra-driver-gpus-only.yaml is excluded by !**/testdata/**
  • internal/state/testdata/golden/gpucluster-dra-driver-passthrough-support.yaml is excluded by !**/testdata/**
  • internal/state/testdata/golden/gpucluster-gpu-config-manager.yaml is excluded by !**/testdata/**
  • vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
  • vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 81

  • Makefile
  • THIRD_PARTY_NOTICES.md
  • api/nvidia/v1/clusterpolicy_types.go
  • api/nvidia/v1alpha1/gpucluster_types.go
  • assets/state-gpu-config-manager/0100_service_account.yaml
  • assets/state-gpu-config-manager/0200_role.yaml
  • assets/state-gpu-config-manager/0210_clusterrole.yaml
  • assets/state-gpu-config-manager/0300_rolebinding.yaml
  • assets/state-gpu-config-manager/0310_clusterrolebinding.yaml
  • assets/state-gpu-config-manager/0500_daemonset.yaml
  • assets/state-operator-validation/0500_daemonset.yaml
  • bundle/manifests/gpu-operator-certified.clusterserviceversion.yaml
  • cmd/gpu-config-manager/main.go
  • cmd/gpu-config-manager/main_test.go
  • cmd/nvidia-validator/gpuconfig.go
  • cmd/nvidia-validator/gpuconfig_names_test.go
  • cmd/nvidia-validator/gpuconfig_test.go
  • cmd/nvidia-validator/main.go
  • controllers/gpuconfig_test.go
  • controllers/nodelabeling_controller_test.go
  • controllers/object_controls.go
  • controllers/state_manager.go
  • deployments/gpu-operator/templates/clusterpolicy.yaml
  • deployments/gpu-operator/templates/gpucluster.yaml
  • deployments/gpu-operator/values.yaml
  • docker/Dockerfile
  • go.mod
  • internal/gpuconfig/applier/applier.go
  • internal/gpuconfig/applier/applier_test.go
  • internal/gpuconfig/apply/apply.go
  • internal/gpuconfig/apply/apply_test.go
  • internal/gpuconfig/apply/fake/fake.go
  • internal/gpuconfig/apply/nvml.go
  • internal/gpuconfig/apply/nvmllib/nvmllib.go
  • internal/gpuconfig/apply/nvmllib/nvmllib_test.go
  • internal/gpuconfig/config/document.go
  • internal/gpuconfig/config/document_test.go
  • internal/gpuconfig/config/match.go
  • internal/gpuconfig/config/match_test.go
  • internal/gpuconfig/config/schema.go
  • internal/gpuconfig/decision/decide.go
  • internal/gpuconfig/decision/decide_test.go
  • internal/gpuconfig/decision/evidence.go
  • internal/gpuconfig/decision/helpers_test.go
  • internal/gpuconfig/decision/pods.go
  • internal/gpuconfig/gate/gate.go
  • internal/gpuconfig/gate/gate_test.go
  • internal/gpuconfig/kube/client.go
  • internal/gpuconfig/kube/client_test.go
  • internal/gpuconfig/kube/documents.go
  • internal/gpuconfig/kube/documents_test.go
  • internal/gpuconfig/loadid/driver.go
  • internal/gpuconfig/loadid/loadid.go
  • internal/gpuconfig/loadid/loadid_test.go
  • internal/gpuconfig/publish/publish.go
  • internal/gpuconfig/publish/publish_test.go
  • internal/gpuconfig/statefile/file.go
  • internal/gpuconfig/statefile/file_test.go
  • internal/gpuconfig/statefile/state.go
  • internal/gpuconfig/statefile/state_test.go
  • internal/gpuconfig/status/sync.go
  • internal/gpuconfig/status/sync_test.go
  • internal/state/dra_driver.go
  • internal/state/dra_driver_test.go
  • internal/state/gpu_config_manager.go
  • internal/state/gpu_config_manager_test.go
  • internal/state/gpucluster.go
  • internal/state/gpucluster_render_test.go
  • internal/state/gpucluster_scc_test.go
  • internal/state/manager.go
  • internal/state/manager_test.go
  • internal/state/types.go
  • manifests/state-dra-driver/0150_kubeletplugin-driver-validation-role.yaml
  • manifests/state-dra-driver/0450_scc.openshift.yaml
  • manifests/state-dra-driver/0500_daemonset.yaml
  • manifests/state-gpu-config-manager/0100_serviceaccount.yaml
  • manifests/state-gpu-config-manager/0200_role.yaml
  • manifests/state-gpu-config-manager/0210_clusterrole.yaml
  • manifests/state-gpu-config-manager/0300_rolebinding.yaml
  • manifests/state-gpu-config-manager/0310_clusterrolebinding.yaml
  • manifests/state-gpu-config-manager/0500_daemonset.yaml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment thread internal/gpuconfig/decision/pods.go Outdated
return false
}
st := statuses(pod)
if s, ok := st[toolkitContainer]; ok && s.State.Terminated != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Require a successful exit from the toolkit wait.

If kubelet kills toolkit-validation while it is still waiting for toolkit-ready, the device plugin has not advanced. On a reload, this check treats the failed termination as allocator evidence and records a skip for the whole load. Require exit code zero for the toolkit init container. Keep the separate check for evidence that a later container started. (raw.githubusercontent.com)

Comment thread internal/gpuconfig/decision/pods.go Outdated
// Containers after the gate; all of them for a pod without one.
later := names
if i := slices.Index(names, GateContainer); i >= 0 {
if s, ok := st[GateContainer]; ok && s.State.Terminated != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Do not count a failed gpu-config init container as a gate pass.

On a driver reload, a replacement DRA plugin’s gpu-config init container can terminate with a nonzero exit code before the gate passes. This check reports a pass even if no later container ran. The decider records Skipped and will not apply settings again for that load. Require exit code zero here. Keep the separate check for evidence that a later container ran, and add a nonzero-exit regression case. (kubernetes.io)

)

// fromCache asks the API server to answer from its watch cache.
var fromCache = &client.GetOptions{Raw: &metav1.GetOptions{ResourceVersion: "0"}}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Do not use unconstrained cached reads to disable the gate.

With ResourceVersion: "0", Kubernetes can return stale data. After the manager DaemonSet is created, DaemonSetExists can still report it absent. A stale "false" deployment label has the same effect. (kubernetes.io)

Gate.readExpected treats either result as no manager expected. The gate can then record a pass before configuration completes. That pass makes the applier skip configuration for the driver load. The lock prevents concurrent writes, but does not correct the stale expectation. Use Most Recent reads before allowing either result to release the gate. (raw.githubusercontent.com)

Comment thread internal/gpuconfig/kube/client.go Outdated
Comment on lines +145 to +146
ResourceVersion: resourceVersion,
ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Quick win

Use a current pod list for the GPU safety check.

A Node's metadata.resourceVersion identifies its last modification, not the time of a quorum read. If the Node remains at version 100 and a GPU pod starts at version 200, this request can accept a pod-cache snapshot at version 150. NotOlderThan permits that result. (kubernetes.io)

The decision caller uses this list to detect GPU pods before applying settings. An omitted pod or stale pod status can therefore bypass that check. Request Most Recent list semantics by leaving both resource-version fields unset. Update the test to check this freshness requirement. (raw.githubusercontent.com)

Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/13-gpucluster branch from 25ec543 to e7fb17d Compare November 19, 2026 00:16
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
…r step

Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/13-gpucluster branch from e7fb17d to 9f5c870 Compare November 19, 2026 00:50
@henry118 henry118 changed the title gpu-config: deploy the GPU configuration manager from GPUCluster gpu-config [13/13]: deploy the GPU configuration manager from GPUCluster Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants