Repository navigation
Description
Is your feature request related to a specific problem?
GPU Operator currently supports either dynamically generated MIG profiles or a user-provided static MIG configuration.
When no custom ConfigMap is configured, MIG Manager generates hardware-aware profiles from the detected GPU capabilities. This keeps the profile set aligned with new GPU models, MIG profile changes, and upstream updates. The generated configuration also contains device-specific entries using device-filter.
However, when a custom MIG ConfigMap is configured, MIG Manager uses only that ConfigMap. The generated profiles are no longer available. Users therefore have to copy the complete upstream configuration, including all standard profiles and device-filter entries, and add their own profiles manually.
This creates an operational risk: the copied configuration will inevitably become stale as the upstream profile database evolves. Someone must periodically compare it with NVIDIA’s upstream 0400_configmap.yaml and update it, which is easy to miss across multiple clusters.
Describe the solution you would like
Please add an opt-in merge mode that combines the dynamically generated configuration with user-defined profiles.
For example:
migManager:
config:
name: custom-mig-parted-config
mergeWithGenerated: trueThe resulting profile set would contain both the generated profiles and the custom profiles. A custom ConfigMap would only need to define organization-specific profiles, for example:
version: v1
mig-configs:
x100-80gb-10-20-20-20:
- device-filter: ["0x233010DE", "0x233110DE"]
devices: all
mig-enabled: true
mig-devices:
"1g.10gb": 1
"2g.20gb": 3
x100-80gb-20-20-40:
- device-filter: ["0x233010DE", "0x233110DE"]
devices: all
mig-enabled: true
mig-devices:
"2g.20gb": 2
"3g.40gb": 1On an H100 80GB node, users would then be able to select both generated profiles such as all-1g.10gb, all-2g.20gb, and all-balanced, and custom profiles such as x100-80gb-20-20-40.
Describe alternatives you have considered
The current workaround is to vendor the entire upstream 0400_configmap.yaml into every deployment and append custom profiles to it. This works, but creates a permanent maintenance burden and a risk of silently falling behind upstream.
Another workaround would be to patch the generated ConfigMap using a Kubernetes Job or hook. This is racy because the generated ConfigMap is node-specific and may be recreated or overwritten by MIG Manager.
Additional details
Suggested merge semantics:
- Generated profiles remain available by default.
- Custom profiles are appended.
- If a custom profile has the same name as a generated profile, the custom definition takes precedence.
- Duplicate names should produce a clear log message.
device-filtermust work for both generated and custom profiles.- Invalid custom profiles should fail validation with an actionable error.
- Existing behavior must remain unchanged when merge mode is disabled.
This would preserve the convenience and freshness of dynamically generated profiles while allowing clusters to define a small set of stable, organization-specific combined layouts.
Activity
botinca commentedon Sep 17, 2026
A related concern is the current
single/mixedMIG strategy switch. With custom profiles that contain multiple MIG device types (for example,2g.20gb: 2plus3g.40gb: 1), the distinction between the two modes may be outdated or at least unnecessarily manual.Could the strategy be auto-detected from the selected MIG configuration? In particular, if the selected profile contains multiple MIG device types, the operator/device plugin could automatically use the required mixed behavior. Alternatively, the
single/mixedsetting could be deprecated or made optional when the selected profile unambiguously requires mixed mode.This would avoid a second independent setting that can disagree with the selected profile and lead to confusing failures or missing
nvidia.com/mig-*resources.rajathagasthya commentedon Sep 18, 2026
Thanks for feature request!
We don't plan to add a merge mode. The generated config is derived from each node's hardware, so a merged result would differ per node, and we don't want to have precedence rules between generated and custom entries.
One thing we can consider doing is have the mig-manager always publish the hardware-generated profiles to the per-node
<node>-mig-configConfigMap, even when a custom config is in use. That gives you a fresh configs to copy and lets you build your custom ConfigMap from it however you want.On the
single/mixedquestion: the strategy sets which resource name a node advertises (nvidia.com/gpuornvidia.com/mig-*). Deriving it from the applied profile would change a node's resources whenever itsmig.configlabel changes, breaking pods that requestnvidia.com/gpu. FYImixedworks for uniform profiles too if you just want one cluster-wide setting.botinca commentedon Sep 18, 2026
Yes, thanks for the clarification (reg mig strategy). I carefully read through the documentation again this morning, so the question about the migration strategy is no longer relevant.
The "mixed" strategy seems more straightforward, though less compatible. And, in all likelihood, with the arrival of DRA, these naming distinctions will lose their significance entirely.
Reg the main topic:
Well, that might be okay, but I see a potential race condition here: if I were to modify a custom ConfigMap on the fly—say, using a job or a hook—the event triggering the read of that ConfigMap could occur before the update event. This doesn't strike me as an elegant or reliable way to handle customization.
What prevents us from describing all possible profiles in the dynamic version? The number of profile combinations is certainly significant. I realize few people need this, but https://github.com/NVIDIA/gpu-operator/blob/main/assets/state-mig-manager/0400_configmap.yaml wouldn't become much heavier with an extra 100 lines.
kvnloo commentedon Sep 27, 2026
The maintainer direction makes sense to me: avoiding automatic precedence/merge rules keeps generated hardware-specific state separate from user-owned config.
The alternative you mentioned — always publishing the hardware-generated profiles to the per-node
<node>-mig-configConfigMap even when a custom config is active — seems useful on its own.I would define that contract as:
<node>-mig-configalways reflects the current node hardware;That avoids the race from trying to mutate a custom ConfigMap in place while still giving automation a fresh source from which to build organization-specific configurations.
Would a focused PR implementing only that publication behavior be useful?
rajathagasthya commentedon Sep 28, 2026
Enumerating every combination of profiles has a permanent cost. Every generated name becomes something users select by label and we can never rename, and every combination has to be validated for each new GPU. It still wouldn't remove the need for custom configs (e.g. if users want different profiles for different devices), so those users would be back to the same question. The current split is that generated configs cover the common uniform layouts and custom config covers anything else.
As for the race condition you're describing: I believe it can only occur when a target profile is preset on the node and the profiles reported by the driver changes (which is incredibly rare), which then triggers your automation. One option is to set MIG config to
all-disabledbefore copying configs and then set it to the desired profile.We can have mig-manager write the ConfigMap with generated profiles unconditionally. That is a good middle ground considering the complexity enumeration would bring IMO.
Side note: we will remove https://github.com/NVIDIA/gpu-operator/blob/main/assets/state-mig-manager/0400_configmap.yaml and won't use it as fallback anymore with the dynamically generated config.