Skip to content

[Feature]: Merge custom MIG profiles with dynamically generated profiles #2913

Open

Description

@botinca

Is your feature request related to a specific problem?

GPU Operator currently supports either dynamically generated MIG profiles or a user-provided static MIG configuration.

When no custom ConfigMap is configured, MIG Manager generates hardware-aware profiles from the detected GPU capabilities. This keeps the profile set aligned with new GPU models, MIG profile changes, and upstream updates. The generated configuration also contains device-specific entries using device-filter.

However, when a custom MIG ConfigMap is configured, MIG Manager uses only that ConfigMap. The generated profiles are no longer available. Users therefore have to copy the complete upstream configuration, including all standard profiles and device-filter entries, and add their own profiles manually.

This creates an operational risk: the copied configuration will inevitably become stale as the upstream profile database evolves. Someone must periodically compare it with NVIDIA’s upstream 0400_configmap.yaml and update it, which is easy to miss across multiple clusters.

Describe the solution you would like

Please add an opt-in merge mode that combines the dynamically generated configuration with user-defined profiles.

For example:

migManager:
  config:
    name: custom-mig-parted-config
    mergeWithGenerated: true

The resulting profile set would contain both the generated profiles and the custom profiles. A custom ConfigMap would only need to define organization-specific profiles, for example:

version: v1
mig-configs:
  x100-80gb-10-20-20-20:
    - device-filter: ["0x233010DE", "0x233110DE"]
      devices: all
      mig-enabled: true
      mig-devices:
        "1g.10gb": 1
        "2g.20gb": 3

  x100-80gb-20-20-40:
    - device-filter: ["0x233010DE", "0x233110DE"]
      devices: all
      mig-enabled: true
      mig-devices:
        "2g.20gb": 2
        "3g.40gb": 1

On an H100 80GB node, users would then be able to select both generated profiles such as all-1g.10gb, all-2g.20gb, and all-balanced, and custom profiles such as x100-80gb-20-20-40.

Describe alternatives you have considered

The current workaround is to vendor the entire upstream 0400_configmap.yaml into every deployment and append custom profiles to it. This works, but creates a permanent maintenance burden and a risk of silently falling behind upstream.

Another workaround would be to patch the generated ConfigMap using a Kubernetes Job or hook. This is racy because the generated ConfigMap is node-specific and may be recreated or overwritten by MIG Manager.

Additional details

Suggested merge semantics:

  • Generated profiles remain available by default.
  • Custom profiles are appended.
  • If a custom profile has the same name as a generated profile, the custom definition takes precedence.
  • Duplicate names should produce a clear log message.
  • device-filter must work for both generated and custom profiles.
  • Invalid custom profiles should fail validation with an actionable error.
  • Existing behavior must remain unchanged when merge mode is disabled.

This would preserve the convenience and freshness of dynamically generated profiles while allowing clusters to define a small set of stable, organization-specific combined layouts.

Activity

  1. added
    featureissue/PR that proposes a new feature or functionality
    needs-triageissue or PR has not been assigned a priority-px label
    on Sep 17, 2026
  2. botinca commented on Sep 17, 2026

    @botinca
    Author

    A related concern is the current single / mixed MIG strategy switch. With custom profiles that contain multiple MIG device types (for example, 2g.20gb: 2 plus 3g.40gb: 1), the distinction between the two modes may be outdated or at least unnecessarily manual.

    Could the strategy be auto-detected from the selected MIG configuration? In particular, if the selected profile contains multiple MIG device types, the operator/device plugin could automatically use the required mixed behavior. Alternatively, the single / mixed setting could be deprecated or made optional when the selected profile unambiguously requires mixed mode.

    This would avoid a second independent setting that can disagree with the selected profile and lead to confusing failures or missing nvidia.com/mig-* resources.

  3. rajathagasthya commented on Sep 18, 2026

    @rajathagasthya
    Contributor

    Thanks for feature request!

    We don't plan to add a merge mode. The generated config is derived from each node's hardware, so a merged result would differ per node, and we don't want to have precedence rules between generated and custom entries.

    One thing we can consider doing is have the mig-manager always publish the hardware-generated profiles to the per-node <node>-mig-config ConfigMap, even when a custom config is in use. That gives you a fresh configs to copy and lets you build your custom ConfigMap from it however you want.

    On the single/mixed question: the strategy sets which resource name a node advertises (nvidia.com/gpu or nvidia.com/mig-*). Deriving it from the applied profile would change a node's resources whenever its mig.config label changes, breaking pods that request nvidia.com/gpu. FYI mixed works for uniform profiles too if you just want one cluster-wide setting.

  4. botinca commented on Sep 18, 2026

    @botinca
    Author

    Yes, thanks for the clarification (reg mig strategy). I carefully read through the documentation again this morning, so the question about the migration strategy is no longer relevant.

    The "mixed" strategy seems more straightforward, though less compatible. And, in all likelihood, with the arrival of DRA, these naming distinctions will lose their significance entirely.

    Reg the main topic:

    That gives you a fresh configs to copy and lets you build your custom ConfigMap from it however you want.

    Well, that might be okay, but I see a potential race condition here: if I were to modify a custom ConfigMap on the fly—say, using a job or a hook—the event triggering the read of that ConfigMap could occur before the update event. This doesn't strike me as an elegant or reliable way to handle customization.

    What prevents us from describing all possible profiles in the dynamic version? The number of profile combinations is certainly significant. I realize few people need this, but https://github.com/NVIDIA/gpu-operator/blob/main/assets/state-mig-manager/0400_configmap.yaml wouldn't become much heavier with an extra 100 lines.

  5. kvnloo commented on Sep 27, 2026

    @kvnloo

    The maintainer direction makes sense to me: avoiding automatic precedence/merge rules keeps generated hardware-specific state separate from user-owned config.

    The alternative you mentioned — always publishing the hardware-generated profiles to the per-node <node>-mig-config ConfigMap even when a custom config is active — seems useful on its own.

    I would define that contract as:

    • generated profile discovery always runs;
    • <node>-mig-config always reflects the current node hardware;
    • selecting a custom ConfigMap changes what MIG Manager consumes, not whether generated discovery is published;
    • the generated ConfigMap remains operator-owned/read-only output;
    • modifying or replacing the custom ConfigMap is never part of the generated-config reconciliation path.

    That avoids the race from trying to mutate a custom ConfigMap in place while still giving automation a fresh source from which to build organization-specific configurations.

    Would a focused PR implementing only that publication behavior be useful?

  6. rajathagasthya commented on Sep 28, 2026

    @rajathagasthya
    Contributor

    What prevents us from describing all possible profiles in the dynamic version? The number of profile combinations is certainly significant. I realize few people need this, but https://github.com/NVIDIA/gpu-operator/blob/main/assets/state-mig-manager/0400_configmap.yaml wouldn't become much heavier with an extra 100 lines.

    Enumerating every combination of profiles has a permanent cost. Every generated name becomes something users select by label and we can never rename, and every combination has to be validated for each new GPU. It still wouldn't remove the need for custom configs (e.g. if users want different profiles for different devices), so those users would be back to the same question. The current split is that generated configs cover the common uniform layouts and custom config covers anything else.

    As for the race condition you're describing: I believe it can only occur when a target profile is preset on the node and the profiles reported by the driver changes (which is incredibly rare), which then triggers your automation. One option is to set MIG config to all-disabled before copying configs and then set it to the desired profile.

    We can have mig-manager write the ConfigMap with generated profiles unconditionally. That is a good middle ground considering the complexity enumeration would bring IMO.

    Side note: we will remove https://github.com/NVIDIA/gpu-operator/blob/main/assets/state-mig-manager/0400_configmap.yaml and won't use it as fallback anymore with the dynamically generated config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

featureissue/PR that proposes a new feature or functionalitylifecycle/frozen

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions