Skip to content

[Bug]: Precompiled node pools drop kernels that sanitize to the same name #3098

Open

Description

@SebTardif

Describe the bug

With spec.usePrecompiled: true, getNodePools keeps one node pool per sanitized kernel name. getSanitizedKernelVersion strips x86_64, aarch64, and an optional _64k. Two nodes whose raw feature.node.kubernetes.io/kernel-version.full values differ only by architecture or page size collapse into one pool.

The surviving pool's node selector and driver image tag use the first node's raw kernel. The other nodes get no driver DaemonSet. The NVIDIADriver can still become Ready.

This was found by reading internal/state/nodepool.go on main (operator v26.7.2). It is not a live cluster repro.

To reproduce

From the code path. TestGetSanitizedKernelVersion already expects these two strings to become 5.14.0-427.37.1.el9.4:

  • 5.14.0-427.37.1.el9_4.x86_64
  • 5.14.0-427.37.1.el9_4.aarch64

Put one GPU node on each kernel, with the same OS, under one NVIDIADriver that sets usePrecompiled: true.

On current main, getNodePools returns one pool. The unit test is:

go test -count=1 -run TestGetNodePoolsKeepsPrecompiledKernelsThatSanitizeTogether ./internal/state/

That test fails with 1 pool instead of 2.

Expected behavior

One pool per raw kernel. Nodes with the same raw kernel still share a pool. The sanitized name can stay shared. The selector and the image tag for each pool must keep that pool's own raw kernel.

I will open a pull request with that change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions