Skip to content

test_setup_choices fails on a host with a real CUDA install: find_nvcc()'s /usr/local/cuda* glob is not mocked #414

Closed

Description

@rwkeyes

What happens. tools/test_setup_choices.py::ExperimentalSm60.test_find_nvcc_below errors on any machine that has a CUDA toolkit installed:

ERROR: test_find_nvcc_below (test_setup_choices.ExperimentalSm60)
Traceback (most recent call last):
  File "tools/test_setup_choices.py", line 132, in test_find_nvcc_below
    self.assertEqual(setup.find_nvcc(), (str(new), (13, 0)))
  File "setup.py", line 604, in find_nvcc
    v = re.search(r"release (\d+)\.(\d+)", out([c, "--version"]))
  File "tools/test_setup_choices.py", line 131, in <lambda>
    mock.patch.object(setup, "out", lambda cmd: versions[cmd[0]]):
KeyError: '/usr/local/cuda-13.2/bin/nvcc'

Reproduced on two hosts, each failing on its own toolkit path — CUDA 13.2 at /usr/local/cuda-13.2 (plus /usr/bin/nvcc), and CUDA 12.8 at /usr/local/cuda-12.8:

KeyError: '/usr/local/cuda-12.8/bin/nvcc'

Seen from python -m unittest discover -s tools -p 'test_*.py' on 0.1.32 (c499bd1), on both a HIP/gfx1100 box and a CUDA 12.8 box.

Why. The test mocks WIN, shutil.which, CUDA_PATH and setup.out, but find_nvcc() also enumerates the real filesystem:

cands += [str(p / "bin" / "nvcc") for p in sorted(Path("/usr/local").glob("cuda*"), reverse=True)]
cands += [str(p / "bin" / "nvcc") for p in sorted(Path("/opt").glob("cuda*"), reverse=True)]   # Arch (#46)

Those real candidates are appended to the mocked ones. They are normally newer than the test's fake 13.0, so they win the loop, and the mocked out() has no entry for them → KeyError.

Confirmation. Faking the glob out — returning nothing for cuda* — makes the test pass unchanged:

real = pathlib.Path.glob
pathlib.Path.glob = lambda self, pat, **k: iter(()) if "cuda" in str(pat) else real(self, pat, **k)
# -> failures=0 errors=0

So the cause is the unmocked glob and nothing about the host.

Suggested fix. Mock the cuda-root enumeration as well (mock.patch.object(Path, "glob", ...)), or make the roots injectable so the test can point find_nvcc() at its temp directory.

This is not a product bug — find_nvcc() does exactly what it should, and the engine/setup path is unaffected. It is test hermeticity only, and it is invisible on a machine without CUDA installed, which is presumably why CI does not catch it.

Activity

  1. rwkeyes commented on Oct 1, 2026

    @rwkeyes
    ContributorAuthor

    Confirmed in a pristine tree, to rule out my own working branch:

    git worktree add <dir> v0.1.32          # detached HEAD at c499bd1
    cmake -S . -B build                     # configure exit 0
    cmake --build build -j8                 # build exit 0
    python -m unittest discover -s tools -p 'test_*.py'
    
    Ran 138 tests in 0.555s
    FAILED (errors=2)
    KeyError: '/usr/local/cuda-13.2/bin/nvcc'
    

    Same failure, same cause, with none of my local changes present — so this is not something my branch introduced.

    The other error in that run is unrelated and environmental, and I am explicitly not claiming it as a bug: test_iq_pack cannot import without llama.cpp's gguf-py, and it says so with an actionable message (run setup.py (it clones llama.cpp into third_party/), or set STRATA_GGUF_PY). That is fine behaviour; it just means a fresh worktree runs 138 of the intended tests until the checkout is present.

  2. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Owner

    Thanks! Fixed for the next release: the mocked --version answers an empty string for any toolkit the test didn't create, so a real CUDA install on the host is skipped. This issue closes with the release.

  3. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Owner

    Fixed in 0.1.33. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions