Linux build fails on Ubuntu 26.04 (glibc 2.43) with CUDA 12.9 (nvcc host-math conflict) #601

Environment

• OS: Ubuntu 26.04, glibc 2.43 (ldd --version → 2.43-2ubuntu2.3)
• GPU: 2× RTX 3060 12 GB, driver 595.84
• CUDA toolkits installed: 12.8 and 12.9 (/usr/local/cuda-12.8, /usr/local/cuda-12.9; /usr/local/cuda → 12.9)
• gcc/g++ available: 13, 14, 15 (default g++ = 15)
• Strata: v0.1.37, commit db4f91a, Linux source tree. (v0.1.38 came out the same day; its find_nvcc() is byte-identical to 0.1.37's, so this should still reproduce there.)

Problem 1 — default gcc is too new for the CUDA headers

/usr/local/cuda-12.9/bin/../targets/x86_64-linux/include/crt/host_config.h:143:2:
error: #error -- unsupported GNU version! gcc versions later than 14 are not supported!

Workaround: CXX=/usr/bin/g++-14 CUDAHOSTCXX=/usr/bin/g++-14 ./setup.sh …
(a successful run prints [ok] build tools present (CUDA 12.8) and later [ok] engine compiled: …)

Problem 2 — CUDA 12.9 host-math declarations conflict with glibc 2.43 (appears once #1 is fixed)

/usr/include/x86_64-linux-gnu/bits/mathcalls.h(83): error: exception specification is
incompatible with that of previous function "cospi" (declared at line 2601 of
.../crt/math_functions.h)

… same for sinpi, rsqrt, cospif, sinpif, rsqrtf ⇒ 6 errors detected in the compilation of "t.cu".
CMake's compiler probe is itself the first CUDA compile, so it fails there —
Compiling the CUDA compiler identification source file "CMakeCUDACompilerId.cu" failed — i.e. nothing of Strata is compiled.

Minimal reproduction — no Strata code needed:

printf '#include <cuda_runtime.h>\nint main(){return 0;}\n' > t.cu
/usr/local/cuda-12.9/bin/nvcc -ccbin=/usr/bin/g++-14 -c t.cu -o t.o   # 6 errors
/usr/local/cuda-12.8/bin/nvcc -ccbin=/usr/bin/g++-14 -c t.cu -o t.o   # OK

Tried and it did not help: -D__GLIBC_USE_IEC_60559_FUNCS_EXT_C23=0
(glibc re-defines that macro in bits/libc-header-start.h L96–102, so a command-line define is overwritten).

Why the obvious workaround fails

find_nvcc() (v0.1.37, setup.py L701–722) builds its candidate list from shutil.which("nvcc")
plus every toolkit under /usr/local/cuda* and /opt/cuda*, then keeps the highest version
(the loop comment says "every toolkit found; the newest wins"). So pointing PATH at 12.8 does not
help — 12.8 is found and evaluated, but loses the version comparison to 12.9.

Workaround we used (patch shown as an illustration, applied to v0.1.37's find_nvcc()):

if os.environ.get("STRATA_NVCC"):          # our local patch
    cands = [os.environ["STRATA_NVCC"]]    # replace the whole candidate list
else:
    cands = [shutil.which("nvcc")]
    cands += [str(p / "bin" / "nvcc") for p in sorted(Path("/usr/local").glob("cuda*"), reverse=True)]
    cands += [str(p / "bin" / "nvcc") for p in sorted(Path("/opt").glob("cuda*"), reverse=True)]   # Arch (#46)

then STRATA_NVCC=/usr/local/cuda-12.8/bin/nvcc CUDAHOSTCXX=/usr/bin/g++-14 CXX=/usr/bin/g++-14 ./setup.sh …

Suggestion (any of these would unblock new Linux distros)

1. Document "on glibc 2.43 use CUDA ≤ 12.8" in TROUBLESHOOTING / AI_SETUP (we only tested 2.43; 2.41/2.42 untested);
2. Add --nvcc <path> (or honour STRATA_NVCC) — the codebase already filters toolkits by version for the
   Pascal/Volta path (find_nvcc(below=(13, 0))), so a user-facing override fits the existing design.
   (CUDA_HOME is not read — only Windows-style CUDA_PATH — so on Linux there is currently no way in.)
3. ⚠️ The pinned llama.cpp already documents this exact failure and gives the fix:
   third_party/llama.cpp/docs/build.md §"Fixing Compatibility Issues with Old CUDA and New glibc" (L231–259)
   shows the same cospi / math_functions.h error and patches the CUDA headers to add noexcept (true)
   to cospi/cospif/sinpi/… Strata pins llama.cpp 3cf03257f219… (setup.py L92), so the conflict and its
   upstream-sanctioned workaround are already in the dependency tree — worth linking from TROUBLESHOOTING.