You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] CUDA extension failures terminate the Python process #9
The CUDA extension's CUCHECK and CUDACHECK macros print an error and call exit(EXIT_FAILURE). Any checked CUDA driver or runtime failure therefore
terminates the Python process from inside the library.
Python caller --> pybind binding --> CUDA error --> exit()
|
`--> process ends
The caller cannot catch the error, attach operation context, clean up its own
state, or coordinate an orderly distributed shutdown.
Reproduction
Pass nvl_multicast_import a valid POSIX file descriptor for /dev/null. The
descriptor is valid at the OS boundary but is not a CUDA-exported shareable
allocation, so the CUDA import call returns an error.
On an H100 NVL with origin/master, the child process printed the old macro's
CUDA error and exited with status 1. The parent regression failed because no
Python exception was catchable:
assert result.returncode == 0
actual returncode: 1
stderr: Failed: CUDA error ... 'invalid device ordinal'
This is a defined error path: it does not fabricate or reinterpret an opaque
CUDA allocation handle.
Expected behavior
Checked CUDA calls raise a Python-visible RuntimeError through pybind.
The message identifies the failed expression, symbolic error name, numeric
code, and CUDA description when available.
Partially acquired VMM resources are released while the C++ stack unwinds.
Successful allocation, mapping, and multicast ownership behavior remains
unchanged.
Some asynchronous CUDA failures can leave a CUDA context unusable, and the
application may still decide to terminate. That recovery policy should remain
with the caller rather than being enforced by a library macro for every CUDA
error.
changed the title [-]CUDA extension errors terminate the Python process instead of raising exceptions[/-][+][Bug] CUDA extension failures terminate the Python process[/+]on Jul 28, 2026
Activity
[-]CUDA extension errors terminate the Python process instead of raising exceptions[/-][+][Bug] CUDA extension failures terminate the Python process[/+]