[BUG] Default device memory allocation is too aggressive

[JerryHOTS]

Software versions

Python      :  3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
Platform    :  Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
Legion      :  24.11.1 (commit: ac6aae07cb18fa9de978b073766dd9e3def29dbb)
Legate      :  24.11.1
[0 - 7f06e35b3600]    0.000000 {5}{module_config}: Module numa can not detect resources.
[0 - 7f06e35b3600]    0.000000 {4}{openmp}: numa support not found (or not working)
[0 - 7f06e35b3600]    0.000000 {5}{gpu}: /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/realm/cuda/cuda_module.cc(3985):CUDA_DRIVER_FNPTR(cuIpcGetMemHandle)(&alloc.ipc_handle, alloc.dev_ptr) = 2(CUDA_ERROR_OUT_OF_MEMORY): out of memory

(Following's the output for nvidia-smi btw)

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.72                 Driver Version: 566.14         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070 ...    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   48C    P8              1W /  120W |    7825MiB /   8188MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      3830      C   /python3.12                                 N/A      |
+-----------------------------------------------------------------------------------------+

Jupyter notebook / Jupyter Lab version

No response

Expected behavior

No error

Observed behavior

When I run legate-issue or run import cupynumeric as np in python it raises the following error:

[0 - 7f06e35b3600]    0.000000 {5}{module_config}: Module numa can not detect resources.
[0 - 7f06e35b3600]    0.000000 {4}{openmp}: numa support not found (or not working)
[0 - 7f06e35b3600]    0.000000 {5}{gpu}: /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/realm/cuda/cuda_module.cc(3985):CUDA_DRIVER_FNPTR(cuIpcGetMemHandle)(&alloc.ipc_handle, alloc.dev_ptr) = 2(CUDA_ERROR_OUT_OF_MEMORY): out of memory

Example code or instructions

legate-issue

import cupynumeric as np

Stack traceback or browser console output

No response

[manopapad]

Multiple issues here:

• Legate is attempting by default to reserve a large proportion of the device memory, and that's failing.
  • @JerryHOTS could you please try running with LEGATE_SHOW_CONFIG=1? The run will still fail, but we'll be able to see how much device memory Legate is attempting to reserve.
  • @JerryHOTS could you please try running with LEGATE_CONFIG="--fbmem 1000"? This will instruct Legate to try and reserve less memory.
  • @muraj looks like memory allocation is failing with CUDA_ERROR_OUT_OF_MEMORY, but in cuIpcGetMemHandle. Is it possible that there is, in fact, enough memory, but we're hitting a different system limit related to CUDA-IPC?
• Realm is complaining that we tried to instantiate NUMA-aligned memory, but that's not possible on the current system. This is not actually causing a failure in this case, but it is polluting the error message.
  • @eddy16112 is working on a Realm fix that would allow us to detect this issue before we put in our request for NUMA-aligned memory.
  • @JerryHOTS you can possibly make this go away by adding --omps 0 to LEGATE_CONFIG.
• legate-issue is not working. This is an outcome of the fact that legate-issue is itself trying to import cupynumeric, so if the issue is with initialization itself, then legate-issue is going to hit it too.
  • @bryevdv is looking into a fix.

[JerryHOTS]

Thank you very much for your prompt response!

Running with LEGATE_SHOW_CONFIG=1 gives: (terminal killed halfway)

Python      :  3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
Platform    :  Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
Legion      :  24.11.1 (commit: ac6aae07cb18fa9de978b073766dd9e3def29dbb)
Legate      :  24.11.1
Legate hardware configuration: --cpus=1 --gpus=1 --omps=1 --ompthreads=6 --utility=2 --sysmem=12677 --numamem=0 --fbmem=7778 --zcmem=128 --regmem=0
[0 - 7fc0c6d24600]    0.000000 {5}{module_config}: Module numa can not detect resources.
[0 - 7fc0c6d24600]    0.000000 {4}{openmp}: numa support not found (or not working)
[0 - 7fc0c6d24600]    0.000000 {5}{gpu}: /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/realm/cuda/cuda_module.cc(3985):CUDA_DRIVER_FNPTR(cuIpcGetMemHandle)(&alloc.ipc_handle, alloc.dev_ptr) = 2(CUDA_ERROR_OUT_OF_MEMORY): out of memory

Running with LEGATE_CONFIG="--fbmem 1000" gives: (successful run)

Python      :  3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
Platform    :  Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
Legion      :  24.11.1 (commit: ac6aae07cb18fa9de978b073766dd9e3def29dbb)
Legate      :  24.11.1
[0 - 7f67147be600]    0.000000 {5}{module_config}: Module numa can not detect resources.
[0 - 7f67147be600]    0.000000 {4}{openmp}: numa support not found (or not working)
cuPynumeric :  24.11.02
Numpy       :  1.26.4
Scipy       :  1.14.1
Numba       :  (failed to detect)
CTK package :  cuda-version-12.6-h7480c83_3 (conda-forge)
GPU driver  :  566.14
GPU devices :  
  GPU 0: NVIDIA GeForce RTX 4070 Laptop GPU

And setting --omps=0 doesn't solve the problem, with or without setting "--fbmem 1000":

Python      :  3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
Platform    :  Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
Legion      :  24.11.1 (commit: ac6aae07cb18fa9de978b073766dd9e3def29dbb)
Legate      :  24.11.1
Legate hardware configuration: --cpus=1 --gpus=1 --omps=0 --ompthreads=0 --utility=2 --sysmem=12677 --numamem=0 --fbmem=7778 --zcmem=128 --regmem=0
[0 - 7fb3d0369600]    0.000000 {5}{module_config}: Module numa can not detect resources.
[0 - 7fb3d0369600]    0.000000 {5}{gpu}: /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/realm/cuda/cuda_module.cc(3985):CUDA_DRIVER_FNPTR(cuIpcGetMemHandle)(&alloc.ipc_handle, alloc.dev_ptr) = 2(CUDA_ERROR_OUT_OF_MEMORY): out of memory

[manopapad]

OK, so it looks like indeed Legate's default device memory reservation was too aggressive. It tried reserving 7778MiB, based on the total memory size of 8188MiB. But based on your nvidia-smi output, it looks like only 7825MiB is actually available, i.e. quite close to what Legate requested (although it's not clear where the rest is going).

It's still curious that the error occurs at cuIpcGetMemHandle rather than cuMemAlloc.

There is an ongoing discussion about shifting our device memory allocation logic to happen dynamically through CUDA memory allocation calls, rather than allocating a pool in the beginning and allocating out of that. But there are multiple issues to work through, so it won't be available in the immediate term.

For now, I suggest you explicitly tell Legate how much memory to reserve, e.g. LEGATE_CONFIG="--fbmem 7000", or whatever the maximum number is that works.

[JerryHOTS]

Thank you again for your help! When it's convenient, I would appreciate it if you could let me know when the fix becomes available.

[manopapad]

The full fix is getting delayed. For now we will query the currently available memory at startup (using cuMemGetInfo), and allocate 90-95% of that. This will work around the common scenario where another process (typically a graphics display server) is already using up some of the GPU device memory. Assuming the other memory use remains constant for the duration of the cuPyNumeric run, we should be able to co-exist with the prior user.

[eddy16112]

The fix has been merged https://github.com/nv-legate/legate.internal/pull/2166.

[manopapad]

It will take a few days for the fix to be propagated onto the nightly packages, need to make some updates to cuPyNumeric to align with latest Legate. I'll notify when that's done.

[manopapad]

@JerryHOTS as of 25.05.00.dev43 we should be taking current device memory into account at startup. Could you please confirm that this is working for you?

[manopapad]

@marcinz while we wait for @JerryHOTS to test the fix, I know you have had the same issue in the past, can you confirm the latest nightlies fix this for you?

[JerryHOTS]

@manopapad Could you help me with this problem I encountered after installation? I tried both conda legate and pip, but when I run import cupynumeric as np, i get the following

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[1], line 1
----> 1 import cupynumeric as np

File ~/miniconda3/envs/cupy/lib/python3.12/site-packages/cupynumeric/__init__.py:29
     25 from __future__ import annotations
     27 import numpy as _np
---> 29 from . import linalg, random, fft, ma
     30 from ._array.array import ndarray
     31 from ._array.util import maybe_convert_to_np_ndarray

File ~/miniconda3/envs/cupy/lib/python3.12/site-packages/cupynumeric/linalg/__init__.py:19
     15 from __future__ import annotations
     17 import numpy.linalg as _nplinalg
---> 19 from .._array.util import maybe_convert_to_np_ndarray
     20 from .._utils.coverage import clone_module
     21 from .linalg import *

File ~/miniconda3/envs/cupy/lib/python3.12/site-packages/cupynumeric/_array/util.py:32
     20 from typing import (
     21     TYPE_CHECKING,
     22     Any,
   (...)     27     cast,
     28 )
     30 import numpy as np
---> 32 from ..runtime import runtime
     33 from ..types import NdShape
     35 if TYPE_CHECKING:

File ~/miniconda3/envs/cupy/lib/python3.12/site-packages/cupynumeric/runtime.py:22
     19 from functools import lru_cache, reduce
     20 from typing import TYPE_CHECKING, Any, Literal, Sequence, TypeGuard
---> 22 import legate.core.types as ty
     23 import numpy as np
     24 from legate.core import LEGATE_MAX_DIM, Scalar, TaskTarget, get_legate_runtime

File ~/miniconda3/envs/cupy/lib/python3.12/site-packages/legate/core/__init__.py:144
    141 _np.random.seed = _warn_seed(_np.random.seed)
    142 _random.seed = _warn_seed(_random.seed)
--> 144 get_legate_runtime()  # Starts the runtime

File runtime.pyx:1103, in legate.core._lib.runtime.runtime.get_legate_runtime()

File runtime.pyx:1114, in legate.core._lib.runtime.runtime.get_legate_runtime()

File runtime.pyx:1086, in legate.core._lib.runtime.runtime.initialize()

RuntimeError: LEGATE ERROR: ================================================================================
LEGATE ERROR: System: Linux, 5.15.167.4-microsoft-standard-WSL2, DESKTOP-GHP1H7N, #1 SMP Tue Nov 5 00:21:55 UTC 2024, x86_64
LEGATE ERROR: Legate version: 25.3.2 (75dc0a92bbd2dfb79b6b680a0f37cbd0370d0181)
LEGATE ERROR: Legion version: 25.3.0 (04b7d5068c5e75f29684703e8a1b8568b3e59b9a)
LEGATE ERROR: Configure options: --LEGATE_ARCH=arch-conda --with-python --with-cc=/tmp/conda-croot/legate/_build_env/bin/x86_64-conda-linux-gnu-cc --with-cxx=/tmp/conda-croot/legate/_build_env/bin/x86_64-conda-linux-gnu-c++ --build-march=x86-64 --legion-max-dim=6 --with-openmp --with-cuda --with-cal --build-type=release --with-ucx
LEGATE ERROR: Exception stack contains 1 exception(s) (bottom-most exception first):
LEGATE ERROR:
LEGATE ERROR: #0 legate::AutoConfigurationError: Legate auto-configuration failed: Not enough CPU cores to split across 1 OpenMP processor(s). Have 1, but need 1 for CPU processors, 2 for utility processors, 1 for GPU processors, and at least 1 for OpenMP processors (1 core each). Use LEGATE_CONFIG to set configuration parameters manually, and/or disable automatic configuration with LEGATE_AUTO_CONFIG=0.
LEGATE ERROR: Stack trace (most recent call first):
LEGATE ERROR: #0   0x00007fcc98190ead at /home/jerryhts/miniconda3/envs/cupy/lib/python3.12/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.03.02
LEGATE ERROR: #1   0x00007fcc984ca6f1 at /home/jerryhts/miniconda3/envs/cupy/lib/python3.12/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.03.02
LEGATE ERROR: #2   0x00007fcc9849cb34 at /home/jerryhts/miniconda3/envs/cupy/lib/python3.12/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.03.02
LEGATE ERROR: #3   0x00007fcc98469d11 at /home/jerryhts/miniconda3/envs/cupy/lib/python3.12/site-packages/legate/core/_lib/data/../../../../../../liblegate.so.25.03.02
LEGATE ERROR: #4   0x00007fcbd7330392 at /home/jerryhts/miniconda3/envs/cupy/lib/python3.12/site-packages/legate/core/_lib/runtime/runtime.cpython-312-x86_64-linux-gnu.so
LEGATE ERROR: #5   0x00007fcbd7331313 at /home/jerryhts/miniconda3/envs/cupy/lib/python3.12/site-packages/legate/core/_lib/runtime/runtime.cpython-312-x86_64-linux-gnu.so
LEGATE ERROR: #6   (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #7   0x000055e7172b5f1d in PyObject_Vectorcall at /usr/local/src/conda/python-3.12.10/Objects/call.c:325:12
LEGATE ERROR: #8   0x000055e7171b6b4e in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2715:19
LEGATE ERROR: #9   0x000055e7173544f0 in PyEval_EvalCode at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
LEGATE ERROR: #10  (inlined)          in builtin_exec_impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:1093:0
LEGATE ERROR: #11  0x000055e717370c8b in builtin_exec at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:543:0
LEGATE ERROR: #12  0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #13  (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #14  0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #15  (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #16  0x000055e7172bfa1d in object_vacall at /usr/local/src/conda/python-3.12.10/Objects/call.c:850:14
LEGATE ERROR: #17  0x000055e7172e8bef in PyObject_CallMethodObjArgs at /usr/local/src/conda/python-3.12.10/Objects/call.c:911:0
LEGATE ERROR: #18  (inlined)          in import_find_and_load at /usr/local/src/conda/python-3.12.10/Python/import.c:2793:0
LEGATE ERROR: #19  0x000055e7172e7c2c in PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.12.10/Python/import.c:2876:0
LEGATE ERROR: #20  (inlined)          in builtin___import___impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:276:0
LEGATE ERROR: #21  0x000055e7172f7ec5 in builtin___import__ at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:107:0
LEGATE ERROR: #22  0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #23  (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #24  0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #25  (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #26  0x000055e7172bfa1d in object_vacall at /usr/local/src/conda/python-3.12.10/Objects/call.c:850:14
LEGATE ERROR: #27  0x000055e7172e8bef in PyObject_CallMethodObjArgs at /usr/local/src/conda/python-3.12.10/Objects/call.c:911:0
LEGATE ERROR: #28  (inlined)          in import_find_and_load at /usr/local/src/conda/python-3.12.10/Python/import.c:2793:0
LEGATE ERROR: #29  0x000055e7172e7c2c in PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.12.10/Python/import.c:2876:0
LEGATE ERROR: #30  (inlined)          in import_name at /usr/local/src/conda/python-3.12.10/Python/ceval.c:2510:15
LEGATE ERROR: #31  0x000055e7171c0ed9 in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2144:19
LEGATE ERROR: #32  0x000055e7173544f0 in PyEval_EvalCode at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
LEGATE ERROR: #33  (inlined)          in builtin_exec_impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:1093:0
LEGATE ERROR: #34  0x000055e717370c8b in builtin_exec at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:543:0
LEGATE ERROR: #35  0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #36  (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #37  0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #38  (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #39  0x000055e7172bfa1d in object_vacall at /usr/local/src/conda/python-3.12.10/Objects/call.c:850:14
LEGATE ERROR: #40  0x000055e7172e8bef in PyObject_CallMethodObjArgs at /usr/local/src/conda/python-3.12.10/Objects/call.c:911:0
LEGATE ERROR: #41  (inlined)          in import_find_and_load at /usr/local/src/conda/python-3.12.10/Python/import.c:2793:0
LEGATE ERROR: #42  0x000055e7172e7c2c in PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.12.10/Python/import.c:2876:0
LEGATE ERROR: #43  (inlined)          in import_name at /usr/local/src/conda/python-3.12.10/Python/ceval.c:2510:15
LEGATE ERROR: #44  0x000055e7171c0ed9 in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2144:19
LEGATE ERROR: #45  0x000055e7173544f0 in PyEval_EvalCode at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
LEGATE ERROR: #46  (inlined)          in builtin_exec_impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:1093:0
LEGATE ERROR: #47  0x000055e717370c8b in builtin_exec at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:543:0
LEGATE ERROR: #48  0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #49  (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #50  0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #51  (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #52  0x000055e7172bfa1d in object_vacall at /usr/local/src/conda/python-3.12.10/Objects/call.c:850:14
LEGATE ERROR: #53  0x000055e7172e8bef in PyObject_CallMethodObjArgs at /usr/local/src/conda/python-3.12.10/Objects/call.c:911:0
LEGATE ERROR: #54  (inlined)          in import_find_and_load at /usr/local/src/conda/python-3.12.10/Python/import.c:2793:0
LEGATE ERROR: #55  0x000055e7172e7c2c in PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.12.10/Python/import.c:2876:0
LEGATE ERROR: #56  (inlined)          in import_name at /usr/local/src/conda/python-3.12.10/Python/ceval.c:2510:15
LEGATE ERROR: #57  0x000055e7171c0ed9 in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2144:19
LEGATE ERROR: #58  0x000055e7173544f0 in PyEval_EvalCode at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
LEGATE ERROR: #59  (inlined)          in builtin_exec_impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:1093:0
LEGATE ERROR: #60  0x000055e717370c8b in builtin_exec at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:543:0
LEGATE ERROR: #61  0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #62  (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #63  0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #64  (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #65  0x000055e7172bfa1d in object_vacall at /usr/local/src/conda/python-3.12.10/Objects/call.c:850:14
LEGATE ERROR: #66  0x000055e7172e8bef in PyObject_CallMethodObjArgs at /usr/local/src/conda/python-3.12.10/Objects/call.c:911:0
LEGATE ERROR: #67  (inlined)          in import_find_and_load at /usr/local/src/conda/python-3.12.10/Python/import.c:2793:0
LEGATE ERROR: #68  0x000055e7172e7c2c in PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.12.10/Python/import.c:2876:0
LEGATE ERROR: #69  (inlined)          in builtin___import___impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:276:0
LEGATE ERROR: #70  0x000055e7172f7ec5 in builtin___import__ at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:107:0
LEGATE ERROR: #71  0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #72  (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #73  0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #74  (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #75  0x000055e7172bfa1d in object_vacall at /usr/local/src/conda/python-3.12.10/Objects/call.c:850:14
LEGATE ERROR: #76  0x000055e7172e8bef in PyObject_CallMethodObjArgs at /usr/local/src/conda/python-3.12.10/Objects/call.c:911:0
LEGATE ERROR: #77  0x000055e7172e7eab in PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.12.10/Python/import.c:2945:0
LEGATE ERROR: #78  (inlined)          in import_name at /usr/local/src/conda/python-3.12.10/Python/ceval.c:2510:15
LEGATE ERROR: #79  0x000055e7171c0ed9 in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2144:19
LEGATE ERROR: #80  0x000055e7173544f0 in PyEval_EvalCode at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
LEGATE ERROR: #81  (inlined)          in builtin_exec_impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:1093:0
LEGATE ERROR: #82  0x000055e717370c8b in builtin_exec at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:543:0
LEGATE ERROR: #83  0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #84  (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #85  0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #86  (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #87  0x000055e7172bfa1d in object_vacall at /usr/local/src/conda/python-3.12.10/Objects/call.c:850:14
LEGATE ERROR: #88  0x000055e7172e8bef in PyObject_CallMethodObjArgs at /usr/local/src/conda/python-3.12.10/Objects/call.c:911:0
LEGATE ERROR: #89  (inlined)          in import_find_and_load at /usr/local/src/conda/python-3.12.10/Python/import.c:2793:0
LEGATE ERROR: #90  0x000055e7172e7c2c in PyImport_ImportModuleLevelObject at /usr/local/src/conda/python-3.12.10/Python/import.c:2876:0
LEGATE ERROR: #91  (inlined)          in import_name at /usr/local/src/conda/python-3.12.10/Python/ceval.c:2510:15
LEGATE ERROR: #92  0x000055e7171c0ed9 in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2144:19
LEGATE ERROR: #93  0x000055e7173544f0 in PyEval_EvalCode at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
LEGATE ERROR: #94  (inlined)          in builtin_exec_impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:1093:0
LEGATE ERROR: #95  0x000055e717370c8b in builtin_exec at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:543:0
LEGATE ERROR: #96  0x000055e7171b73a6 in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2975:19
LEGATE ERROR: #97  (inlined)          in _PyEval_EvalFrame at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_ceval.h:89:16
LEGATE ERROR: #98  0x000055e717368b0e in gen_send_ex2 at /usr/local/src/conda/python-3.12.10/Objects/genobject.c:230:14
LEGATE ERROR: #99  0x000055e717369db6 in gen_send_ex at /usr/local/src/conda/python-3.12.10/Objects/genobject.c:274:9
LEGATE ERROR: #100 0x000055e7171b818a in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3094:19
LEGATE ERROR: #101 (inlined)          in _PyEval_EvalFrame at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_ceval.h:89:16
LEGATE ERROR: #102 (inlined)          in _PyEval_Vector at /usr/local/src/conda/python-3.12.10/Python/ceval.c:1685:12
LEGATE ERROR: #103 (inlined)          in _PyFunction_Vectorcall at /usr/local/src/conda/python-3.12.10/Objects/call.c:419:16
LEGATE ERROR: #104 0x000055e7172e93ab in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #105 0x000055e7172e8f0d in method_vectorcall at /usr/local/src/conda/python-3.12.10/Objects/classobject.c:61:18
LEGATE ERROR: #106 (inlined)          in _PyVectorcall_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:283:24
LEGATE ERROR: #107 0x000055e7172d129a in _PyObject_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:354:16
LEGATE ERROR: #108 (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #109 0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #110 (inlined)          in _PyEval_EvalFrame at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_ceval.h:89:16
LEGATE ERROR: #111 0x000055e717368b0e in gen_send_ex2 at /usr/local/src/conda/python-3.12.10/Objects/genobject.c:230:14
LEGATE ERROR: #112 0x00007fccbf58f273 in task_step_impl at /usr/local/src/conda/python-3.12.10/Modules/_asynciomodule.c:2869:22
LEGATE ERROR: #113 0x00007fccbf58fa62 in task_step at /usr/local/src/conda/python-3.12.10/Modules/_asynciomodule.c:3188:11
LEGATE ERROR: #114 0x000055e7172bf647 in cfunction_vectorcall_O at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:509:24
LEGATE ERROR: #115 0x000055e7173ead4b in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #116 0x000055e7172643b8 in context_run at /usr/local/src/conda/python-3.12.10/Python/context.c:668:0
LEGATE ERROR: #117 0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #118 (inlined)          in PyCFunction_Call at /usr/local/src/conda/python-3.12.10/Objects/call.c:387:12
LEGATE ERROR: #119 0x000055e7171b771b in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:3263:26
LEGATE ERROR: #120 0x000055e7173544f0 in PyEval_EvalCode at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
LEGATE ERROR: #121 (inlined)          in builtin_exec_impl at /usr/local/src/conda/python-3.12.10/Python/bltinmodule.c:1093:0
LEGATE ERROR: #122 0x000055e717370c8b in builtin_exec at /usr/local/src/conda/python-3.12.10/Python/clinic/bltinmodule.c.h:543:0
LEGATE ERROR: #123 0x000055e7172b604a in cfunction_vectorcall_FASTCALL_KEYWORDS at /usr/local/src/conda/python-3.12.10/Objects/methodobject.c:438:24
LEGATE ERROR: #124 (inlined)          in _PyObject_VectorcallTstate at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
LEGATE ERROR: #125 0x000055e7172b5f1d in PyObject_Vectorcall at /usr/local/src/conda/python-3.12.10/Objects/call.c:325:12
LEGATE ERROR: #126 0x000055e7171b6b4e in _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2715:19
LEGATE ERROR: #127 0x000055e717383ea7 in pymain_run_module at /usr/local/src/conda/python-3.12.10/Modules/main.c:300:0
LEGATE ERROR: #128 (inlined)          in pymain_run_python at /usr/local/src/conda/python-3.12.10/Modules/main.c:628:0
LEGATE ERROR: #129 0x000055e71738302c in Py_RunMain at /usr/local/src/conda/python-3.12.10/Modules/main.c:714:0
LEGATE ERROR: #130 0x000055e71733e5d6 in Py_BytesMain at /usr/local/src/conda/python-3.12.10/Modules/main.c:768:12
LEGATE ERROR: #131 0x00007fccc0a861c9 at /lib/x86_64-linux-gnu/libc.so.6
LEGATE ERROR: #132 0x00007fccc0a8628a at /lib/x86_64-linux-gnu/libc.so.6
LEGATE ERROR: #133 0x000055e71733e47c at /home/jerryhts/miniconda3/envs/cupy/bin/python3.12
LEGATE ERROR: ================================================================================

[manopapad]

@JerryHOTS could you please run with LEGATE_AUTO_CONFIG=0? It looks like only one core is being allocated to the Legate execution.

FYI @Jacobfaib it looks like the message with the actionable feedback might be getting lost inside the huge stacktrace in the error message. Should we perhaps print the exception string separately at the top of the message?