UCX tests broken after distributed.utils_test.cleanup import removed

@fjetter

After #4888 which removed from distributed.utils_test import cleanup from files, the UCX test now fails as below:

________________________________________________________ ERROR at setup of test_large_cupy[1000000000] _________________________________________________________file /datasets/pentschev/miniconda3/envs/ucx-1.9-112-21.08.210608/lib/python3.8/site-packages/distributed/comm/tests/test_ucx.py, line 228
  @pytest.mark.slow
  @pytest.mark.asyncio
  @pytest.mark.parametrize("n", [int(1e9), int(2.5e9)])
  async def test_large_cupy(n, cleanup):
      cupy = pytest.importorskip("cupy")
      com, serv_com = await get_comm_pair()

      arr = cupy.ones(n, dtype="u1")                                                                                                                                  msg = {"op": "ping", "data": to_serialize(arr)}

      _, result = await asyncio.gather(com.write(msg), serv_com.read())                                                                                               data2 = result.pop("data")

      assert result["op"] == "ping"                                                                                                                                   assert len(data2) == len(arr)
      await com.close()
      await serv_com.close()                                                                                                                                    E       fixture 'cleanup' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, event_loop, monkeypatch, pytestconfig, record_property,
record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, unused_tcp_port, unused_tcp_port_factory          >       use 'pytest --fixtures [testpath]' for help on them.

/datasets/pentschev/miniconda3/envs/ucx-1.9-112-21.08.210608/lib/python3.8/site-packages/distributed/comm/tests/test_ucx.py:228

This is a file which isn't tested by Distributed's CI currently, which is supposedly the reason the error above wasn't catched. However, it seems that https://github.com/dask/distributed/pull/4888/files#diff-a31c7ed5d35f5ed8233994868c54d625b18e6bacb6794344c4531e62bd9dde59R19-R20 should make the fixture available for all tests, I'm not sure though why that isn't the case for that file in particular.

@fjetter do you happen to have any ideas how to resolve the above without adding the import back? I verified that if I add the import back it then works.

Hmm interesting. Looking at the CI logs from rapidsai/ucx-py#750, it looks like the rootdir reported by pytest

rootdir: /opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/comm/tests

is pointing to a directory within distributed. Maybe the conftest.py file in the root of the distributed source, which contains the cleanup fixuture, isn't being picked up by pytest?

pytest has a --rootdir=<path> option to manually specify the rootdir. Could you try setting that to the root of the distributed source to see if that helps?

Indeed you're right, thanks for checking this. I'm not very experienced with more advanced uses of pytest so didn't realize this could be the issue.

Using --rootdir doesn't seem to work unfortunately:

$ py.test --rootdir=/datasets/pentschev/src/distributed --cache-clear -vs `python -c "import distributed.comm.tests.test_ucx as m;print(m.__file__)"`
===================================================================== test session starts ======================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/bin/python
cachedir: .pytest_cache
rootdir: /datasets/pentschev/src/distributed
plugins: asyncio-0.15.1, anyio-3.1.0
collected 0 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 620, in perform_collect
INTERNALERROR>     rep = collect_one_node(self)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/runner.py", line 456, in collect_one_node
INTERNALERROR>     ihook = collector.ihook
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/nodes.py", line 195, in ihook
INTERNALERROR>     return self.session.gethookproxy(self.fspath)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 519, in gethookproxy
INTERNALERROR>     my_conftestmodules = pm._getconftestmodules(
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/config/__init__.py", line 536, in _getconftestmodules
INTERNALERROR>     mod = self._importconftest(conftestpath, importmode)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/config/__init__.py", line 576, in _importconftest
INTERNALERROR>     self._check_non_top_pytest_plugins(mod, conftestpath)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/config/__init__.py", line 607, in _check_non_top_pytest_plugins
INTERNALERROR>     fail(msg.format(conftestpath, self._confcutdir), pytrace=False)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/outcomes.py", line 153, in fail
INTERNALERROR>     raise Failed(msg=msg, pytrace=pytrace)
INTERNALERROR> Failed: Defining 'pytest_plugins' in a non-top-level conftest is no longer supported:
INTERNALERROR> It affects the entire test suite instead of just below the conftest as expected.
INTERNALERROR>   /datasets/pentschev/src/distributed/conftest.py
INTERNALERROR> Please move it to a top level conftest file at the rootdir:
INTERNALERROR>   None
INTERNALERROR> For more information, visit:
INTERNALERROR>   https://docs.pytest.org/en/stable/deprecations.html#pytest-plugins-in-non-top-level-conftest-files

==================================================================== no tests ran in 0.02s =====================================================================

But assuming it did work, it would still require users to download distributed's source to run any of the pytests that depend on distributed.utils_test.cleanup as conftest.py isn't shipped, so from a packaging perspective this seems like a bug to me (having tests shipped that can't run without the source). Is that a wrong assumption from my part perhaps?

conftest.py isn't shipped

Good catch. If we stick with keeping plugins in the root conftest.py, we should include that file in distributed/MANIFEST.in

Using --rootdir doesn't seem to work unfortunately:

Can you try commenting out this line locally?

distributed/conftest.py

Line 37 in ee06a44

pytest_plugins = ["distributed.pytest_resourceleaks"]

I see a different error now:

$ py.test --rootdir=/datasets/pentschev/src/distributed --cache-clear -vs `python -c "import distributed.comm.tests.test_ucx as m;print(m.__file__)"`
===================================================================== test session starts ======================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/bin/python
cachedir: .pytest_cache
rootdir: /datasets/pentschev/src/distributed
plugins: asyncio-0.15.1, anyio-3.1.0
collected 0 items / 1 error
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1463, in getoption
INTERNALERROR>     val = getattr(self.option, name)
INTERNALERROR> AttributeError: 'Namespace' object has no attribute 'runslow'
INTERNALERROR>
INTERNALERROR> The above exception was the direct cause of the following exception:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/main.py", line 637, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/datasets/pentschev/src/distributed/conftest.py", line 28, in pytest_collection_modifyitems
INTERNALERROR>     if config.getoption("--runslow"):
INTERNALERROR>   File "/datasets/pentschev/miniconda3/envs/ucx-master-112-21.08.210608/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1474, in getoption
INTERNALERROR>     raise ValueError(f"no option named {name!r}") from e
INTERNALERROR> ValueError: no option named 'runslow'

================================================================= 2 warnings, 1 error in 0.04s =================================================================

I'm not sure, maybe I'm using it incorrectly? Also, on the link you posted before for --rootdir there's no explicit mention to conftest.py, so perhaps it's not intended to be used with that file, but only with config files such as pytest.ini?

@jakirkham

This feels like a step in the right direction as pytest has at least found /datasets/pentschev/src/distributed/conftest.py which contains the missing fixtures. The --runslow option is also specific to distributed, but is added in the same root conftest.py file

distributed/conftest.py

Lines 23 to 24 in ddec3c9

    
           def pytest_addoption(parser): 
        
               parser.addoption("--runslow", action="store_true", help="run slow tests")

So I'm not sure why we're getting ValueError: no option named 'runslow'. Perhaps @jakirkham has some thoughts on some good next steps to try?

Yes, it has found it, I'm wondering whether it expects to do so or it has found it but isn't supposed to support it, and thus a bug in one direction or another.

so from a packaging perspective this seems like a bug to me (having tests shipped that can't run without the source). Is that a wrong assumption from my part perhaps?

I agree but frankly I was always wondering why we were shipping tests. If this is what we intend to do we should also add tests for this. anything wrong with checking out source?

So I'm not sure why we're getting ValueError: no option named 'runslow'

While it looks the same, fixture discovery and plugin discovery works slightly differently even though it's all based in conftest.py, see https://docs.pytest.org/en/6.2.x/writing_plugins.html#plugin-discovery-order-at-tool-startup and https://docs.pytest.org/en/6.2.x/reference.html#pytest.hookspec.pytest_addoption
I assume your particular way of calling things is now treating the root conftest.py as some sort deeply nested file and the docs for addoption already say that this is impossible. There is also a reference in https://docs.pytest.org/en/6.2.x/customize.html#rootdir

The --rootdir=path command-line option can be used to force a specific directory. Note that contrary to other command-line options, --rootdir cannot be used with addopts inside pytest.ini because the rootdir is used to find pytest.ini already.

that config options are not possible using this but that isn't quite what's happening for us but maybe its related

good next steps to try?

My preference would be to simply not ship tests but there are probably good reasons for doing so I am not aware of.

Otherwise we could try to move all fixture and plugin code to a proper plugin (see https://docs.pytest.org/en/6.2.x/writing_plugins.html#making-your-plugin-installable-by-others). Effectively registering distributed.utils_test + addoptions as a pytest plugin and allow others to call it. That would allow us to remove the asterisk import as well

INTERNALERROR> Failed: Defining 'pytest_plugins' in a non-top-level conftest is no longer supported:
INTERNALERROR> It affects the entire test suite instead of just below the conftest as expected.
INTERNALERROR> /datasets/pentschev/src/distributed/conftest.py
INTERNALERROR> Please move it to a top level conftest file at the rootdir:

There it is. See my above comment about "nested conftest" files.

BTW is this something you can reproduce locally or is this only happening on CI?

I agree but frankly I was always wondering why we were shipping tests. If this is what we intend to do we should also add tests for this. anything wrong with checking out source?

I could check the source, there's no reason why we can't, it was more of a question as to what is the intended goal. If the tests are shipped, then I believe it should be ensured that they work, otherwise not shipping them at all seems fine.

My preference would be to simply not ship tests but there are probably good reasons for doing so I am not aware of.

Otherwise we could try to move all fixture and plugin code to a proper plugin (see https://docs.pytest.org/en/6.2.x/writing_plugins.html#making-your-plugin-installable-by-others). Effectively registering distributed.utils_test + addoptions as a pytest plugin and allow others to call it. That would allow us to remove the asterisk import as well

Either alternative is fine by me, as long as we know what is appropriate for use.

BTW is this something you can reproduce locally or is this only happening on CI?

I can reproduce locally, also reproducible with other tests that don't require UCX, e.g.:

$ py.test --cache-clear -vs `python -c "import distributed.cli.tests.test_dask_spec as m;print(m.__file__)"`
===================================================================== test session starts ======================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /datasets/pentschev/miniconda3/envs/ucx-1.9-112-21.08.210608/bin/python
cachedir: .pytest_cache
rootdir: /datasets/pentschev
plugins: asyncio-0.15.1, anyio-3.1.0
collected 3 items

../../../miniconda3/envs/ucx-1.9-112-21.08.210608/lib/python3.8/site-packages/distributed/cli/tests/test_dask_spec.py::test_text ERROR
../../../miniconda3/envs/ucx-1.9-112-21.08.210608/lib/python3.8/site-packages/distributed/cli/tests/test_dask_spec.py::test_file ERROR
../../../miniconda3/envs/ucx-1.9-112-21.08.210608/lib/python3.8/site-packages/distributed/cli/tests/test_dask_spec.py::test_errors PASSED

============================================================================ ERRORS ============================================================================
_________________________________________________________________ ERROR at setup of test_text __________________________________________________________________
file /datasets/pentschev/miniconda3/envs/ucx-1.9-112-21.08.210608/lib/python3.8/site-packages/distributed/cli/tests/test_dask_spec.py, line 11
  @pytest.mark.skipif(COMPILED, reason="Fails with cythonized scheduler")
  @pytest.mark.asyncio
  async def test_text(cleanup):
      with popen(
          [
              sys.executable,
              "-m",
              "distributed.cli.dask_spec",
              "--spec",
              '{"cls": "dask.distributed.Scheduler", "opts": {"port": 9373}}',
          ]
      ) as sched:
          with popen(
              [
                  sys.executable,
                  "-m",
                  "distributed.cli.dask_spec",
                  "tcp://localhost:9373",
                  "--spec",
                  '{"cls": "dask.distributed.Worker", "opts": {"nanny": false, "nthreads": 3, "name": "foo"}}',
              ]
          ) as w:
              async with Client("tcp://localhost:9373", asynchronous=True) as client:
                  await client.wait_for_workers(1)
                  info = await client.scheduler.identity()
                  [w] = info["workers"].values()
                  assert w["name"] == "foo"
                  assert w["nthreads"] == 3
E       fixture 'cleanup' not found
>       available fixtures: anyio_backend, anyio_backend_name, anyio_backend_options, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, event_loop, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, unused_tcp_port, unused_tcp_port_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/datasets/pentschev/miniconda3/envs/ucx-1.9-112-21.08.210608/lib/python3.8/site-packages/distributed/cli/tests/test_dask_spec.py:11

@mrocklin

My preference would be to simply not ship tests but there are probably good reasons for doing so I am not aware of.

That does seem like the simpler option. Perhaps @mrocklin @jcrist or @jakirkham know why we ship tests?

Historically some projects encouraged users to run tests to make sure that things worked. Something like the following:

import numpy
numpy.test()

This predated widespread use of tools like pytest though.

I think that it's fine to not ship tests.

@pentschev

@pentschev @madsbk it looks like rapidsai/dask-cuda#658 updated dask-cuda's CI to checkout the distributed source. Is it okay to close this issue?

Peter is currently OOTO. Let's wait until he's back

it looks like rapidsai/dask-cuda#658 updated dask-cuda's CI to checkout the distributed source. Is it okay to close this issue?

We updated our CI to indeed resolve this by cloning the distributed repo and running tests from there, so from our side the issue was resolved. However, do we have a resolution on what distributed will do moving forward, will it stop shipping tests? I think it would be good to properly deal with the tests (even if that means not shipping them anymore) to prevent future users from seeing the same issue.

@jrbourbeau

@jrbourbeau do you have any thoughts here?

@jakirkham

Yeah, not shipping tests sounds reasonable to me. I opened up #5054 to update our MANIFEST.in accordingly. @jakirkham if you get a chance, any feedback you may have on that PR is welcome

@pentschev

Now that PR ( #5054 ) is in, are we good to close? Or is there anything else needed here @pentschev?

@pentschev

I think this is good to close, though @pentschev let me know if there's anything else to do here

@jakirkham

Yup, all good from my side. Thanks @jakirkham and @jrbourbeau !

This is unfortunate, since versioneer is totally busted from GitHub tarballs. Can tests be restored in sdists, but not installed?

Navigation Menu

Repository navigation

UCX tests broken after `distributed.utils_test.cleanup` import removed #4902

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def pytest_addoption(parser):
	parser.addoption("--runslow", action="store_true", help="run slow tests")

UCX tests broken after distributed.utils_test.cleanup import removed #4902

Description

Activity

jrbourbeau commented on Jun 10, 2021

jrbourbeau commented on Jun 10, 2021

pentschev commented on Jun 10, 2021

jrbourbeau commented on Jun 10, 2021

pentschev commented on Jun 10, 2021

jrbourbeau commented on Jun 10, 2021

pentschev commented on Jun 10, 2021

fjetter commented on Jun 11, 2021

fjetter commented on Jun 11, 2021

pentschev commented on Jun 11, 2021

jrbourbeau commented on Jun 11, 2021

mrocklin commented on Jun 14, 2021

jrbourbeau commented on Jun 23, 2021

jakirkham commented on Jun 23, 2021

pentschev commented on Jun 28, 2021

jakirkham commented on Jul 13, 2021

jrbourbeau commented on Jul 13, 2021

jakirkham commented on Jul 22, 2021

jrbourbeau commented on Jul 22, 2021

pentschev commented on Jul 22, 2021

QuLogic commented on Mar 6, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

UCX tests broken after `distributed.utils_test.cleanup` import removed #4902