Repository navigation
Open
Description
In one of my latest stress tests I encountered a warning (windows, obviously) about the time it takes to create a scratch directory
C:\Miniconda3\envs\dask-distributed\lib\contextlib.py:142: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
followed by a failing nanny start due to a timeout of 15s
2022-06-27 21:01:13,838 - distributed.utils_test - ERROR - Failed to start gen_cluster: TimeoutError: Nanny start timed out after 15s.; retrying
Traceback (most recent call last):
File "d:\a\distributed\distributed\distributed\nanny.py", line 357, in start_unsafe
response = await self.instantiate()
File "d:\a\distributed\distributed\distributed\nanny.py", line 416, in instantiate
result = await asyncio.wait_for(
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 432, in wait_for
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "d:\a\distributed\distributed\distributed\core.py", line 491, in start
await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "d:\a\distributed\distributed\distributed\utils_test.py", line 1081, in async_fn
s, ws = await start_cluster(
File "d:\a\distributed\distributed\distributed\utils_test.py", line 924, in start_cluster
await asyncio.gather(*workers)
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 650, in _wrap_awaitable
return (yield from awaitable.__await__())
File "d:\a\distributed\distributed\distributed\core.py", line 494, in start
raise asyncio.TimeoutError(
asyncio.exceptions.TimeoutError: Nanny start timed out after 15s.https://github.com/fjetter/distributed/runs/7077483712?check_suite_focus=true
cc @graingert
Metadata
Metadata
Assignees
Labels
No labels
Activity
fjetter commentedon Jun 29, 2022
I did a bit of digging about our directory failures on windows. There was a related change to pull in locket as a dependency instead of vendoring because of a windows problem, see #6166 Our version actually included the windows fix already so this was not the problem.
I compared their version and ours. Modulo formatting, typing, pyupgrade, etc. this commit is the only diff between our version pr #6166 and locket 1.0 mwilliamson/locket.py@dafd24b
can’t figure out if this is related but I have another CI stress running on https://github.com/fjetter/distributed/tree/stress_ci_win_only to see if this changes anything.
what caught my interest is a comment of pitrou on a locket PR indicating that if a lock would not be properly released, we’d end up with a failure trying to remove a tmp directory which makes sense. might of course be unrelated
mwilliamson/locket.py#8 (comment)
might also relate to #6649