Skip to content

Warning about scratch directories creating time taking too long #6639

Open

Description

@fjetter

In one of my latest stress tests I encountered a warning (windows, obviously) about the time it takes to create a scratch directory

C:\Miniconda3\envs\dask-distributed\lib\contextlib.py:142: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.

followed by a failing nanny start due to a timeout of 15s

2022-06-27 21:01:13,838 - distributed.utils_test - ERROR - Failed to start gen_cluster: TimeoutError: Nanny start timed out after 15s.; retrying
Traceback (most recent call last):
  File "d:\a\distributed\distributed\distributed\nanny.py", line 357, in start_unsafe
    response = await self.instantiate()
  File "d:\a\distributed\distributed\distributed\nanny.py", line 416, in instantiate
    result = await asyncio.wait_for(
  File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 432, in wait_for
    await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "d:\a\distributed\distributed\distributed\core.py", line 491, in start
    await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
  File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "d:\a\distributed\distributed\distributed\utils_test.py", line 1081, in async_fn
    s, ws = await start_cluster(
  File "d:\a\distributed\distributed\distributed\utils_test.py", line 924, in start_cluster
    await asyncio.gather(*workers)
  File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 650, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "d:\a\distributed\distributed\distributed\core.py", line 494, in start
    raise asyncio.TimeoutError(
asyncio.exceptions.TimeoutError: Nanny start timed out after 15s.

https://github.com/fjetter/distributed/runs/7077483712?check_suite_focus=true

cc @graingert

Activity

  1. fjetter commented on Jun 29, 2022

    @fjetter
    MemberAuthor

    I did a bit of digging about our directory failures on windows. There was a related change to pull in locket as a dependency instead of vendoring because of a windows problem, see #6166 Our version actually included the windows fix already so this was not the problem.

    I compared their version and ours. Modulo formatting, typing, pyupgrade, etc. this commit is the only diff between our version pr #6166 and locket 1.0 mwilliamson/locket.py@dafd24b
    can’t figure out if this is related but I have another CI stress running on https://github.com/fjetter/distributed/tree/stress_ci_win_only to see if this changes anything.

    what caught my interest is a comment of pitrou on a locket PR indicating that if a lock would not be properly released, we’d end up with a failure trying to remove a tmp directory which makes sense. might of course be unrelated
    mwilliamson/locket.py#8 (comment)

    might also relate to #6649

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions