Repository navigation
Dask really slow towards the end of the job #2835
Description
Hi I am using dask distributed with 300 workers. I have a cluster of 32 machines with over 300 cpus and I want the computation to be distributed across all workers. Currently when i run my job, it works fairly well and distributes all the tasks, however towards the end, the workers stop working and there is only one or two that has a high cpu usage.
Generally I have a 8000 timeseries that I want to process and I want the computation to be distributed so I use dask delayed, and then call compute on the list.
predictions = nostradamus.run_prophet(data,
brands_list=brands_1
)
x = client.compute(predictions, retries=10)
wait(x)
ds =[i.result() for i in x] . # constructs a list of dataframes
I have the following images from the dashboard. I just don't understand why it stops distributing at the very end.
Any help would be appreciated!
Reacted by Zaharid and Alexander Kaigorodov
Activity
mrocklin commentedon Jul 12, 2019
You might try looking at what the workers are working on using the Client.call_stack method or looking in the worker pages in the "Info" tab of the dashboard.
As always, if you're able to provide a minimal reproducible example that's often a helpful approach. See this blog
ohashmi1 commentedon Jul 12, 2019
I look at the info tab but it just shows that none of them are processing anything. I am pretty new to dask, don't really know how to debug. Not sure what to look for in call stack. Could it because I persist dataframes in memory? The memory is never 100% used tho. It only happens at the very end, workers progressively do less work as I advance my computation.
There isnt a whole lot of code i can reproduce. It also only happens when i have a large dataset. It works fine on smaller sets.
mrocklin commentedon Jul 12, 2019
birdsarah commentedon Jul 26, 2019
I think I'm getting something similar. I ended up with workers that looked like below.
One worker has collected a whole bunch of tasks that are all in processing but not doing anything and then nothing else is processing.
Manually using client.retire_worker helped to unstick things for me.
There were definitely tasks on the queue for the worker that I retired that had been put there after another worker died as one I clicked on had been marked "suspicious: 1" which i believe happens when a task was running previously on a killedworker.
mrocklin commentedon Jul 26, 2019
mrocklin commentedon Jul 26, 2019
rbubley commentedon Jul 28, 2019
I’ve also had this, and in my case it seemed to be some sort of race condition in the scheduler. For me, it was fixed by moving the scheduler from toolz to cytoolz, which also seemed to imply it was some sort of timing issue.
birdsarah commentedon Jul 28, 2019
rbubley commentedon Jul 29, 2019
@birdsarah, it is literally as simple as a pip/conda install of cytoolz: distributed attempts to import cytoolz in preference to toolz.
(Various bits of the distributed library code uses this pattern:
)
birdsarah commentedon Jul 29, 2019
In a recent run with the same problem. I had about 450 jobs of 100k plus remaining and listed as ready but they weren't in any worker. Call to client.call_stack returned empty dict.
I discovered that, presumably due to some packaging conflicts, I had both toolz and cytoolz installed. I have removed toolz and am going to see what happens.
birdsarah commentedon Jul 29, 2019
Oh....okay so the option is toolz only or toolz and cytoolz. (dask won't run without plain toolz)
birdsarah commentedon Jul 30, 2019
Hold moly that seems to have done the trick. Have only got a couple of results and so am loathed to get too excited, but I think you might be on for hero of the week @rbubley.
@mrocklin I also observed the loads redistributing as you described once I removed cytoolz.
mrocklin commentedon Jul 30, 2019
Just so I'm clear, was there a mismatch in environments where some workers/clients/schedulers had cytoolz and some didn't, or were environments homogeneous before, and another homogeneous change in those environments helped restore things?
rbubley commentedon Jul 30, 2019
@mrocklin, in my case, they were homogeneous (no cytoolz on client/scheduler/worker) and I saw the problem; the problem was fixed when it became inhomogeneous (cytoolz only on scheduler). But my understanding is that cytoolz is supposed to have the same results as toolz (up to the documented return values; non-guaranteed orderings can change), and conceptually this shouldn't be a library where homogeneity is required. (Although something subtle around orderings is always possible.)
What we should expect is that changes between toolz and cytoolz affects timings, and could therefore surface previously unobserved race conditions. Given the observed effects, i.e. some workers being idle when they shouldn't be, I would guess that the issue is with the stealing code - perhaps some series of actions is assumed to be atomic when it isn't.
birdsarah commentedon Jul 30, 2019
Things were homogenous. What I meant was that on initial environment creation dependency management had meant that both toolz and cytoolz were created. That was then packaged with
conda packand shipped to all workers. When I removed cytoolz, and again repackaged withconda packand restarted DaskYarn cluster things improved.mrocklin commentedon Jul 30, 2019
So @rbubley 's situation improved when he added cytoolz and @birdsarah 's situation improved when she removed it?
birdsarah commentedon Jul 30, 2019
To be clear I couldn't test having only cytoolz in my environment. dask / daskyarn wouldn't import.
Edit: I couldn't readily test.
birdsarah commentedon Sep 13, 2019
I have been reliably removing cytoolz from my environment for a while.
Just to note that today I experienced this issue for the first time in a while, with cytoolz removed,. It may be unrelated, but thought I'd document. It was on
write_parquettasks which seem to have completed but not registered as such. When I retire the affected worker the job can finish with no apparent loss of data.nsmith- commentedon Aug 28, 2020
I'm not 100% sure if I am seeing the same thing discussed here, but often I see one or a handful of jobs stalled in "processing" on some number of workers. If I go to the task in the dashboard I see, e.g.
If I click the link to the call stack I see
Task not actively running. It may be finished or not yet startedwhich seems odd. If I retire the worker (e.g. in this case,client.retire_workers(["tls://10.128.2.162:40554"])), then the task gets unstuck and the job completes.Zaharid commentedon Dec 23, 2020
I am having the same problem.
Some task (of several similar ones launched with client.map so nothing particular) gets stuck as described above by others, while the worker or the scheduler is not using much cpu and trying to get the call stack reports the task as not running. Shutting the worker seems to help.
It happens quite frequently to the point that dask is hard to use without manual intervention. Is there any debugging information I can provide?
The environment I have in all my workers is the latest dask from conda forge, which installs cytoolz as a dependency.
Details
Zaharid commentedon Dec 23, 2020
After some investigation I think that what is happening is that some memory limit (80%?) is hit and that somehow causes tasks to be marked as started but not actually be processed. This can happen because of e.g. #3530.
Zaharid commentedon Jan 4, 2021
Here a way to reproduce the problem reliably on my machine:
Running the script above I see a continous stream of warnings
but I do not see them when the cluster is similarly deadlocked with the production code.
simone-codeluppi commentedon Mar 30, 2021
Hi
I have a similar issue as reported above.
I am processing 800-1000 3D microscopy images using
dask/dask-jobqueueon anhtcondormanaged cluster. Each 3D image is processed in parallel (embarrassingly parallel application). Each image is processed in a separatedjobmanaged by aworker. The min and max number ofjobs/workersis regulated by adaptive scaling (min, max). The expected processing time of a task is quite long (~1hr)The processing time of each task increases over time and usually the last tasks take hrs (~5-10hrs) to be completed or the processing stalls. The tasks that are run for longer time or stall are random (different images can be processed last). This is similar to what is reported here: #2835 but having
ctoolzdidn't solve the issueThe
task logfor this long running tasks shows that the task status cycle betweenwaiting --> runningtoprocessing-->releasedmany many times. The processing resources aren't an issue. In some cases an error related to connection to the scheduler is printed (see below) so it seems that the worker (still up in the dashboard) looses the connection to the scheduler (temporarily)It is difficult for me to provide a working minimal example because the issue happens only when the processing has been going on for quite a while (hrs) and not always after the same amount of time.
Any hint to what to look for in order to run a proper debugging will be helpful
Thanks.
jacobtomlinson commentedon Oct 18, 2024
This issue still requires more info but there has been no activity here for many years, so I'm going to close it out.