Repository navigation
Release 2021.10.0 #189
Description
With our usual two week release cadence, we would normally release tomorrow. However, @fjetter has been able to identify several fixes which should resolve the cluster deadlock issue flagged right before the last release (xref #182) and was also reported by another user in dask/distributed#5366:
- Resolve work stealing deadlock caused by race in move_task_confirm distributed#5379
- Fix a race condition which would allow a rescheduled task to be reported missing even though it is not distributed#5160
- Fix regression where unknown tasks were allowed to be stolen distributed#5392
I'd prefer to bump the 2021.10.0 release to next Friday, October 15, to give us time to get those fixes in and confirm they indeed resolve dask/distributed#5366.
Additionally, since the last release we've merged a large refactor to the distributed worker state machine (xref dask/distributed#5046) which resulted in some follow-up work (dask/distributed#5316). @fjetter @crusaderky, checking in, is there any other follow-up work related to the worker state refactor we should prioritize before releasing, or are we okay on that front?
@quasiben you mentioned on the community call earlier today that RAPIDS is in a code freeze and a release is planned for today. I suspect this means bumping the dask + distributed release back a week is fine, but wanted to double check.
Activity
jakirkham commentedon Oct 7, 2021
Thanks James! 😄
Not seeing any issue with postponing to next Friday, but will make sure others know and raise any concerns here
For RAPIDS 21.10, which is coming out soon, we are pinning to Dask + Distributed 2021.9.1. So the Dask + Distributed release shouldn't impact that
Ben please feel free to correct me on any of this 🙂
crusaderky commentedon Oct 11, 2021
None on my side
jrbourbeau commentedon Oct 15, 2021
We've merged all current deadlock-related PRs, which seemed to have helped with the deadlock report offline, but unfortunately doesn't fully resolve the issue (there was a subsequent deadlock that took longer to trigger). There was a publicly reported deadlock issue, which we think is similar to the offline report (xref dask/distributed#5366). I've commented here dask/distributed#5366 (comment) to see if they're still encountering cluster deadlocking behavior with the latest
mainbranch ofdistributed.In order to not release a version of
distributedwhich is known to deadlock, I'll suggest we bump releasing back another week. @jakirkham @quasiben is there any issue with this on your end?jakirkham commentedon Oct 15, 2021
Makes sense. Thanks for the update James. No issues on our end :)
fjetter commentedon Oct 22, 2021
The deadlocks led me to dask/distributed#5426 which seemed to fix the problems which were originally reported by a power user.
There is still an open issue about a deadlock but that appears to already affecting the current stable version
2021.09.01dask/distributed#5366jrbourbeau commentedon Oct 22, 2021
Thanks for all your efforts on this @fjetter. I'll suggest that, since dask/distributed#5366 was already reported with a released version of
distributed, we merge dask/distributed#5426, which is a known improvement, and releasejakirkham commentedon Oct 22, 2021
@pentschev mentioned this morning that there were some changes that were causing us some issues in RAPIDS. Peter are all of those fixed or are there outstanding issues that we should be addressing before releasing?
pentschev commentedon Oct 22, 2021
They were just some minor changes in dask/distributed#5438 and dask/distributed#5446 that changed default/previous behavior so broke our tests, but nothing critical and both have been addressed in rapidsai/dask-cuda#757 and rapidsai/dask-cuda#758, respectively. Therefore, I think we're good to the extent I'm aware of. Thanks @jakirkham for the ping.
jrbourbeau commentedon Oct 22, 2021
Thanks for flagging those
dask-cudaissues @jakirkham and @pentschev for resolving them downstream.dask/distributed#5426 has been merged so I think we're in good shape to push out the 2021.10.0 release. I'll plan to push the release out in around an hour and will, as usual, ping this issue when I start that process.
jakirkham commentedon Oct 22, 2021
Sounds good. Have mentioned internally as well. Will let you know if anything comes up. Otherwise think we should be good to go
jrbourbeau commentedon Oct 22, 2021
🚀