Repository navigation
dask-streamz based on the actor interface #369
Description
The stateful actors of Dask closely match the design of streamz nodes.
If checkpoint/restart is part of the actor code, then a major deficit of the current actors model (loss of data when a worker goes down) would be circumvented.
Activity
martindurant commentedon Oct 28, 2020
@CJ-Wright , I bet you find this idea fascinating...
roveo commentedon Dec 21, 2020
Are actors stable enough to use? I've been thinking about making something that would continuously maintain the state for some pre-defined computation as new data comes in. Something like differential dataflow, but in python and less head-expolding.
Also, what happens to an actor if the client exits/crashes? Is the memory on workers released?
martindurant commentedon Dec 21, 2020
They are not, but could be
If the client stops, then yes, futures would be released, and actors should be cleaned up.