You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Indeed, a dask client requires an event loop. It is possible to supply an event loop instance at client creation, and I don't know exactly what it required of that loop instance - probably anything asyncio compliant. However, dask's loop can also run in a separate thread, so maybe it is possible to have two loops going (which may get complicated when actually dealing with dask-streamz).
Have you tried using the anyIO (or other) loop for dask too? That would be the favourite solution if it works. Otherwise, I think I do basically agree that streamz should work regardless, but in practice, it may be tricky to demarcate dask-free parts of the code.
There is no AnyIO implementation for dask.
There's an AsyncIO implementation and a Tornado implementation.
I have never used dask so no on both accounts.
It might be possible to run dask with trio-asyncio but I haven't really checked.
True, but being able to use Dask is important too. If you can implement anyIO (or something else) for the non-Dask code without greatly complicating the code, we would consider it. Of course we would still prefer smooth interaction between streamz and Dask on any io-loop.
Activity
CJ-Wright commentedon Mar 30, 2020
What would that entail?
CJ-Wright commentedon Mar 30, 2020
Also how would that interact with dask?
martindurant commentedon Mar 30, 2020
Indeed, a dask client requires an event loop. It is possible to supply an event loop instance at client creation, and I don't know exactly what it required of that loop instance - probably anything asyncio compliant. However, dask's loop can also run in a separate thread, so maybe it is possible to have two loops going (which may get complicated when actually dealing with dask-streamz).
thedrow commentedon Mar 30, 2020
It's possible but not necessary if you aren't using dask.
martindurant commentedon Mar 30, 2020
Have you tried using the anyIO (or other) loop for dask too? That would be the favourite solution if it works. Otherwise, I think I do basically agree that streamz should work regardless, but in practice, it may be tricky to demarcate dask-free parts of the code.
thedrow commentedon Mar 30, 2020
There is no AnyIO implementation for dask.
There's an AsyncIO implementation and a Tornado implementation.
I have never used dask so no on both accounts.
It might be possible to run dask with
trio-asynciobut I haven't really checked.I want to implement streamz on another framework.
thedrow commentedon Mar 30, 2020
This framework should not tie itself to a specific event loop.
What if the next best thing comes around and everybody wants to use it?
martindurant commentedon Mar 30, 2020
True, but being able to use Dask is important too. If you can implement anyIO (or something else) for the non-Dask code without greatly complicating the code, we would consider it. Of course we would still prefer smooth interaction between streamz and Dask on any io-loop.
thedrow commentedon Mar 30, 2020
If you'd like to integrate Dask with other event loops I suggest taking it upstream.
I took a look at the code. It is currently impossible since you are tightly coupled with the Tornado event loop.
martindurant commentedon Mar 30, 2020
Would uvloop, which is asyncio compatible, be a solution for you?
dhirschfeld commentedon Mar 31, 2020
xref: python-trio/trio-asyncio#22 (comment)
thedrow commentedon Mar 31, 2020
No. I need trio support.
RiverAi7z commentedon Sep 17, 2024
Could you please give some example of using python default asynicio and uvloop?
martindurant commentedon Sep 17, 2024
https://streamz.readthedocs.io/en/latest/async.html has asyncio examples and tornado syntax too.
To use a uvloop, you can make it the default one with something like
and all further use can be the same. I don't know if there's any further configuration or control you can do with it.