[madsbk]

Warning, this is a prototype and very hacky!

This PR introduces tasks that can generate new tasks on-the-fly.
Please see dask/dask#6173 for details.

[mrocklin]

Interesting approach!

As a warning, the scheduler generally doesn't run user Python code. This is what allows us to consider rewriting it in another language, and is also often critical for security in many deployments.

So as a prototype this is really interesting, but as something that would ever get merged in this seems unlikely to me. If you ever felt like sharing your thoughts on an issue somewhere I suspect that you would get good feedback from the community. That might give you more information on how to progress on this issue.

[madsbk]

@mrocklin, yes this is primarily to investigate how far we can get with a task generating approach. I will write an issue when we know its potential.

Regarding Python code on the scheduler, we can just as easy generate the tasks at the client. I just thought it was neat that we didn't even had to serialize/communicate the tasks :)