Repository navigation
TaskState metadata #4188
Description
Recently I was working with a group that computed user-defined attributes for each task (e.g. the start and stop time when running the task) and stored them in an external database. However, after #4107 we now have a TaskState object on both the worker and scheduler which could be used to store custom task-level information directly on the cluster.
For example, we could add a new TaskState.metadata mapping to both the scheduler and worker TaskState classes and synchronize them when Worker.send_task_state_to_scheduler is called. This would give users an option to store rich information relevant to their particular workloads
This seems like useful functionality to me and I wanted to see if other groups may also benefit from adding task-level metadata
Activity
mrocklin commentedon Oct 26, 2020
Thinking again about this this might be simple enough to implement that it might make sense to just create a PR. It might make things more clear and I'm guessing it's only a few lines.
quasiben commentedon Oct 27, 2020
I think this would be great! It'd be cool to optionally store timeseries info about tasks in addition to the aggregate behavior we currently store now. Specifically, with this kind of tooling we may be able to resolve #4159
jrbourbeau commentedon Oct 27, 2020
I opened up a PR to implement this over in #4191