[Tracking] Improve benchmark visibility / monitoring #148

Activity

1. jrbourbeau

   added

   infrastructureWork related to infrastuctureWork related to infrastucture

   on May 27, 2022
2. jrbourbeau

   self-assigned this

   on May 27, 2022
3. mrocklin commented on May 27, 2022

   

   mrocklin

   on May 27, 2022

   Member

   More actions

   I'd love to learn more about this. In particular when thinking about Coiled Analtyics (which I'm thinking about now) I'm curious what we would need to build to satisfy this need. I don't think that folks should block on Coiled Analytics work here though (asv or whatever is probably better). I'm just looking forward to learning more.
4. fjetter commented on Jun 24, 2022

   

   fjetter

   on Jun 24, 2022

   Last edited by fjetter

   Contributor

   More actions

   A couple of requirements I see for our benchmarking suite

   • Ability to record historical data
   • Compare two commits or two different configurations (e.g. to verify feature toggles or test PRs)
   • Be able to test PRs or commits on dask/dask and dask/distributed (this is likely rather a coiled-runtime feature overall)
   • Multiple metrics can be collected in the same run, e.g. total runtime, peak memory usage, (probably something else as well)
   • store various artifacts produced during the run (e.g. performance report or MemorySampler output)

   Some of this does not need to be part of the tooling we're using and could be part of additional helper scripts or part of how the CI works.
5. hayesgb

   assigned

   ian-r-rose

   and

   ncclementi

   and unassigned

   jrbourbeau

   on Jun 24, 2022
6. hayesgb

   changed the title [-]Improve benchmark visibility / monitoring[/-] [+][Tracking] Improve benchmark visibility / monitoring[/+] on Jun 24, 2022
7. hayesgb

   added

   epic

   on Jun 24, 2022
8. hayesgb

   added this to the Benchmarking Execution Times milestone on Jun 24, 2022
9. ncclementi commented on Aug 30, 2022

   

   ncclementi

   on Aug 30, 2022

   Contributor

   More actions

   Closing this as all the tracking points in the opening comment have been completed.
10. ncclementi

    closed this as completedon Aug 30, 2022