Add action to run benchmarks

[dalonsoa]

Description

The build and test action has been updated to run the benchmarks and publish the summary in the action itself, as well as creating an Alert comment in the relevant places wherever there has been a regression.

As we know, the output produced by Criterion is not the standard that the action expects, so we (my friend Claude and myself) embedded in the action a small python script to adapt the output.

The benchmark result, used to compared with future runs, is stored in the GitHub cache, which only last 7 days. If we want to keep it for longer, then we would need to find somewhere else to save it.

A summary of the benchmark is included in the Summary section of the workflow, regardless of it being successful or not.

Fixes #1453

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc
• Update release notes for the latest release if this PR adds a new feature or fixes a bug
  present in the previous release

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.31%. Comparing base (236974e) to head (8166e68).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1509   +/-   ##
=======================================
  Coverage   90.31%   90.31%           
=======================================
  Files          60       60           
  Lines        8663     8663           
  Branches     8663     8663           
=======================================
  Hits         7824     7824           
  Misses        525      525           
  Partials      314      314           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds benchmark execution and reporting to the existing CI workflow to address #1453 by running cargo bench in GitHub Actions and publishing benchmark results/alerts.

Changes:

• Added a new benchmark job to run cargo bench on ubuntu-latest.
• Integrated benchmark-action/github-action-benchmark@v1 to publish a job summary and comment on regressions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dc2917]

Looks good, and seems logical, just wondering if it's not cleaner to stick the python code in a .py file to run as a script

[dc2917]

Might it be tidier to stick this in a .py file?

[tsmbland]

This seems reasonable. Curious how it works in practice:

• If I make a series of pushes to a PR branch, each time it will compare against the last push, not against main, right?
• If you make a single push to a PR branch, will this compare against main or against nothing?
• If, for example, one push regressed by 20% and the next by another 20%, the benchmarks will never fail, even though performance has regressed by more than 30% overall. Right?
• If a series of commits are pushed together, does the alert comment get added to the latest of those commits (even though it could have been an earlier commit that caused the regression)? Or is it not tied to a specific commit?
• In most cases, we'll only know how much a PR has actually regressed against main after merging into main (provided the last merge to main wasn't more than 7 days ago)?

I'd also be tempted just to run the full benchmarking on pushes to main, and include in the testing suite a very small test just to make sure the benchmarks can compile/run without error.

Also, I'm not sure we need to run the full scaling for the assets benchmark, especially because the github summary doesn't give us a nice way to visualise this (it's not even in numerical order!). Would it be sufficient just to run the highest n (20) for sequential and parallel, and exclude 1-19, since we're mostly concerned about regression here rather than scaling?