Enhance Test Improver with Flaky Test Detection and Mitigation #413

See comments in elastic/beats#48983.

Flaky tests are a major problem in large codebases. New tests and functionality are added and modified at a high rate. Running a test once or twice might succeed, but for various reasons it may not be completely stable. Adding new tests and improving coverage is always welcome, but it also risks introducing flakyness. This usually only becomes apparent after the PR is merged and CI runs the tests more frequently.

It would be beneficial to include a flaky test detection or mitigation strategy as part of the test steps. For example, running each test multiple times instead of just once can help ensure stability and surface intermittent failures earlier.

There are tools specifically designed for this purpose. In Go, for instance, you can use https://pkg.go.dev/golang.org/x/tools/cmd/stress
or the -test.count flag to execute tests repeatedly. Running tests multiple times as separate processes can also reveal different classes of flakiness, since process-level isolation may expose issues that in-process reruns do not.

Even custom scripts can be effective in this regard, such as the stresstest.sh script we use in Beats: https://github.com/elastic/beats/blob/main/script/stresstest.sh to run stress under process isolation.