Failed requests to broker holds up ephemeral runner reconciler #4699

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.15.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Request a high number of jobs per second, i.e. >5/second
2. Hit the "Runner registrations" rate limit for self hosted runners https://docs.github.com/en/actions/reference/limits
3. A number of runners stuck with `Completed` status not being cleaned up

Describe the bug

When hitting the self hosted runner registrations limit
controller_runtime_active_workers{controller="ephemeralrunner"} reaches max while the workers are held up by the retrying actionsClient.GenerateJitRunnerConfig

Describe the expected behavior

Workers should fail fast when hitting the rate limit, perhaps needing a separate queue of some kind for requests to the broker.

Additional Context

I've been performing some load testing with 0.15.0, and while not being able to find any other bottlenecks in reconciliation anymore, this one stands out due to how many pods stuck in the `Completed` state pile up.
It's also not possible to determine through the logs if the rate limits are being hit, it possibly needs something like this https://github.com/actions/scaleset/pull/118
I believe this has been an issue in earlier versions as well but has been less obvious due to hitting other bottlenecks first.

Controller Logs

--

Runner Pod Logs

--