[SPARK-57467][SCHEDULER] Reuse identical resource profiles when available

[psavalle]

What changes were proposed in this pull request?

When registering a resource profile, check if one with the same requests already exists. If so, return it, so that the same executors can be reused.

ResourceProfileManager has a method getEquivalentProfile, but it is only called in DAGScheduler.mergeResourceProfilesForStage when stageResourceProfiles.size > 1. Even there, it seems it may be prone to a race condition since it does not acquire the lock for both steps (checking for a profile, adding one if needed).

Why are the changes needed?

Currently, every time a profile is registered, new executors will need to spin up -- even if the resource requests are the same. This is especially problematic with Spark Connect, where different isolated sessions may try and register the same profile, but all end up with isolated executors.

We could also consider putting this behavior behind a configuration flag, if it is desirable to retain the previous behavior by default.

Does this PR introduce any user-facing change?

Previously, equivalent resource profiles would get different profile IDs, and separate executors.

With this change, equivalent resource profiles will get the same ID, and share executors.

How was this patch tested?

Added tests

Was this patch authored or co-authored using generative AI tooling?

Yes, modified initial changes provided by Claude.

Generated-by: 2.1.159 (Claude Code)

[psavalle]

Not really sure would could have a look, maybe @wbo4958? Thank you!

[uros-b]

Regarding the PR description, it seems that there actually is an observable behavior change. Repeated CreateResourceProfile calls with identical resources now return the same profile ID instead of a new one each time. So, equivalent profiles now share executors under dynamic allocation instead of each getting a separate pool.

If there is a user-facing change, it should be notead and we will need an appropriate release note.

[uros-b]

In any case, I don't have full context here - so @tgravescs @wbo4958 @HyukjinKwon @Ngone51 please review this PR.

[psavalle]

Makes sense, I've updated the PR description, thank you.

[mridulm]

+CC @max2718281 PTAL