branch-4.1: [fix](s3) Add S3 rate limit observability #64038

[gavinchou]

What problem does this PR solve?

Backport #64038 to branch-4.1.

Add observability for S3 local rate limiting and S3/Azure 429 responses so throttling-related performance issues can be diagnosed and alerted more directly.

What is changed?

• Add explicit S3 local rate limiter bvars for sleep duration, sleep count, and rejected count.
• Rename the old ambiguous S3 local rate limiter bvars so their meaning is clear.
• Log the first and every N local S3 rate limiter throttled/rejected requests without reading bvar values for decisions.
• Add S3 429 retry/failure bvars for S3/Azure object clients, including Azure batch delete failures.
• Capture the limiter configuration and result under the limiter lock to avoid concurrent reset races.
• Adapt the Cloud configuration change to branch-4.1 without bringing in unrelated master-only fault-injection settings.

New or renamed bvar metrics

Renamed local S3 GET limiter metrics:

• get_rate_limit_ns -> s3_get_rate_limit_sleep_ns
• get_rate_limit_exceed_req_num -> s3_get_rate_limit_sleep_count

Renamed local S3 PUT limiter metrics:

• put_rate_limit_ns -> s3_put_rate_limit_sleep_ns
• put_rate_limit_exceed_req_num -> s3_put_rate_limit_sleep_count

New local S3 limiter rejection metrics:

• s3_get_rate_limit_rejected_count
• s3_put_rate_limit_rejected_count

New S3/Azure 429 metrics:

• s3_request_retry_too_many_requests_count
• s3_request_failed_too_many_requests_count

Metric examples:

s3_get_rate_limit_sleep_ns : 123456789
s3_get_rate_limit_sleep_count : 42
s3_get_rate_limit_rejected_count : 3
s3_put_rate_limit_sleep_ns : 987654321
s3_put_rate_limit_sleep_count : 27
s3_put_rate_limit_rejected_count : 1
s3_request_retry_too_many_requests_count : 8
s3_request_failed_too_many_requests_count : 2

Example local limiter logs:

S3 GET request is throttled by local rate limiter, sleep_ms=12, sleep_count=1000, token_per_second=1000, bucket_tokens=2000, token_limit=5000
S3 PUT request is rejected by local rate limiter, rejected_count=1, token_per_second=1000, bucket_tokens=2000, token_limit=5000

Tests

• sh run-cloud-ut.sh --run --filter='s3_rate_limiter_test:*' (7/7 passed)
• sh run-be-ut.sh --run --filter='S3UTILTest.*:S3FileWriterTest.*' (24/24 passed)
• sh run-be-ut.sh --run --filter='AzureObjStorageClientTlsHelperTest.*' (2/2 passed)

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[gavinchou]

run buildall

[gavinchou]

/review

[github-actions]

I found two P2 observability correctness issues in the Azure paths: retry accounting is based on the current attempt index rather than whether a 429 actually causes a retry, and batch-delete accounting stops before inspecting later independent subresponses.

Critical checkpoint conclusions:

• Goal/test proof: the patch adds the intended local limiter metrics/logging and broad S3/Azure final-429 accounting, but the two inline issues leave common Azure throttling sequences undercounted or misclassified. The added tests do not cover either sequence.
• Scope/focus: the backport is otherwise focused on object-storage rate-limit observability.
• Concurrency/lifecycle: add_with_config() captures the outcome and limiter configuration under the holder shared lock while reset uses the exclusive lock; callbacks run after unlock. No remaining lifecycle or reset race was found.
• Configuration/compatibility: BE and Cloud define the mutable nonnegative log interval consistently. The metric renames are user-visible but documented; no storage, persistence, wire-format, or FE/BE compatibility issue was found.
• Parallel/error paths: BE S3, BE Azure, Cloud S3, Cloud Azure, paged/list paths, expected-not-found exemptions, local synthetic failures, and outer/deferred batch paths were checked. Apart from the two inline issues, terminal accounting is balanced without wrapper double-counting.
• Testing/validation: review was static against authoritative pr.diff and head ba9cd8fbda1d59f7005d5d19dbed5b2650621177; no local build or test was run because the review contract explicitly prohibits builds. CI status is external to this conclusion.
• User focus: no additional user-provided focus was specified.

Convergence: after the initial candidate merge, two normal full-review agents and a separate risk-focused agent rechecked the same two-comment set and all returned NO_NEW_VALUABLE_FINDINGS in round 2.

[hello-stephen]

BE UT Coverage Report

Increment line coverage 37.50% (9/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.57% (22724/40167)
Line Coverage	40.32% (222515/551840)
Region Coverage	36.64% (175676/479489)
Branch Coverage	37.73% (78466/207953)

a

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 41.67% (10/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.94% (28622/39238)
Line Coverage	56.33% (309397/549227)
Region Coverage	53.57% (258499/482572)
Branch Coverage	54.47% (113425/208240)