branch-3.1: [Fix](MS)Handle missing endpoint in DLF-only Metastore scenario with region fallback #54300 - #54522
Merged
Merged
Conversation
…region fallback (#54300) #50238 ### PR Description This PR fixes a special case where the endpoint parameter might be missing when using DLF as the Hive Metastore (MS) without configuring any storage backend. #### Background In certain deployments, users rely solely on DLF as the metastore and do not explicitly configure object storage credentials. In these cases: Only DLF access_key, secret_key, and region are provided. dlf.endpoint is omitted by the user. ``` create catalog if not exists pm_dlf properties ( "type" = "paimon", "paimon.catalog.type" = "dlf", "warehouse" = "oss://asf-doris-datalake-test/p2_regression_case", "dlf.proxy.mode" = "DLF_ONLY", "dlf.uid" = "1289998888", "dlf.region" = "cn-beijing", "dlf.catalog.id" = "asf", "dlf.access_key" = "ABSHJJJKKKLKJHVGFGHJKLKJHGHJKJHGHJK", "dlf.secret_key" = "UYGFHJKHGVVHBJKHVGBJKL:JHGVBJKLHVGHJHVGH" ); switch pm_dlf; show databases; exception: Exception:java.sql.SQLException: errCode = 2, detailMessage = UserException: errCode = 2, detailMessage = No storage properties found for schema: oss at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at ``` The system attempts to initialize storage configuration, but fails because endpoint is required. This issue occurs because the existing logic assumes that the storage module always receives a valid endpoint, which is not the case when DLF is used as MS only. #### Fix When dlf.endpoint is not set, and dlf.region is available, this patch automatically derives a valid default endpoint based on the region and network mode (public or VPC). This logic is consistent with the existing behavior of DLF SDKs and avoids unnecessary configuration burden for users using DLF as MS only. #### Compatibility Backward compatible: No breaking change for users who already provide an endpoint. Recommended best practice remains: Users are encouraged to explicitly configure both endpoint and region for clarity.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
TPC-H: Total hot run time: 32501 ms |
TPC-DS: Total hot run time: 192262 ms |
ClickBench: Total hot run time: 29.18 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
morrySnow
approved these changes
Aug 13, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #54300