From 7632a074e4b678900dfebb466681f8fa2f073800 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 20 Jun 2026 22:42:42 +0800 Subject: [PATCH 1/4] [P5-T29] paimon B8 (C1): delete dead fe-core legacy subtree + sever reverse-refs Batch 1 / commit 1 of removing the legacy paimon subsystem from fe-core after the SPI cutover (#64446). Mirrors P4 #64300; per that precedent the reverse-ref removal and dead-file deletion must land as one compiling unit (PaimonUtils:57 calls the removed ExternalMetaCacheMgr.paimon()). Delete 33 dead main files (datasource/paimon/* except the LIVE PaimonVendedCredentialsProvider + datasource/metacache/paimon/* + datasource/systable/PaimonSysTable) and 5 dead tests (SUT is a deleted class). Sever live reverse-refs to the deleted classes: - ExternalCatalog: drop the PAIMON db-creation switch arm (PluginDriven forces PLUGIN). - ExternalMetaCacheMgr / ExternalMetaCacheRouteResolver: drop the paimon engine routing/accessor/registration (PluginDriven catalogs route to the default cache). - Env.getDdlStmt: drop the dead PAIMON_EXTERNAL_TABLE branch; KEEP the LIVE PLUGIN_EXTERNAL_TABLE D-046 SHOW CREATE LOCATION/PROPERTIES rendering. - UserAuthentication: drop the PaimonSysExternalTable branch (the live PluginDrivenSysExternalTable branch already authorizes sys-tables). - ShowPartitionsCommand: drop the legacy paimon clauses + handleShowPaimonTablePartitions(); KEEP hasPartitionStatsCapability() (drives the 5-column path post-cutover) and the live TableType.PAIMON_EXTERNAL_TABLE enum. - Scrub 3 dangling {@link}/@see PaimonSysTable/PaimonSysExternalTable javadocs. Decouple the STILL-CONSUMED property/metastore/Paimon* classes from the deleted PaimonExternalCatalog by inlining getPaimonCatalogType() string literals ("hms"/"filesystem"/"dlf"/"rest"/"jdbc"). These thin metastore-property classes stay in fe-core; their paimon-SDK catalog-building methods are stripped in commit 2. Tests: swap StatementContextTest preload mock PaimonExternalTable -> PluginDrivenMvccExternalTable; trim ExternalMetaCacheRouteResolverTest's deleted paimon route + fixture predicates; repoint Jdbc/Rest metastore-props constant assertions to literals. fe-core test-compile + checkstyle clean; 49 affected tests pass. KEEP for Batch 2 (docker-gated): PaimonVendedCredentialsProvider + the generic VendedCredentialsFactory paimon seam; the 5 paimon maven deps remain until then. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_011mTrPcvMZtFjsxWJM5TRnG --- .../java/org/apache/doris/catalog/Env.java | 24 - .../doris/datasource/ExternalCatalog.java | 3 - .../datasource/ExternalMetaCacheMgr.java | 8 - .../PluginDrivenSysExternalTable.java | 2 +- .../ExternalMetaCacheRouteResolver.java | 6 - .../PaimonLatestSnapshotProjectionLoader.java | 83 -- .../paimon/PaimonPartitionInfoLoader.java | 58 -- .../metacache/paimon/PaimonTableLoader.java | 48 - .../paimon/DorisToPaimonTypeVisitor.java | 109 --- .../paimon/PaimonDLFExternalCatalog.java | 29 - .../paimon/PaimonExternalCatalog.java | 192 ---- .../paimon/PaimonExternalCatalogFactory.java | 48 - .../paimon/PaimonExternalDatabase.java | 37 - .../paimon/PaimonExternalMetaCache.java | 116 --- .../paimon/PaimonExternalTable.java | 429 --------- .../paimon/PaimonFileExternalCatalog.java | 29 - .../paimon/PaimonHMSExternalCatalog.java | 29 - .../datasource/paimon/PaimonMetadataOps.java | 405 -------- .../datasource/paimon/PaimonMvccSnapshot.java | 32 - .../datasource/paimon/PaimonPartition.java | 61 -- .../paimon/PaimonPartitionInfo.java | 56 -- .../paimon/PaimonRestExternalCatalog.java | 29 - .../paimon/PaimonSchemaCacheKey.java | 56 -- .../paimon/PaimonSchemaCacheValue.java | 47 - .../datasource/paimon/PaimonSnapshot.java | 45 - .../paimon/PaimonSnapshotCacheValue.java | 37 - .../paimon/PaimonSysExternalTable.java | 277 ------ .../paimon/PaimonTableCacheValue.java | 44 - .../doris/datasource/paimon/PaimonUtil.java | 711 -------------- .../doris/datasource/paimon/PaimonUtils.java | 59 -- .../paimon/profile/PaimonMetricRegistry.java | 72 -- .../profile/PaimonScanMetricsReporter.java | 152 --- .../source/PaimonPredicateConverter.java | 201 ---- .../paimon/source/PaimonScanNode.java | 900 ------------------ .../paimon/source/PaimonSource.java | 102 -- .../datasource/paimon/source/PaimonSplit.java | 159 ---- .../paimon/source/PaimonValueConverter.java | 162 ---- .../PaimonAliyunDLFMetaStoreProperties.java | 3 +- .../PaimonFileSystemMetaStoreProperties.java | 3 +- .../PaimonHMSMetaStoreProperties.java | 3 +- .../PaimonJdbcMetaStoreProperties.java | 3 +- .../PaimonRestMetaStoreProperties.java | 3 +- .../datasource/systable/NativeSysTable.java | 2 - .../datasource/systable/PaimonSysTable.java | 68 -- .../systable/PluginDrivenSysTable.java | 2 +- .../rules/analysis/UserAuthentication.java | 6 +- .../plans/commands/ShowPartitionsCommand.java | 65 +- .../ExternalMetaCacheRouteResolverTest.java | 9 +- .../paimon/PaimonExternalMetaCacheTest.java | 123 --- .../paimon/PaimonMetadataOpsTest.java | 259 ----- .../datasource/paimon/PaimonUtilTest.java | 146 --- .../paimon/source/PaimonScanNodeTest.java | 658 ------------- .../PaimonJdbcMetaStorePropertiesTest.java | 3 +- .../PaimonRestMetaStorePropertiesTest.java | 4 +- .../doris/nereids/StatementContextTest.java | 4 +- .../planner/PaimonPredicateConverterTest.java | 99 -- .../P5-T29-paimon-legacy-removal-design.md | 183 ++++ 57 files changed, 202 insertions(+), 6301 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonLatestSnapshotProjectionLoader.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonPartitionInfoLoader.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonTableLoader.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonDLFExternalCatalog.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalogFactory.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalDatabase.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonFileExternalCatalog.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMvccSnapshot.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartition.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartitionInfo.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonRestExternalCatalog.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheKey.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheValue.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshot.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshotCacheValue.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonTableCacheValue.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtils.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonMetricRegistry.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonScanMetricsReporter.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSource.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSplit.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PaimonSysTable.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCacheTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/planner/PaimonPredicateConverterTest.java create mode 100644 plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java index cd1730dbf59256..68c4e28249a8c7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java @@ -108,8 +108,6 @@ import org.apache.doris.datasource.hive.event.MetastoreEventsProcessor; import org.apache.doris.datasource.iceberg.IcebergExternalTable; import org.apache.doris.datasource.iceberg.IcebergSysExternalTable; -import org.apache.doris.datasource.paimon.PaimonExternalTable; -import org.apache.doris.datasource.paimon.PaimonSysExternalTable; import org.apache.doris.deploy.DeployManager; import org.apache.doris.deploy.impl.LocalFileDeployManager; import org.apache.doris.dictionary.DictionaryManager; @@ -4914,28 +4912,6 @@ public static void getDdlStmt(Command command, String dbName, TableIf table, Lis } } sb.append("\n)"); - } else if (table.getType() == TableType.PAIMON_EXTERNAL_TABLE) { - addTableComment(table, sb); - PaimonExternalTable paimonExternalTable; - if (table instanceof PaimonExternalTable) { - paimonExternalTable = (PaimonExternalTable) table; - } else if (table instanceof PaimonSysExternalTable) { - paimonExternalTable = ((PaimonSysExternalTable) table).getSourceTable(); - } else { - throw new RuntimeException("Unexpected Paimon table type: " + table.getClass().getSimpleName()); - } - Map properties = paimonExternalTable.getTableProperties(); - sb.append("\nLOCATION '").append(properties.getOrDefault("path", "")).append("'"); - sb.append("\nPROPERTIES ("); - Iterator> iterator = properties.entrySet().iterator(); - while (iterator.hasNext()) { - Entry prop = iterator.next(); - sb.append("\n \"").append(prop.getKey()).append("\" = \"").append(prop.getValue()).append("\""); - if (iterator.hasNext()) { - sb.append(","); - } - } - sb.append("\n)"); } else if (table.getType() == TableType.PLUGIN_EXTERNAL_TABLE) { addTableComment(table, sb); PluginDrivenExternalTable pluginExternalTable; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java index d3a3293a2f85e9..3ce105527fed5c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java @@ -50,7 +50,6 @@ import org.apache.doris.datasource.lakesoul.LakeSoulExternalDatabase; import org.apache.doris.datasource.metacache.MetaCache; import org.apache.doris.datasource.operations.ExternalMetadataOps; -import org.apache.doris.datasource.paimon.PaimonExternalDatabase; import org.apache.doris.datasource.test.TestExternalCatalog; import org.apache.doris.datasource.test.TestExternalDatabase; import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; @@ -963,8 +962,6 @@ protected ExternalDatabase buildDbForInit(String remote return new LakeSoulExternalDatabase(this, dbId, localDbName, remoteDbName); case TEST: return new TestExternalDatabase(this, dbId, localDbName, remoteDbName); - case PAIMON: - return new PaimonExternalDatabase(this, dbId, localDbName, remoteDbName); case TRINO_CONNECTOR: return new PluginDrivenExternalDatabase(this, dbId, localDbName, remoteDbName); case REMOTE_DORIS: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java index ab6fefa8949447..8893d5800d2542 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java @@ -32,7 +32,6 @@ import org.apache.doris.datasource.metacache.MetaCacheEntryDef; import org.apache.doris.datasource.metacache.MetaCacheEntryInvalidation; import org.apache.doris.datasource.metacache.MetaCacheEntryStats; -import org.apache.doris.datasource.paimon.PaimonExternalMetaCache; import org.apache.doris.fs.FileSystemCache; import com.github.benmanes.caffeine.cache.stats.CacheStats; @@ -63,7 +62,6 @@ public class ExternalMetaCacheMgr { private static final String ENGINE_HIVE = "hive"; private static final String ENGINE_HUDI = "hudi"; private static final String ENGINE_ICEBERG = "iceberg"; - private static final String ENGINE_PAIMON = "paimon"; private static final String ENGINE_DORIS = "doris"; /** @@ -173,11 +171,6 @@ public IcebergExternalMetaCache iceberg(long catalogId) { return (IcebergExternalMetaCache) engine(ENGINE_ICEBERG); } - public PaimonExternalMetaCache paimon(long catalogId) { - prepareCatalogByEngine(catalogId, ENGINE_PAIMON); - return (PaimonExternalMetaCache) engine(ENGINE_PAIMON); - } - public DorisExternalMetaCache doris(long catalogId) { prepareCatalogByEngine(catalogId, ENGINE_DORIS); return (DorisExternalMetaCache) engine(ENGINE_DORIS); @@ -299,7 +292,6 @@ private void registerBuiltinEngineCaches() { cacheRegistry.register(new HiveExternalMetaCache(commonRefreshExecutor, fileListingExecutor)); cacheRegistry.register(new HudiExternalMetaCache(commonRefreshExecutor)); cacheRegistry.register(new IcebergExternalMetaCache(commonRefreshExecutor)); - cacheRegistry.register(new PaimonExternalMetaCache(commonRefreshExecutor)); cacheRegistry.register(new DorisExternalMetaCache(commonRefreshExecutor)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSysExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSysExternalTable.java index f7d628a83fecc0..15653ee288d004 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSysExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSysExternalTable.java @@ -31,7 +31,7 @@ *

Created transiently by {@link org.apache.doris.datasource.systable.PluginDrivenSysTable} during * planning/describe (via {@code createSysExternalTable}); it is NEVER added to a persisted table map * and is NOT GSON-registered, mirroring legacy sys ExternalTables (e.g. - * {@link org.apache.doris.datasource.paimon.PaimonSysExternalTable}).

+ * {@code PaimonSysExternalTable}).

* *

It reports {@link org.apache.doris.catalog.TableIf.TableType#PLUGIN_EXTERNAL_TABLE} (inherited); * no connector-specific table type is introduced. The whole schema/partition/row-count path is reused diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java index 16576ea350005e..65d143ab7c5bb8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java @@ -22,7 +22,6 @@ import org.apache.doris.datasource.doris.RemoteDorisExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergExternalCatalog; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import java.util.ArrayList; import java.util.LinkedHashSet; @@ -38,7 +37,6 @@ public class ExternalMetaCacheRouteResolver { private static final String ENGINE_HIVE = "hive"; private static final String ENGINE_HUDI = "hudi"; private static final String ENGINE_ICEBERG = "iceberg"; - private static final String ENGINE_PAIMON = "paimon"; private static final String ENGINE_DORIS = "doris"; private final ExternalMetaCacheRegistry registry; @@ -66,10 +64,6 @@ private void addBuiltinRoutes(Set resolved, CatalogIf cata resolved.add(registry.resolve(ENGINE_ICEBERG)); return; } - if (catalog instanceof PaimonExternalCatalog) { - resolved.add(registry.resolve(ENGINE_PAIMON)); - return; - } if (catalog instanceof RemoteDorisExternalCatalog) { resolved.add(registry.resolve(ENGINE_DORIS)); return; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonLatestSnapshotProjectionLoader.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonLatestSnapshotProjectionLoader.java deleted file mode 100644 index 8e2c7a73b33901..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonLatestSnapshotProjectionLoader.java +++ /dev/null @@ -1,83 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.metacache.paimon; - -import org.apache.doris.catalog.Column; -import org.apache.doris.datasource.CacheException; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.paimon.PaimonPartitionInfo; -import org.apache.doris.datasource.paimon.PaimonSchemaCacheValue; -import org.apache.doris.datasource.paimon.PaimonSnapshot; -import org.apache.doris.datasource.paimon.PaimonSnapshotCacheValue; - -import org.apache.paimon.CoreOptions; -import org.apache.paimon.Snapshot; -import org.apache.paimon.schema.TableSchema; -import org.apache.paimon.table.DataTable; -import org.apache.paimon.table.Table; - -import java.util.Collections; -import java.util.List; -import java.util.Optional; - -/** - * Resolves the latest snapshot runtime projection from the base table entry. - */ -public final class PaimonLatestSnapshotProjectionLoader { - @FunctionalInterface - public interface SchemaValueLoader { - PaimonSchemaCacheValue load(NameMapping nameMapping, long schemaId); - } - - private final PaimonPartitionInfoLoader partitionInfoLoader; - private final SchemaValueLoader schemaValueLoader; - - public PaimonLatestSnapshotProjectionLoader(PaimonPartitionInfoLoader partitionInfoLoader, - SchemaValueLoader schemaValueLoader) { - this.partitionInfoLoader = partitionInfoLoader; - this.schemaValueLoader = schemaValueLoader; - } - - public PaimonSnapshotCacheValue load(NameMapping nameMapping, Table paimonTable) { - try { - PaimonSnapshot latestSnapshot = resolveLatestSnapshot(paimonTable); - List partitionColumns = schemaValueLoader.load(nameMapping, latestSnapshot.getSchemaId()) - .getPartitionColumns(); - PaimonPartitionInfo partitionInfo = partitionInfoLoader.load(nameMapping, paimonTable, partitionColumns); - return new PaimonSnapshotCacheValue(partitionInfo, latestSnapshot); - } catch (Exception e) { - throw new CacheException("failed to load paimon snapshot %s.%s.%s: %s", - e, nameMapping.getCtlId(), nameMapping.getLocalDbName(), nameMapping.getLocalTblName(), - e.getMessage()); - } - } - - private PaimonSnapshot resolveLatestSnapshot(Table paimonTable) { - Table snapshotTable = paimonTable; - long latestSnapshotId = PaimonSnapshot.INVALID_SNAPSHOT_ID; - Optional optionalSnapshot = paimonTable.latestSnapshot(); - if (optionalSnapshot.isPresent()) { - latestSnapshotId = optionalSnapshot.get().id(); - snapshotTable = paimonTable.copy( - Collections.singletonMap(CoreOptions.SCAN_SNAPSHOT_ID.key(), String.valueOf(latestSnapshotId))); - } - DataTable dataTable = (DataTable) paimonTable; - long latestSchemaId = dataTable.schemaManager().latest().map(TableSchema::id).orElse(0L); - return new PaimonSnapshot(latestSnapshotId, latestSchemaId, snapshotTable); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonPartitionInfoLoader.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonPartitionInfoLoader.java deleted file mode 100644 index c29a359b9592d1..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonPartitionInfoLoader.java +++ /dev/null @@ -1,58 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.metacache.paimon; - -import org.apache.doris.catalog.Column; -import org.apache.doris.common.AnalysisException; -import org.apache.doris.datasource.CacheException; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.paimon.PaimonPartitionInfo; -import org.apache.doris.datasource.paimon.PaimonUtil; - -import org.apache.commons.collections4.CollectionUtils; -import org.apache.paimon.partition.Partition; -import org.apache.paimon.table.Table; - -import java.util.List; - -/** - * Loads partition info for a snapshot projection from the base Paimon table and catalog metadata. - */ -public final class PaimonPartitionInfoLoader { - private final PaimonTableLoader tableLoader; - - public PaimonPartitionInfoLoader(PaimonTableLoader tableLoader) { - this.tableLoader = tableLoader; - } - - public PaimonPartitionInfo load(NameMapping nameMapping, Table paimonTable, List partitionColumns) - throws AnalysisException { - if (CollectionUtils.isEmpty(partitionColumns)) { - return PaimonPartitionInfo.EMPTY; - } - try { - List paimonPartitions = tableLoader.catalog(nameMapping).getPaimonPartitions(nameMapping); - boolean legacyPartitionName = PaimonUtil.isLegacyPartitionName(paimonTable); - return PaimonUtil.generatePartitionInfo(partitionColumns, paimonPartitions, legacyPartitionName); - } catch (Exception e) { - throw new CacheException("failed to load paimon partition info %s.%s.%s: %s", - e, nameMapping.getCtlId(), nameMapping.getLocalDbName(), nameMapping.getLocalTblName(), - e.getMessage()); - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonTableLoader.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonTableLoader.java deleted file mode 100644 index 0a134cfd7d7d32..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonTableLoader.java +++ /dev/null @@ -1,48 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.metacache.paimon; - -import org.apache.doris.catalog.Env; -import org.apache.doris.datasource.CacheException; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; - -import org.apache.paimon.table.Table; - -import java.io.IOException; - -/** - * Loads the base Paimon table handle used by cache entries and runtime projections. - */ -public final class PaimonTableLoader { - - public Table load(NameMapping nameMapping) { - try { - return catalog(nameMapping).getPaimonTable(nameMapping); - } catch (Exception e) { - throw new CacheException("failed to load paimon table %s.%s.%s: %s", - e, nameMapping.getCtlId(), nameMapping.getLocalDbName(), nameMapping.getLocalTblName(), - e.getMessage()); - } - } - - public PaimonExternalCatalog catalog(NameMapping nameMapping) throws IOException { - return (PaimonExternalCatalog) Env.getCurrentEnv().getCatalogMgr() - .getCatalogOrException(nameMapping.getCtlId(), id -> new IOException("Catalog not found: " + id)); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java deleted file mode 100644 index aad8106563b4f4..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java +++ /dev/null @@ -1,109 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.MapType; -import org.apache.doris.catalog.PrimitiveType; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.StructField; -import org.apache.doris.catalog.StructType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.DorisTypeVisitor; - -import org.apache.paimon.types.BigIntType; -import org.apache.paimon.types.BooleanType; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.DataType; -import org.apache.paimon.types.DateType; -import org.apache.paimon.types.DecimalType; -import org.apache.paimon.types.DoubleType; -import org.apache.paimon.types.FloatType; -import org.apache.paimon.types.IntType; -import org.apache.paimon.types.RowType; -import org.apache.paimon.types.TimestampType; -import org.apache.paimon.types.VarBinaryType; -import org.apache.paimon.types.VarCharType; -import org.apache.paimon.types.VariantType; - -import java.util.ArrayList; -import java.util.List; -import java.util.concurrent.atomic.AtomicInteger; - -public class DorisToPaimonTypeVisitor extends DorisTypeVisitor { - - @Override - public DataType struct(StructType struct, List fieldResults) { - List fields = struct.getFields(); - List newFields = new ArrayList<>(fields.size()); - AtomicInteger atomicInteger = new AtomicInteger(-1); - for (int i = 0; i < fields.size(); i++) { - StructField field = fields.get(i); - DataType fieldType = fieldResults.get(i).copy(field.getContainsNull()); - String comment = field.getComment(); - DataField dataField = new DataField(atomicInteger.incrementAndGet(), field.getName(), fieldType, comment); - newFields.add(dataField); - } - return new RowType(newFields); - } - - @Override - public DataType field(StructField field, DataType typeResult) { - return typeResult; - } - - @Override - public DataType array(ArrayType array, DataType elementResult) { - return new org.apache.paimon.types.ArrayType(elementResult.copy(array.getContainsNull())); - } - - @Override - public DataType map(MapType map, DataType keyResult, DataType valueResult) { - return new org.apache.paimon.types.MapType(keyResult.copy(false), - valueResult.copy(map.getIsValueContainsNull())); - } - - @Override - public DataType atomic(Type atomic) { - PrimitiveType primitiveType = atomic.getPrimitiveType(); - if (primitiveType.equals(PrimitiveType.BOOLEAN)) { - return new BooleanType(); - } else if (primitiveType.equals(PrimitiveType.INT)) { - return new IntType(); - } else if (primitiveType.equals(PrimitiveType.BIGINT)) { - return new BigIntType(); - } else if (primitiveType.equals(PrimitiveType.FLOAT)) { - return new FloatType(); - } else if (primitiveType.equals(PrimitiveType.DOUBLE)) { - return new DoubleType(); - } else if (primitiveType.isCharFamily()) { - return new VarCharType(VarCharType.MAX_LENGTH); - } else if (primitiveType.equals(PrimitiveType.DATE) || primitiveType.equals(PrimitiveType.DATEV2)) { - return new DateType(); - } else if (primitiveType.equals(PrimitiveType.DECIMALV2) || primitiveType.isDecimalV3Type()) { - return new DecimalType(((ScalarType) atomic).getScalarPrecision(), ((ScalarType) atomic).getScalarScale()); - } else if (primitiveType.equals(PrimitiveType.DATETIME) || primitiveType.equals(PrimitiveType.DATETIMEV2)) { - return new TimestampType(); - } else if (primitiveType.isVarbinaryType()) { - return new VarBinaryType(VarBinaryType.MAX_LENGTH); - } else if (primitiveType.isVariantType()) { - return new VariantType(); - } - throw new UnsupportedOperationException("Not a supported type: " + primitiveType); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonDLFExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonDLFExternalCatalog.java deleted file mode 100644 index a982abe5b017f8..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonDLFExternalCatalog.java +++ /dev/null @@ -1,29 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import java.util.Map; - -@Deprecated -public class PaimonDLFExternalCatalog extends PaimonExternalCatalog { - - public PaimonDLFExternalCatalog(long catalogId, String name, String resource, - Map props, String comment) { - super(catalogId, name, resource, props, comment); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java deleted file mode 100644 index 75e86769cd980a..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java +++ /dev/null @@ -1,192 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.catalog.Env; -import org.apache.doris.common.DdlException; -import org.apache.doris.datasource.CatalogProperty; -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.InitCatalogLog; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.SessionContext; -import org.apache.doris.datasource.metacache.CacheSpec; -import org.apache.doris.datasource.property.metastore.AbstractPaimonProperties; - -import org.apache.commons.lang3.exception.ExceptionUtils; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.Identifier; -import org.apache.paimon.partition.Partition; - -import java.util.ArrayList; -import java.util.List; -import java.util.Map; - -// The subclasses of this class are all deprecated, only for meta persistence compatibility. -public class PaimonExternalCatalog extends ExternalCatalog { - private static final Logger LOG = LogManager.getLogger(PaimonExternalCatalog.class); - public static final String PAIMON_CATALOG_TYPE = "paimon.catalog.type"; - public static final String PAIMON_FILESYSTEM = "filesystem"; - public static final String PAIMON_HMS = "hms"; - public static final String PAIMON_DLF = "dlf"; - public static final String PAIMON_REST = "rest"; - public static final String PAIMON_JDBC = "jdbc"; - public static final String PAIMON_TABLE_CACHE_ENABLE = "meta.cache.paimon.table.enable"; - public static final String PAIMON_TABLE_CACHE_TTL_SECOND = "meta.cache.paimon.table.ttl-second"; - public static final String PAIMON_TABLE_CACHE_CAPACITY = "meta.cache.paimon.table.capacity"; - protected String catalogType; - protected Catalog catalog; - - private AbstractPaimonProperties paimonProperties; - - public PaimonExternalCatalog(long catalogId, String name, String resource, Map props, - String comment) { - super(catalogId, name, InitCatalogLog.Type.PAIMON, comment); - catalogProperty = new CatalogProperty(resource, props); - } - - @Override - protected void initLocalObjectsImpl() { - paimonProperties = (AbstractPaimonProperties) catalogProperty.getMetastoreProperties(); - catalogType = paimonProperties.getPaimonCatalogType(); - catalog = createCatalog(); - initPreExecutionAuthenticator(); - metadataOps = new PaimonMetadataOps(this, catalog); - } - - @Override - protected synchronized void initPreExecutionAuthenticator() { - if (executionAuthenticator == null) { - executionAuthenticator = paimonProperties.getExecutionAuthenticator(); - } - } - - public String getCatalogType() { - makeSureInitialized(); - return catalogType; - } - - @Override - public boolean tableExist(SessionContext ctx, String dbName, String tblName) { - makeSureInitialized(); - return metadataOps.tableExist(dbName, tblName); - } - - @Override - protected List listTableNamesFromRemote(SessionContext ctx, String dbName) { - return metadataOps.listTableNames(dbName); - } - - public List getPaimonPartitions(NameMapping nameMapping) { - makeSureInitialized(); - try { - return executionAuthenticator.execute(() -> { - List partitions = new ArrayList<>(); - try { - partitions = catalog.listPartitions(Identifier.create(nameMapping.getRemoteDbName(), - nameMapping.getRemoteTblName())); - } catch (Catalog.TableNotExistException e) { - LOG.warn("TableNotExistException", e); - } - return partitions; - }); - } catch (Exception e) { - throw new RuntimeException("Failed to get Paimon table partitions:" + getName() + "." - + nameMapping.getRemoteDbName() + "." + nameMapping.getRemoteTblName() + ", because " - + ExceptionUtils.getRootCauseMessage(e), e); - } - } - - public org.apache.paimon.table.Table getPaimonTable(NameMapping nameMapping) { - return getPaimonTable(nameMapping, null, null); - } - - public org.apache.paimon.table.Table getPaimonTable(NameMapping nameMapping, String branch, - String queryType) { - makeSureInitialized(); - try { - Identifier identifier; - if (branch != null && queryType != null) { - identifier = new Identifier(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), - branch, queryType); - } else if (branch != null) { - identifier = new Identifier(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), - branch); - } else if (queryType != null) { - identifier = new Identifier(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), - "main", queryType); - } else { - identifier = new Identifier(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName()); - } - return executionAuthenticator.execute(() -> catalog.getTable(identifier)); - } catch (Exception e) { - throw new RuntimeException("Failed to get Paimon table:" + getName() + "." - + nameMapping.getRemoteDbName() + "." + nameMapping.getRemoteTblName() + "$" + queryType - + ", because " + ExceptionUtils.getRootCauseMessage(e), e); - } - } - - protected Catalog createCatalog() { - try { - return paimonProperties.initializeCatalog(getName(), new ArrayList<>(catalogProperty - .getOrderedStoragePropertiesList())); - } catch (Exception e) { - throw new RuntimeException("Failed to create catalog, catalog name: " + getName() + ", exception: " - + ExceptionUtils.getRootCauseMessage(e), e); - } - } - - public Map getPaimonOptionsMap() { - makeSureInitialized(); - return paimonProperties.getCatalogOptionsMap(); - } - - @Override - public void checkProperties() throws DdlException { - super.checkProperties(); - CacheSpec.checkBooleanProperty(catalogProperty.getOrDefault(PAIMON_TABLE_CACHE_ENABLE, null), - PAIMON_TABLE_CACHE_ENABLE); - CacheSpec.checkLongProperty(catalogProperty.getOrDefault(PAIMON_TABLE_CACHE_TTL_SECOND, null), - -1L, PAIMON_TABLE_CACHE_TTL_SECOND); - CacheSpec.checkLongProperty(catalogProperty.getOrDefault(PAIMON_TABLE_CACHE_CAPACITY, null), - 0L, PAIMON_TABLE_CACHE_CAPACITY); - catalogProperty.checkMetaStoreAndStorageProperties(AbstractPaimonProperties.class); - } - - @Override - public void notifyPropertiesUpdated(Map updatedProps) { - super.notifyPropertiesUpdated(updatedProps); - if (updatedProps.keySet().stream() - .anyMatch(key -> CacheSpec.isMetaCacheKeyForEngine(key, PaimonExternalMetaCache.ENGINE))) { - Env.getCurrentEnv().getExtMetaCacheMgr().removeCatalogByEngine(getId(), PaimonExternalMetaCache.ENGINE); - } - } - - @Override - public void onClose() { - super.onClose(); - if (null != catalog) { - try { - catalog.close(); - } catch (Exception e) { - LOG.warn("Failed to close paimon catalog: {}", getName(), e); - } - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalogFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalogFactory.java deleted file mode 100644 index affe4995f107c2..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalogFactory.java +++ /dev/null @@ -1,48 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.common.DdlException; -import org.apache.doris.datasource.ExternalCatalog; - -import org.apache.commons.lang3.StringUtils; - -import java.util.Map; - -public class PaimonExternalCatalogFactory { - - public static ExternalCatalog createCatalog(long catalogId, String name, String resource, Map props, - String comment) throws DdlException { - String metastoreType = props.get(PaimonExternalCatalog.PAIMON_CATALOG_TYPE); - if (StringUtils.isEmpty(metastoreType)) { - metastoreType = PaimonExternalCatalog.PAIMON_FILESYSTEM; - } - metastoreType = metastoreType.toLowerCase(); - switch (metastoreType) { - case PaimonExternalCatalog.PAIMON_HMS: - case PaimonExternalCatalog.PAIMON_FILESYSTEM: - case PaimonExternalCatalog.PAIMON_DLF: - case PaimonExternalCatalog.PAIMON_REST: - case PaimonExternalCatalog.PAIMON_JDBC: - return new PaimonExternalCatalog(catalogId, name, resource, props, comment); - default: - throw new DdlException("Unknown " + PaimonExternalCatalog.PAIMON_CATALOG_TYPE - + " value: " + metastoreType); - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalDatabase.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalDatabase.java deleted file mode 100644 index fdbad45c5d0af9..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalDatabase.java +++ /dev/null @@ -1,37 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.ExternalDatabase; -import org.apache.doris.datasource.InitDatabaseLog; - -public class PaimonExternalDatabase extends ExternalDatabase { - - public PaimonExternalDatabase(ExternalCatalog extCatalog, Long id, String name, String remoteName) { - super(extCatalog, id, name, remoteName, InitDatabaseLog.Type.PAIMON); - } - - @Override - public PaimonExternalTable buildTableInternal(String remoteTableName, String localTableName, long tblId, - ExternalCatalog catalog, - ExternalDatabase db) { - return new PaimonExternalTable(tblId, localTableName, remoteTableName, (PaimonExternalCatalog) extCatalog, - (PaimonExternalDatabase) db); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java deleted file mode 100644 index 1d08ba1274e256..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java +++ /dev/null @@ -1,116 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.datasource.CacheException; -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.datasource.metacache.AbstractExternalMetaCache; -import org.apache.doris.datasource.metacache.MetaCacheEntryDef; -import org.apache.doris.datasource.metacache.MetaCacheEntryInvalidation; -import org.apache.doris.datasource.metacache.paimon.PaimonLatestSnapshotProjectionLoader; -import org.apache.doris.datasource.metacache.paimon.PaimonPartitionInfoLoader; -import org.apache.doris.datasource.metacache.paimon.PaimonTableLoader; - -import org.apache.paimon.table.Table; - -import java.util.Map; -import java.util.concurrent.ExecutorService; - -/** - * Paimon engine implementation of {@link AbstractExternalMetaCache}. - * - *

Registered entries: - *

    - *
  • {@code table}: loaded Paimon table handle per table mapping
    • - *
  • {@code schema}: schema cache keyed by table identity + schema id
    • - *
- * - *

Latest snapshot metadata is modeled as a runtime projection memoized inside the table cache - * value instead of as an independent cache entry. - * - *

Invalidation behavior: - *

    - *
  • db/table invalidation clears table and schema entries by matching local names
    • - *
  • partition-level invalidation falls back to table-level invalidation
    • - *
- */ -public class PaimonExternalMetaCache extends AbstractExternalMetaCache { - public static final String ENGINE = "paimon"; - public static final String ENTRY_TABLE = "table"; - public static final String ENTRY_SCHEMA = "schema"; - - private final EntryHandle tableEntry; - private final EntryHandle schemaEntry; - private final PaimonTableLoader tableLoader; - private final PaimonLatestSnapshotProjectionLoader latestSnapshotProjectionLoader; - - public PaimonExternalMetaCache(ExecutorService refreshExecutor) { - super(ENGINE, refreshExecutor); - tableLoader = new PaimonTableLoader(); - latestSnapshotProjectionLoader = new PaimonLatestSnapshotProjectionLoader( - new PaimonPartitionInfoLoader(tableLoader), this::getPaimonSchemaCacheValue); - tableEntry = registerEntry(MetaCacheEntryDef.of(ENTRY_TABLE, NameMapping.class, PaimonTableCacheValue.class, - this::loadTableCacheValue, defaultEntryCacheSpec(), - MetaCacheEntryInvalidation.forNameMapping(nameMapping -> nameMapping))); - schemaEntry = registerEntry(MetaCacheEntryDef.of(ENTRY_SCHEMA, PaimonSchemaCacheKey.class, - SchemaCacheValue.class, this::loadSchemaCacheValue, defaultSchemaCacheSpec(), - MetaCacheEntryInvalidation.forNameMapping(PaimonSchemaCacheKey::getNameMapping))); - } - - public Table getPaimonTable(ExternalTable dorisTable) { - NameMapping nameMapping = dorisTable.getOrBuildNameMapping(); - return tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getPaimonTable(); - } - - public Table getPaimonTable(NameMapping nameMapping) { - return tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getPaimonTable(); - } - - public PaimonSnapshotCacheValue getSnapshotCache(ExternalTable dorisTable) { - NameMapping nameMapping = dorisTable.getOrBuildNameMapping(); - return tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getLatestSnapshotCacheValue(); - } - - public PaimonSchemaCacheValue getPaimonSchemaCacheValue(NameMapping nameMapping, long schemaId) { - SchemaCacheValue schemaCacheValue = schemaEntry.get(nameMapping.getCtlId()) - .get(new PaimonSchemaCacheKey(nameMapping, schemaId)); - return (PaimonSchemaCacheValue) schemaCacheValue; - } - - private PaimonTableCacheValue loadTableCacheValue(NameMapping nameMapping) { - Table paimonTable = tableLoader.load(nameMapping); - return new PaimonTableCacheValue(paimonTable, - () -> latestSnapshotProjectionLoader.load(nameMapping, paimonTable)); - } - - private SchemaCacheValue loadSchemaCacheValue(PaimonSchemaCacheKey key) { - ExternalTable dorisTable = findExternalTable(key.getNameMapping(), ENGINE); - return dorisTable.initSchemaAndUpdateTime(key).orElseThrow(() -> - new CacheException("failed to load paimon schema cache value for: %s.%s.%s, schemaId: %s", - null, key.getNameMapping().getCtlId(), key.getNameMapping().getLocalDbName(), - key.getNameMapping().getLocalTblName(), key.getSchemaId())); - } - - @Override - protected Map catalogPropertyCompatibilityMap() { - return singleCompatibilityMap(ExternalCatalog.SCHEMA_CACHE_TTL_SECOND, ENTRY_SCHEMA); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java deleted file mode 100644 index 1775a984f2cd5f..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java +++ /dev/null @@ -1,429 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.analysis.TableScanParams; -import org.apache.doris.analysis.TableSnapshot; -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.MTMV; -import org.apache.doris.catalog.PartitionItem; -import org.apache.doris.catalog.PartitionType; -import org.apache.doris.common.AnalysisException; -import org.apache.doris.common.DdlException; -import org.apache.doris.datasource.CacheException; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.SchemaCacheKey; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.datasource.mvcc.MvccSnapshot; -import org.apache.doris.datasource.mvcc.MvccTable; -import org.apache.doris.datasource.mvcc.MvccUtil; -import org.apache.doris.datasource.systable.PaimonSysTable; -import org.apache.doris.datasource.systable.SysTable; -import org.apache.doris.mtmv.MTMVBaseTableIf; -import org.apache.doris.mtmv.MTMVRefreshContext; -import org.apache.doris.mtmv.MTMVRelatedTableIf; -import org.apache.doris.mtmv.MTMVSnapshotIdSnapshot; -import org.apache.doris.mtmv.MTMVSnapshotIf; -import org.apache.doris.mtmv.MTMVTimestampSnapshot; -import org.apache.doris.statistics.AnalysisInfo; -import org.apache.doris.statistics.BaseAnalysisTask; -import org.apache.doris.statistics.ExternalAnalysisTask; -import org.apache.doris.thrift.THiveTable; -import org.apache.doris.thrift.TTableDescriptor; -import org.apache.doris.thrift.TTableType; - -import com.google.common.collect.Lists; -import com.google.common.collect.Maps; -import com.google.common.collect.Sets; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.apache.paimon.CoreOptions; -import org.apache.paimon.Snapshot; -import org.apache.paimon.partition.Partition; -import org.apache.paimon.schema.TableSchema; -import org.apache.paimon.table.DataTable; -import org.apache.paimon.table.Table; -import org.apache.paimon.table.source.Split; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.DataTypeRoot; - -import java.util.Collections; -import java.util.HashMap; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.Set; -import java.util.stream.Collectors; - -public class PaimonExternalTable extends ExternalTable implements MTMVRelatedTableIf, MTMVBaseTableIf, MvccTable { - - private static final Logger LOG = LogManager.getLogger(PaimonExternalTable.class); - - public PaimonExternalTable(long id, String name, String remoteName, PaimonExternalCatalog catalog, - PaimonExternalDatabase db) { - super(id, name, remoteName, catalog, db, TableType.PAIMON_EXTERNAL_TABLE); - } - - @Override - public String getMetaCacheEngine() { - return PaimonExternalMetaCache.ENGINE; - } - - public String getPaimonCatalogType() { - return ((PaimonExternalCatalog) catalog).getCatalogType(); - } - - protected synchronized void makeSureInitialized() { - super.makeSureInitialized(); - if (!objectCreated) { - objectCreated = true; - } - } - - public Table getPaimonTable(Optional snapshot) { - if (snapshot.isPresent()) { - // MTMV scenario: get from snapshot cache - return getOrFetchSnapshotCacheValue(snapshot).getSnapshot().getTable(); - } else { - // Normal query scenario: get directly from table cache - return PaimonUtils.getPaimonTable(this); - } - } - - private PaimonSnapshotCacheValue getPaimonSnapshotCacheValue(Optional tableSnapshot, - Optional scanParams) { - makeSureInitialized(); - - // Current limitation: cannot specify both table snapshot and scan parameters simultaneously. - if (tableSnapshot.isPresent() || (scanParams.isPresent() && scanParams.get().isTag())) { - // If a snapshot is specified, - // use the specified snapshot and the corresponding schema(not the latest - // schema). - try { - Table baseTable = getBasePaimonTable(); - DataTable dataTable = (DataTable) baseTable; - Snapshot snapshot; - Map scanOptions = new HashMap<>(); - - if (tableSnapshot.isPresent()) { - TableSnapshot snapshotOpt = tableSnapshot.get(); - String value = snapshotOpt.getValue(); - if (snapshotOpt.getType() == TableSnapshot.VersionType.TIME) { - snapshot = PaimonUtil.getPaimonSnapshotByTimestamp( - dataTable, value, PaimonUtil.isDigitalString(value)); - scanOptions.put(CoreOptions.SCAN_SNAPSHOT_ID.key(), String.valueOf(snapshot.id())); - } else { - if (PaimonUtil.isDigitalString(value)) { - snapshot = PaimonUtil.getPaimonSnapshotBySnapshotId(dataTable, value); - scanOptions.put(CoreOptions.SCAN_SNAPSHOT_ID.key(), String.valueOf(snapshot.id())); - } else { - snapshot = PaimonUtil.getPaimonSnapshotByTag(dataTable, value); - scanOptions.put(CoreOptions.SCAN_TAG_NAME.key(), value); - } - } - } else { - String tagName = PaimonUtil.extractBranchOrTagName(scanParams.get()); - snapshot = PaimonUtil.getPaimonSnapshotByTag(dataTable, tagName); - scanOptions.put(CoreOptions.SCAN_TAG_NAME.key(), tagName); - } - - Table scanTable = baseTable.copy(scanOptions); - return new PaimonSnapshotCacheValue(PaimonPartitionInfo.EMPTY, - new PaimonSnapshot(snapshot.id(), snapshot.schemaId(), scanTable)); - } catch (Exception e) { - LOG.warn("Failed to get Paimon snapshot for table {}", getOrBuildNameMapping().getFullLocalName(), e); - throw new RuntimeException( - "Failed to get Paimon snapshot: " + (e.getMessage() == null ? "unknown cause" : e.getMessage()), - e); - } - } else if (scanParams.isPresent() && scanParams.get().isBranch()) { - try { - Table baseTable = getBasePaimonTable(); - String branch = PaimonUtil.resolvePaimonBranch(scanParams.get(), baseTable); - Table table = ((PaimonExternalCatalog) catalog).getPaimonTable(getOrBuildNameMapping(), branch, null); - Optional latestSnapshot = table.latestSnapshot(); - long latestSnapshotId = PaimonSnapshot.INVALID_SNAPSHOT_ID; - if (latestSnapshot.isPresent()) { - latestSnapshotId = latestSnapshot.get().id(); - } - // Branches in Paimon can have independent schemas and snapshots. - // TODO: Add time travel support for paimon branch tables. - DataTable dataTable = (DataTable) table; - Long schemaId = dataTable.schemaManager().latest().map(TableSchema::id).orElse(0L); - return new PaimonSnapshotCacheValue(PaimonPartitionInfo.EMPTY, - new PaimonSnapshot(latestSnapshotId, schemaId, dataTable)); - } catch (Exception e) { - LOG.warn("Failed to get Paimon branch for table {}", getOrBuildNameMapping().getFullLocalName(), e); - throw new RuntimeException( - "Failed to get Paimon branch: " + (e.getMessage() == null ? "unknown cause" : e.getMessage()), - e); - } - } else { - // Otherwise, use the latest snapshot and the latest schema. - return PaimonUtils.getLatestSnapshotCacheValue(this); - } - } - - @Override - public TTableDescriptor toThrift() { - List schema = getFullSchema(); - if (PaimonExternalCatalog.PAIMON_HMS.equals(getPaimonCatalogType()) - || PaimonExternalCatalog.PAIMON_FILESYSTEM.equals(getPaimonCatalogType()) - || PaimonExternalCatalog.PAIMON_DLF.equals(getPaimonCatalogType()) - || PaimonExternalCatalog.PAIMON_REST.equals(getPaimonCatalogType()) - || PaimonExternalCatalog.PAIMON_JDBC.equals(getPaimonCatalogType())) { - THiveTable tHiveTable = new THiveTable(dbName, name, new HashMap<>()); - TTableDescriptor tTableDescriptor = new TTableDescriptor(getId(), TTableType.HIVE_TABLE, schema.size(), 0, - getName(), dbName); - tTableDescriptor.setHiveTable(tHiveTable); - return tTableDescriptor; - } else { - throw new IllegalArgumentException( - "Currently only supports hms/dlf/rest/filesystem/jdbc catalog, do not support: " - + getPaimonCatalogType()); - } - } - - @Override - public BaseAnalysisTask createAnalysisTask(AnalysisInfo info) { - makeSureInitialized(); - return new ExternalAnalysisTask(info); - } - - @Override - public long fetchRowCount() { - makeSureInitialized(); - long rowCount = 0; - List splits = getBasePaimonTable().newReadBuilder().newScan().plan().splits(); - for (Split split : splits) { - rowCount += split.rowCount(); - } - if (rowCount == 0) { - LOG.info("Paimon table {} row count is 0, return -1", name); - } - return rowCount > 0 ? rowCount : UNKNOWN_ROW_COUNT; - } - - @Override - public void beforeMTMVRefresh(MTMV mtmv) throws DdlException { - } - - @Override - public Map getAndCopyPartitionItems(Optional snapshot) { - return Maps.newHashMap(getNameToPartitionItems(snapshot)); - } - - @Override - public PartitionType getPartitionType(Optional snapshot) { - if (isPartitionInvalid(snapshot)) { - return PartitionType.UNPARTITIONED; - } - return getPartitionColumns(snapshot).size() > 0 ? PartitionType.LIST : PartitionType.UNPARTITIONED; - } - - @Override - public Set getPartitionColumnNames(Optional snapshot) { - return getPartitionColumns(snapshot).stream() - .map(c -> c.getName().toLowerCase()).collect(Collectors.toSet()); - } - - @Override - public List getPartitionColumns(Optional snapshot) { - if (isPartitionInvalid(snapshot)) { - return Collections.emptyList(); - } - return getPaimonSchemaCacheValue(snapshot).getPartitionColumns(); - } - - public boolean isPartitionInvalid(Optional snapshot) { - PaimonSnapshotCacheValue paimonSnapshotCacheValue = getOrFetchSnapshotCacheValue(snapshot); - return paimonSnapshotCacheValue.getPartitionInfo().isPartitionInvalid(); - } - - @Override - public MTMVSnapshotIf getPartitionSnapshot(String partitionName, MTMVRefreshContext context, - Optional snapshot) - throws AnalysisException { - Partition paimonPartition = getOrFetchSnapshotCacheValue(snapshot).getPartitionInfo().getNameToPartition() - .get(partitionName); - if (paimonPartition == null) { - throw new AnalysisException("can not find partition: " + partitionName); - } - return new MTMVTimestampSnapshot(paimonPartition.lastFileCreationTime()); - } - - @Override - public MTMVSnapshotIf getTableSnapshot(MTMVRefreshContext context, Optional snapshot) - throws AnalysisException { - return getTableSnapshot(snapshot); - } - - public Map getPartitionSnapshot( - Optional snapshot) { - - return getOrFetchSnapshotCacheValue(snapshot).getPartitionInfo() - .getNameToPartition(); - } - - @Override - public MTMVSnapshotIf getTableSnapshot(Optional snapshot) throws AnalysisException { - PaimonSnapshotCacheValue paimonSnapshot = getOrFetchSnapshotCacheValue(snapshot); - return new MTMVSnapshotIdSnapshot(paimonSnapshot.getSnapshot().getSnapshotId()); - } - - @Override - public long getNewestUpdateVersionOrTime() { - return getPaimonSnapshotCacheValue(Optional.empty(), Optional.empty()).getPartitionInfo().getNameToPartition() - .values().stream() - .mapToLong(Partition::lastFileCreationTime).max().orElse(0); - } - - @Override - public boolean isPartitionColumnAllowNull() { - // Paimon will write to the 'null' partition regardless of whether it is' null or 'null'. - // The logic is inconsistent with Doris' empty partition logic, so it needs to return false. - // However, when Spark creates Paimon tables, specifying 'not null' does not take effect. - // In order to successfully create the materialized view, false is returned here. - // The cost is that Paimon partition writes a null value, and the materialized view cannot detect this data. - return true; - } - - @Override - public MvccSnapshot loadSnapshot(Optional tableSnapshot, Optional scanParams) { - return new PaimonMvccSnapshot(getPaimonSnapshotCacheValue(tableSnapshot, scanParams)); - } - - @Override - public Map getNameToPartitionItems(Optional snapshot) { - return getOrFetchSnapshotCacheValue(snapshot).getPartitionInfo().getNameToPartitionItem(); - } - - @Override - public boolean supportInternalPartitionPruned() { - return true; - } - - @Override - public boolean supportsExternalMetadataPreload() { - return true; - } - - @Override - public boolean supportsLatestSnapshotPreload() { - return true; - } - - @Override - public List getFullSchema() { - return getPaimonSchemaCacheValue(MvccUtil.getSnapshotFromContext(this)).getSchema(); - } - - @Override - public Optional initSchema(SchemaCacheKey key) { - makeSureInitialized(); - PaimonSchemaCacheKey paimonSchemaCacheKey = (PaimonSchemaCacheKey) key; - try { - Table table = getBasePaimonTable(); - TableSchema tableSchema = ((DataTable) table).schemaManager().schema(paimonSchemaCacheKey.getSchemaId()); - List columns = tableSchema.fields(); - List dorisColumns = Lists.newArrayListWithCapacity(columns.size()); - Set partitionColumnNames = Sets.newHashSet(tableSchema.partitionKeys()); - List partitionColumns = Lists.newArrayList(); - for (DataField field : columns) { - Column column = new Column(field.name().toLowerCase(), - PaimonUtil.paimonTypeToDorisType(field.type(), getCatalog().getEnableMappingVarbinary(), - getCatalog().getEnableMappingTimestampTz()), - true, - null, true, field.description(), true, - -1); - PaimonUtil.updatePaimonColumnUniqueId(column, field); - if (field.type().getTypeRoot() == DataTypeRoot.TIMESTAMP_WITH_LOCAL_TIME_ZONE) { - column.setWithTZExtraInfo(); - } - dorisColumns.add(column); - if (partitionColumnNames.contains(field.name())) { - partitionColumns.add(column); - } - } - return Optional.of(new PaimonSchemaCacheValue(dorisColumns, partitionColumns, tableSchema)); - } catch (Exception e) { - throw new CacheException("failed to initSchema for: %s.%s.%s.%s", - null, getCatalog().getName(), key.getNameMapping().getLocalDbName(), - key.getNameMapping().getLocalTblName(), - paimonSchemaCacheKey.getSchemaId()); - } - } - - @Override - public Optional getSchemaCacheValue() { - return Optional.of(getPaimonSchemaCacheValue(MvccUtil.getSnapshotFromContext(this))); - } - - private PaimonSchemaCacheValue getPaimonSchemaCacheValue(Optional snapshot) { - PaimonSnapshotCacheValue snapshotCacheValue = getOrFetchSnapshotCacheValue(snapshot); - return PaimonUtils.getSchemaCacheValue(this, snapshotCacheValue); - } - - private PaimonSnapshotCacheValue getOrFetchSnapshotCacheValue(Optional snapshot) { - if (snapshot.isPresent()) { - return ((PaimonMvccSnapshot) snapshot.get()).getSnapshotCacheValue(); - } else { - // Use new lazy-loading snapshot cache API - return PaimonUtils.getSnapshotCacheValue(snapshot, this); - } - } - - @Override - public Map getSupportedSysTables() { - makeSureInitialized(); - return PaimonSysTable.SUPPORTED_SYS_TABLES; - } - - @Override - public String getComment() { - Table table = getBasePaimonTable(); - return table.comment().isPresent() ? table.comment().get() : ""; - } - - public Map getTableProperties() { - Table table = getBasePaimonTable(); - if (table instanceof DataTable) { - DataTable dataTable = (DataTable) table; - Map properties = new LinkedHashMap<>(dataTable.coreOptions().toMap()); - - if (!dataTable.primaryKeys().isEmpty()) { - properties.put(CoreOptions.PRIMARY_KEY.key(), String.join(",", dataTable.primaryKeys())); - } - - return properties; - } else { - return Collections.emptyMap(); - } - } - - @Override - public boolean isPartitionedTable() { - makeSureInitialized(); - return !getBasePaimonTable().partitionKeys().isEmpty(); - } - - private Table getBasePaimonTable() { - return PaimonUtils.getPaimonTable(this); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonFileExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonFileExternalCatalog.java deleted file mode 100644 index 9bc233f4b05f15..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonFileExternalCatalog.java +++ /dev/null @@ -1,29 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import java.util.Map; - -@Deprecated -public class PaimonFileExternalCatalog extends PaimonExternalCatalog { - - public PaimonFileExternalCatalog(long catalogId, String name, String resource, - Map props, String comment) { - super(catalogId, name, resource, props, comment); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java deleted file mode 100644 index 0a4702ab4f1cd1..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonHMSExternalCatalog.java +++ /dev/null @@ -1,29 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import java.util.Map; - -@Deprecated -public class PaimonHMSExternalCatalog extends PaimonExternalCatalog { - - public PaimonHMSExternalCatalog(long catalogId, String name, String resource, - Map props, String comment) { - super(catalogId, name, resource, props, comment); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java deleted file mode 100644 index b8263250c3dafd..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java +++ /dev/null @@ -1,405 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.analysis.PartitionDesc; -import org.apache.doris.catalog.StructField; -import org.apache.doris.catalog.StructType; -import org.apache.doris.catalog.Type; -import org.apache.doris.catalog.info.CreateOrReplaceBranchInfo; -import org.apache.doris.catalog.info.CreateOrReplaceTagInfo; -import org.apache.doris.catalog.info.DropBranchInfo; -import org.apache.doris.catalog.info.DropTagInfo; -import org.apache.doris.common.DdlException; -import org.apache.doris.common.ErrorCode; -import org.apache.doris.common.ErrorReport; -import org.apache.doris.common.UserException; -import org.apache.doris.common.security.authentication.ExecutionAuthenticator; -import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.DorisTypeVisitor; -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.ExternalDatabase; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.operations.ExternalMetadataOps; -import org.apache.doris.nereids.trees.plans.commands.info.ColumnDefinition; -import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; - -import org.apache.commons.lang3.exception.ExceptionUtils; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.apache.paimon.CoreOptions; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.Catalog.DatabaseNotEmptyException; -import org.apache.paimon.catalog.Catalog.DatabaseNotExistException; -import org.apache.paimon.catalog.Catalog.TableAlreadyExistException; -import org.apache.paimon.catalog.Catalog.TableNotExistException; -import org.apache.paimon.catalog.Identifier; -import org.apache.paimon.schema.Schema; -import org.apache.paimon.types.DataType; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.stream.Collectors; - -public class PaimonMetadataOps implements ExternalMetadataOps { - - private static final Logger LOG = LogManager.getLogger(PaimonMetadataOps.class); - protected Catalog catalog; - protected ExternalCatalog dorisCatalog; - private ExecutionAuthenticator executionAuthenticator; - private static final String PRIMARY_KEY_IDENTIFIER = "primary-key"; - private static final String PROP_COMMENT = "comment"; - private static final String PROP_LOCATION = "location"; - - public PaimonMetadataOps(ExternalCatalog dorisCatalog, Catalog catalog) { - this.dorisCatalog = dorisCatalog; - this.catalog = catalog; - this.executionAuthenticator = dorisCatalog.getExecutionAuthenticator(); - } - - - @Override - public boolean createDbImpl(String dbName, boolean ifNotExists, Map properties) - throws DdlException { - try { - return executionAuthenticator.execute(() -> performCreateDb(dbName, ifNotExists, properties)); - } catch (Exception e) { - throw new DdlException("Failed to create database: " - + dbName + ": " + Util.getRootCauseMessage(e), e); - } - } - - private boolean performCreateDb(String dbName, boolean ifNotExists, Map properties) - throws DdlException, Catalog.DatabaseAlreadyExistException { - if (databaseExist(dbName)) { - if (ifNotExists) { - LOG.info("create database[{}] which already exists", dbName); - return true; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_DB_CREATE_EXISTS, dbName); - } - } - - if (!properties.isEmpty() && dorisCatalog instanceof PaimonExternalCatalog) { - String catalogType = ((PaimonExternalCatalog) dorisCatalog).getCatalogType(); - if (!PaimonExternalCatalog.PAIMON_HMS.equals(catalogType)) { - throw new DdlException( - "Not supported: create database with properties for paimon catalog type: " + catalogType); - } - } - - catalog.createDatabase(dbName, ifNotExists, properties); - return false; - } - - @Override - public void afterCreateDb() { - dorisCatalog.resetMetaCacheNames(); - } - - @Override - public void dropDbImpl(String dbName, boolean ifExists, boolean force) throws DdlException { - try { - executionAuthenticator.execute(() -> { - performDropDb(dbName, ifExists, force); - return null; - }); - } catch (Exception e) { - throw new DdlException( - "Failed to drop database: " + dbName + ", error message is:" + e.getMessage(), e); - } - } - - private void performDropDb(String dbName, boolean ifExists, boolean force) throws DdlException { - ExternalDatabase dorisDb = dorisCatalog.getDbNullable(dbName); - if (dorisDb == null) { - if (ifExists) { - LOG.info("drop database[{}] which does not exist", dbName); - // Database does not exist and IF EXISTS is specified; treat as no-op. - return; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_DB_DROP_EXISTS, dbName); - // ErrorReport.reportDdlException is expected to throw DdlException. - return; - } - } - - if (force) { - List tableNames = listTableNames(dbName); - if (!tableNames.isEmpty()) { - LOG.info("drop database[{}] with force, drop all tables, num: {}", dbName, tableNames.size()); - } - for (String tableName : tableNames) { - performDropTable(dbName, tableName, true); - } - } - - try { - catalog.dropDatabase(dbName, ifExists, force); - } catch (DatabaseNotExistException e) { - throw new RuntimeException("database " + dbName + " does not exist!"); - } catch (DatabaseNotEmptyException e) { - throw new RuntimeException("database " + dbName + " is not empty! please check!"); - } - } - - @Override - public void afterDropDb(String dbName) { - dorisCatalog.unregisterDatabase(dbName); - } - - @Override - public boolean createTableImpl(CreateTableInfo createTableInfo) throws UserException { - try { - return executionAuthenticator.execute(() -> performCreateTable(createTableInfo)); - } catch (Exception e) { - throw new DdlException( - "Failed to create table: " + createTableInfo.getTableName() + ", error message is:" + e.getMessage(), - e); - } - } - - public boolean performCreateTable(CreateTableInfo createTableInfo) throws UserException { - String dbName = createTableInfo.getDbName(); - ExternalDatabase db = dorisCatalog.getDbNullable(dbName); - if (db == null) { - throw new UserException("Failed to get database: '" + dbName + "' in catalog: " + dorisCatalog.getName()); - } - String tableName = createTableInfo.getTableName(); - // 1. first, check if table exist in remote - if (tableExist(db.getRemoteName(), tableName)) { - if (createTableInfo.isIfNotExists()) { - LOG.info("create table[{}] which already exists", tableName); - return true; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR, tableName); - } - } - - // 2. second, check if table exist in local. - // This is because case sensibility issue, eg: - // 1. lower_case_table_name = 1 - // 2. create table tbl1; - // 3. create table TBL1; TBL1 does not exist in remote because the remote system is case-sensitive. - // but because lower_case_table_name = 1, the table can not be created in Doris because it is conflict with - // tbl1 - ExternalTable dorisTable = db.getTableNullable(tableName); - if (dorisTable != null) { - if (createTableInfo.isIfNotExists()) { - LOG.info("create table[{}] which already exists", tableName); - return true; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR, tableName); - } - } - List columns = createTableInfo.getColumnDefinitions(); - List collect = columns.stream() - .map(col -> new StructField(col.getName(), col.getType().toCatalogDataType(), - col.getComment(), col.isNullable())) - .collect(Collectors.toList()); - StructType structType = new StructType(new ArrayList<>(collect)); - Schema schema = toPaimonSchema(structType, createTableInfo.getPartitionDesc(), createTableInfo.getProperties()); - try { - catalog.createTable(new Identifier(createTableInfo.getDbName(), createTableInfo.getTableName()), - schema, createTableInfo.isIfNotExists()); - } catch (TableAlreadyExistException | DatabaseNotExistException e) { - throw new RuntimeException(e); - } - return false; - } - - private Schema toPaimonSchema(StructType structType, PartitionDesc partitionDesc, Map properties) { - Map normalizedProperties = new HashMap<>(properties); - normalizedProperties.remove(PRIMARY_KEY_IDENTIFIER); - normalizedProperties.remove(PROP_COMMENT); - if (normalizedProperties.containsKey(PROP_LOCATION)) { - String path = normalizedProperties.remove(PROP_LOCATION); - normalizedProperties.put(CoreOptions.PATH.key(), path); - } - - String pkAsString = properties.get(PRIMARY_KEY_IDENTIFIER); - List primaryKeys = pkAsString == null ? Collections.emptyList() : Arrays.stream(pkAsString.split(",")) - .map(String::trim) - .collect(Collectors.toList()); - List partitionKeys = partitionDesc == null ? new ArrayList<>() : partitionDesc.getPartitionColNames(); - Schema.Builder schemaBuilder = Schema.newBuilder() - .options(normalizedProperties) - .primaryKey(primaryKeys) - .partitionKeys(partitionKeys) - .comment(properties.getOrDefault(PROP_COMMENT, null)); - for (StructField field : structType.getFields()) { - schemaBuilder.column(field.getName(), - toPaimontype(field.getType()).copy(field.getContainsNull()), - field.getComment()); - } - return schemaBuilder.build(); - } - - private DataType toPaimontype(Type type) { - return DorisTypeVisitor.visit(type, new DorisToPaimonTypeVisitor()); - } - - @Override - public void afterCreateTable(String dbName, String tblName) { - Optional> db = dorisCatalog.getDbForReplay(dbName); - if (db.isPresent()) { - db.get().resetMetaCacheNames(); - } - LOG.info("after create table {}.{}.{}, is db exists: {}", - dorisCatalog.getName(), dbName, tblName, db.isPresent()); - } - - @Override - public void dropTableImpl(ExternalTable dorisTable, boolean ifExists) throws DdlException { - try { - executionAuthenticator.execute(() -> { - performDropTable(dorisTable.getRemoteDbName(), dorisTable.getRemoteName(), ifExists); - return null; - }); - } catch (Exception e) { - throw new DdlException( - "Failed to drop table: " + dorisTable.getName() + ", error message is:" + e.getMessage(), e); - } - } - - private void performDropTable(String dBName, String tableName, boolean ifExists) throws DdlException { - if (!tableExist(dBName, tableName)) { - if (ifExists) { - LOG.info("drop table[{}] which does not exist", tableName); - return; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_UNKNOWN_TABLE, tableName, dBName); - } - } - try { - catalog.dropTable(Identifier.create(dBName, tableName), ifExists); - } catch (TableNotExistException e) { - throw new RuntimeException("table " + tableName + " does not exist"); - } - } - - @Override - public void afterDropTable(String dbName, String tblName) { - Optional> db = dorisCatalog.getDbForReplay(dbName); - db.ifPresent(externalDatabase -> externalDatabase.unregisterTable(tblName)); - LOG.info("after drop table {}.{}.{}. is db exists: {}", - dorisCatalog.getName(), dbName, tblName, db.isPresent()); - } - - @Override - public void truncateTableImpl(ExternalTable dorisTable, List partitions) throws DdlException { - throw new UnsupportedOperationException("truncate table is not a supported operation!"); - } - - @Override - public void createOrReplaceBranchImpl(ExternalTable dorisTable, CreateOrReplaceBranchInfo branchInfo) - throws UserException { - throw new UnsupportedOperationException("create or replace branch is not a supported operation!"); - } - - @Override - public void createOrReplaceTagImpl(ExternalTable dorisTable, CreateOrReplaceTagInfo tagInfo) throws UserException { - throw new UnsupportedOperationException("create or replace tag is not a supported operation!"); - } - - @Override - public void dropTagImpl(ExternalTable dorisTable, DropTagInfo tagInfo) throws UserException { - throw new UnsupportedOperationException("drop tag is not a supported operation!"); - } - - @Override - public void dropBranchImpl(ExternalTable dorisTable, DropBranchInfo branchInfo) throws UserException { - throw new UnsupportedOperationException("drop branch is not a supported operation!"); - } - - @Override - public List listDatabaseNames() { - try { - return executionAuthenticator.execute(() -> new ArrayList<>(catalog.listDatabases())); - } catch (Exception e) { - throw new RuntimeException("Failed to list databases names, catalog name: " + dorisCatalog.getName(), e); - } - } - - @Override - public List listTableNames(String db) { - try { - return executionAuthenticator.execute(() -> { - List tableNames = new ArrayList<>(); - try { - tableNames.addAll(catalog.listTables(db)); - } catch (DatabaseNotExistException e) { - LOG.warn("DatabaseNotExistException", e); - } - return tableNames; - }); - } catch (Exception e) { - throw new RuntimeException("Failed to list table names, catalog name: " + dorisCatalog.getName(), e); - } - } - - @Override - public boolean tableExist(String dbName, String tblName) { - try { - return executionAuthenticator.execute(() -> { - try { - catalog.getTable(Identifier.create(dbName, tblName)); - return true; - } catch (TableNotExistException e) { - return false; - } - }); - - } catch (Exception e) { - throw new RuntimeException("Failed to check table existence, catalog name: " + dorisCatalog.getName() - + "error message is:" + ExceptionUtils.getRootCauseMessage(e), e); - } - } - - @Override - public boolean databaseExist(String dbName) { - try { - return executionAuthenticator.execute(() -> { - try { - catalog.getDatabase(dbName); - return true; - } catch (DatabaseNotExistException e) { - return false; - } - }); - } catch (Exception e) { - throw new RuntimeException("Failed to check database exist, error message is:" + e.getMessage(), e); - } - } - - public Catalog getCatalog() { - return catalog; - } - - @Override - public void close() { - if (catalog != null) { - catalog = null; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMvccSnapshot.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMvccSnapshot.java deleted file mode 100644 index 2307e91adb3911..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMvccSnapshot.java +++ /dev/null @@ -1,32 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.datasource.mvcc.MvccSnapshot; - -public class PaimonMvccSnapshot implements MvccSnapshot { - private final PaimonSnapshotCacheValue snapshotCacheValue; - - public PaimonMvccSnapshot(PaimonSnapshotCacheValue snapshotCacheValue) { - this.snapshotCacheValue = snapshotCacheValue; - } - - public PaimonSnapshotCacheValue getSnapshotCacheValue() { - return snapshotCacheValue; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartition.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartition.java deleted file mode 100644 index 545448199b3375..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartition.java +++ /dev/null @@ -1,61 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -// https://paimon.apache.org/docs/0.9/maintenance/system-tables/#partitions-table -public class PaimonPartition { - // Partition values, for example: [1, dd] - private final String partitionValues; - // The amount of data in the partition - private final long recordCount; - // Partition file size - private final long fileSizeInBytes; - // Number of partition files - private final long fileCount; - // Last update time of partition - private final long lastUpdateTime; - - public PaimonPartition(String partitionValues, long recordCount, long fileSizeInBytes, long fileCount, - long lastUpdateTime) { - this.partitionValues = partitionValues; - this.recordCount = recordCount; - this.fileSizeInBytes = fileSizeInBytes; - this.fileCount = fileCount; - this.lastUpdateTime = lastUpdateTime; - } - - public String getPartitionValues() { - return partitionValues; - } - - public long getRecordCount() { - return recordCount; - } - - public long getFileSizeInBytes() { - return fileSizeInBytes; - } - - public long getFileCount() { - return fileCount; - } - - public long getLastUpdateTime() { - return lastUpdateTime; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartitionInfo.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartitionInfo.java deleted file mode 100644 index a6339ef5155e15..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonPartitionInfo.java +++ /dev/null @@ -1,56 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.catalog.PartitionItem; - -import com.google.common.collect.Maps; -import org.apache.paimon.partition.Partition; - -import java.util.Map; - -public class PaimonPartitionInfo { - public static final PaimonPartitionInfo EMPTY = new PaimonPartitionInfo(); - - private final Map nameToPartitionItem; - private final Map nameToPartition; - - private PaimonPartitionInfo() { - this.nameToPartitionItem = Maps.newHashMap(); - this.nameToPartition = Maps.newHashMap(); - } - - public PaimonPartitionInfo(Map nameToPartitionItem, - Map nameToPartition) { - this.nameToPartitionItem = nameToPartitionItem; - this.nameToPartition = nameToPartition; - } - - public Map getNameToPartitionItem() { - return nameToPartitionItem; - } - - public Map getNameToPartition() { - return nameToPartition; - } - - public boolean isPartitionInvalid() { - // when transfer to partitionItem failed, will not equal - return nameToPartitionItem.size() != nameToPartition.size(); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonRestExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonRestExternalCatalog.java deleted file mode 100644 index 6360a61a00d7f6..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonRestExternalCatalog.java +++ /dev/null @@ -1,29 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import java.util.Map; - -@Deprecated -public class PaimonRestExternalCatalog extends PaimonExternalCatalog { - - public PaimonRestExternalCatalog(long catalogId, String name, String resource, - Map props, String comment) { - super(catalogId, name, resource, props, comment); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheKey.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheKey.java deleted file mode 100644 index 4eccb269c2fe56..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheKey.java +++ /dev/null @@ -1,56 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.SchemaCacheKey; - -import com.google.common.base.Objects; - -public class PaimonSchemaCacheKey extends SchemaCacheKey { - private final long schemaId; - - public PaimonSchemaCacheKey(NameMapping nameMapping, long schemaId) { - super(nameMapping); - this.schemaId = schemaId; - } - - public long getSchemaId() { - return schemaId; - } - - @Override - public boolean equals(Object o) { - if (this == o) { - return true; - } - if (!(o instanceof PaimonSchemaCacheKey)) { - return false; - } - if (!super.equals(o)) { - return false; - } - PaimonSchemaCacheKey that = (PaimonSchemaCacheKey) o; - return schemaId == that.schemaId; - } - - @Override - public int hashCode() { - return Objects.hashCode(super.hashCode(), schemaId); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheValue.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheValue.java deleted file mode 100644 index e931b52336ba8f..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSchemaCacheValue.java +++ /dev/null @@ -1,47 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.catalog.Column; -import org.apache.doris.datasource.SchemaCacheValue; - -import org.apache.paimon.schema.TableSchema; - -import java.util.List; - -public class PaimonSchemaCacheValue extends SchemaCacheValue { - - private List partitionColumns; - - private TableSchema tableSchema; - // Caching TableSchema can reduce the reading of schema files and json parsing. - - public PaimonSchemaCacheValue(List schema, List partitionColumns, TableSchema tableSchema) { - super(schema); - this.partitionColumns = partitionColumns; - this.tableSchema = tableSchema; - } - - public List getPartitionColumns() { - return partitionColumns; - } - - public TableSchema getTableSchema() { - return tableSchema; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshot.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshot.java deleted file mode 100644 index 96f32370d999b7..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshot.java +++ /dev/null @@ -1,45 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.paimon.table.Table; - -public class PaimonSnapshot { - public static long INVALID_SNAPSHOT_ID = -1; - private final long snapshotId; - private final long schemaId; - private final Table table; - - public PaimonSnapshot(long snapshotId, long schemaId, Table table) { - this.snapshotId = snapshotId; - this.schemaId = schemaId; - this.table = table; - } - - public long getSnapshotId() { - return snapshotId; - } - - public long getSchemaId() { - return schemaId; - } - - public Table getTable() { - return table; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshotCacheValue.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshotCacheValue.java deleted file mode 100644 index c50ecdabfde3df..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSnapshotCacheValue.java +++ /dev/null @@ -1,37 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -public class PaimonSnapshotCacheValue { - - private final PaimonPartitionInfo partitionInfo; - private final PaimonSnapshot snapshot; - - public PaimonSnapshotCacheValue(PaimonPartitionInfo partitionInfo, PaimonSnapshot snapshot) { - this.partitionInfo = partitionInfo; - this.snapshot = snapshot; - } - - public PaimonPartitionInfo getPartitionInfo() { - return partitionInfo; - } - - public PaimonSnapshot getSnapshot() { - return snapshot; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java deleted file mode 100644 index b6999b7c50c558..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java +++ /dev/null @@ -1,277 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.TableIf; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.SchemaCacheKey; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.datasource.systable.SysTable; -import org.apache.doris.statistics.AnalysisInfo; -import org.apache.doris.statistics.BaseAnalysisTask; -import org.apache.doris.statistics.ExternalAnalysisTask; -import org.apache.doris.thrift.THiveTable; -import org.apache.doris.thrift.TTableDescriptor; -import org.apache.doris.thrift.TTableType; - -import com.google.common.collect.Lists; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.apache.paimon.table.DataTable; -import org.apache.paimon.table.Table; -import org.apache.paimon.table.source.Split; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.DataTypeRoot; - -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Optional; - -/** - * Represents a Paimon system table (e.g., snapshots, binlog, audit_log) that wraps a source data table. - * - *

This class enables system tables to be queried using the native table execution path - * (FileQueryScanNode) instead of the TVF path (MetadataScanNode). This provides: - *

    - *
  • Unified execution path with regular tables
    • - *
  • Native vectorized reading for data-oriented system tables
    • - *
  • Better integration with query optimization
    • - *
- * - *

System tables are classified into two categories: - *

    - *
  • Data tables (e.g., binlog, audit_log, ro): Read actual ORC/Parquet data files
    • - *
  • Metadata tables (snapshots, partitions, etc.): Read metadata/manifest files
    • - *
- */ -public class PaimonSysExternalTable extends ExternalTable { - - private static final Logger LOG = LogManager.getLogger(PaimonSysExternalTable.class); - - private final PaimonExternalTable sourceTable; - private final String sysTableType; - private volatile Boolean isDataTable; - private volatile Table paimonSysTable; - private volatile List fullSchema; - private volatile SchemaCacheValue schemaCacheValue; - - /** - * Creates a new Paimon system external table. - * - * @param sourceTable the underlying data table being wrapped - * @param sysTableType the type of system table (e.g., "snapshots", "binlog") - */ - public PaimonSysExternalTable(PaimonExternalTable sourceTable, String sysTableType) { - super(generateSysTableId(sourceTable.getId(), sysTableType), - sourceTable.getName() + "$" + sysTableType, - sourceTable.getRemoteName() + "$" + sysTableType, - (PaimonExternalCatalog) sourceTable.getCatalog(), - (PaimonExternalDatabase) sourceTable.getDatabase(), - TableIf.TableType.PAIMON_EXTERNAL_TABLE); - this.sourceTable = sourceTable; - this.sysTableType = sysTableType; - } - - @Override - public String getMetaCacheEngine() { - return PaimonExternalMetaCache.ENGINE; - } - - @Override - protected synchronized void makeSureInitialized() { - super.makeSureInitialized(); - if (!objectCreated) { - objectCreated = true; - } - } - - /** - * Generate a unique ID for the system table based on source table ID and system table type. - */ - private static long generateSysTableId(long sourceTableId, String sysTableType) { - // Use a simple hash combination to generate a unique ID - return sourceTableId ^ (sysTableType.hashCode() * 31L); - } - - /** - * Returns the Paimon system table instance (e.g., snapshots, binlog). - * Note: system tables currently ignore snapshot semantics. - */ - public Table getSysPaimonTable() { - if (paimonSysTable == null) { - synchronized (this) { - if (paimonSysTable == null) { - PaimonExternalCatalog catalog = (PaimonExternalCatalog) getCatalog(); - paimonSysTable = catalog.getPaimonTable( - sourceTable.getOrBuildNameMapping(), - "main", // branch - sysTableType // queryType: snapshots, binlog, etc. - ); - LOG.info("Created Paimon system table: {} for source table: {}", - sysTableType, sourceTable.getName()); - } - } - } - return paimonSysTable; - } - - /** - * Returns the schema of the system table. - * The schema is derived from the system table's rowType. - */ - @Override - public List getFullSchema() { - return getOrCreateSchemaCacheValue().getSchema(); - } - - public PaimonExternalTable getSourceTable() { - return sourceTable; - } - - @Override - public NameMapping getOrBuildNameMapping() { - return sourceTable.getOrBuildNameMapping(); - } - - public String getSysTableType() { - return sysTableType; - } - - public boolean isDataTable() { - return resolveIsDataTable(); - } - - private boolean resolveIsDataTable() { - if (isDataTable == null) { - synchronized (this) { - if (isDataTable == null) { - isDataTable = getSysPaimonTable() instanceof DataTable; - } - } - } - return isDataTable; - } - - @Override - public BaseAnalysisTask createAnalysisTask(AnalysisInfo info) { - makeSureInitialized(); - return new ExternalAnalysisTask(info); - } - - @Override - public TTableDescriptor toThrift() { - List schema = getFullSchema(); - String catalogType = sourceTable.getPaimonCatalogType(); - if (PaimonExternalCatalog.PAIMON_HMS.equals(catalogType) - || PaimonExternalCatalog.PAIMON_FILESYSTEM.equals(catalogType) - || PaimonExternalCatalog.PAIMON_DLF.equals(catalogType) - || PaimonExternalCatalog.PAIMON_REST.equals(catalogType) - || PaimonExternalCatalog.PAIMON_JDBC.equals(catalogType)) { - THiveTable tHiveTable = new THiveTable(dbName, name, new HashMap<>()); - TTableDescriptor tTableDescriptor = new TTableDescriptor(getId(), TTableType.HIVE_TABLE, schema.size(), 0, - getName(), dbName); - tTableDescriptor.setHiveTable(tHiveTable); - return tTableDescriptor; - } else { - throw new IllegalArgumentException( - "Currently only supports hms/dlf/rest/filesystem/jdbc catalog, do not support: " + catalogType); - } - } - - @Override - public long fetchRowCount() { - makeSureInitialized(); - long rowCount = 0; - List splits = getSysPaimonTable().newReadBuilder().newScan().plan().splits(); - for (Split split : splits) { - rowCount += split.rowCount(); - } - if (rowCount == 0) { - LOG.info("Paimon system table {} row count is 0, return -1", name); - } - return rowCount > 0 ? rowCount : UNKNOWN_ROW_COUNT; - } - - @Override - public Optional initSchema(SchemaCacheKey key) { - return Optional.of(getOrCreateSchemaCacheValue()); - } - - @Override - public Optional getSchemaCacheValue() { - return Optional.of(getOrCreateSchemaCacheValue()); - } - - @Override - public Map getSupportedSysTables() { - return sourceTable.getSupportedSysTables(); - } - - public Map getTableProperties() { - return sourceTable.getTableProperties(); - } - - @Override - public String getComment() { - return "Paimon system table: " + sysTableType + " for " + sourceTable.getName(); - } - - private SchemaCacheValue getOrCreateSchemaCacheValue() { - if (schemaCacheValue == null) { - synchronized (this) { - if (schemaCacheValue == null) { - if (fullSchema == null) { - fullSchema = buildFullSchema(); - } - schemaCacheValue = new SchemaCacheValue(fullSchema); - } - } - } - return schemaCacheValue; - } - - private List buildFullSchema() { - Table sysTable = getSysPaimonTable(); - List fields = sysTable.rowType().getFields(); - List columns = Lists.newArrayListWithCapacity(fields.size()); - - for (DataField field : fields) { - Column column = new Column( - field.name().toLowerCase(), - PaimonUtil.paimonTypeToDorisType( - field.type(), - getCatalog().getEnableMappingVarbinary(), - getCatalog().getEnableMappingTimestampTz()), - true, - null, - true, - field.description(), - true, - field.id()); - PaimonUtil.updatePaimonColumnUniqueId(column, field); - if (field.type().getTypeRoot() == DataTypeRoot.TIMESTAMP_WITH_LOCAL_TIME_ZONE) { - column.setWithTZExtraInfo(); - } - columns.add(column); - } - return columns; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonTableCacheValue.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonTableCacheValue.java deleted file mode 100644 index 7539f28d770bf6..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonTableCacheValue.java +++ /dev/null @@ -1,44 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import com.google.common.base.Suppliers; -import org.apache.paimon.table.Table; - -import java.util.function.Supplier; - -/** - * Cache value for Paimon table metadata and its latest runtime snapshot projection. - */ -public class PaimonTableCacheValue { - private final Table paimonTable; - private final Supplier latestSnapshotCacheValue; - - public PaimonTableCacheValue(Table paimonTable, Supplier latestSnapshotCacheValue) { - this.paimonTable = paimonTable; - this.latestSnapshotCacheValue = Suppliers.memoize(latestSnapshotCacheValue::get); - } - - public Table getPaimonTable() { - return paimonTable; - } - - public PaimonSnapshotCacheValue getLatestSnapshotCacheValue() { - return latestSnapshotCacheValue.get(); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java deleted file mode 100644 index 6c988adad30d97..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java +++ /dev/null @@ -1,711 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.analysis.PartitionValue; -import org.apache.doris.analysis.TableScanParams; -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.ListPartitionItem; -import org.apache.doris.catalog.PartitionItem; -import org.apache.doris.catalog.PartitionKey; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.common.AnalysisException; -import org.apache.doris.common.UserException; -import org.apache.doris.common.util.TimeUtils; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.hive.HiveUtil; -import org.apache.doris.thrift.TColumnType; -import org.apache.doris.thrift.TPrimitiveType; -import org.apache.doris.thrift.schema.external.TArrayField; -import org.apache.doris.thrift.schema.external.TField; -import org.apache.doris.thrift.schema.external.TFieldPtr; -import org.apache.doris.thrift.schema.external.TMapField; -import org.apache.doris.thrift.schema.external.TNestedField; -import org.apache.doris.thrift.schema.external.TSchema; -import org.apache.doris.thrift.schema.external.TStructField; - -import com.google.common.base.Preconditions; -import com.google.common.collect.Lists; -import com.google.common.collect.Maps; -import org.apache.commons.collections4.CollectionUtils; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.apache.paimon.Snapshot; -import org.apache.paimon.data.BinaryRow; -import org.apache.paimon.data.InternalRow; -import org.apache.paimon.data.Timestamp; -import org.apache.paimon.data.serializer.InternalRowSerializer; -import org.apache.paimon.io.DataOutputViewStreamWrapper; -import org.apache.paimon.options.ConfigOption; -import org.apache.paimon.partition.Partition; -import org.apache.paimon.predicate.Predicate; -import org.apache.paimon.reader.RecordReader; -import org.apache.paimon.schema.TableSchema; -import org.apache.paimon.table.DataTable; -import org.apache.paimon.table.FileStoreTable; -import org.apache.paimon.table.SpecialFields; -import org.apache.paimon.table.Table; -import org.apache.paimon.table.source.DataSplit; -import org.apache.paimon.table.source.ReadBuilder; -import org.apache.paimon.tag.Tag; -import org.apache.paimon.types.ArrayType; -import org.apache.paimon.types.BinaryType; -import org.apache.paimon.types.CharType; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.DataType; -import org.apache.paimon.types.DecimalType; -import org.apache.paimon.types.MapType; -import org.apache.paimon.types.RowType; -import org.apache.paimon.types.VarBinaryType; -import org.apache.paimon.types.VarCharType; -import org.apache.paimon.utils.DateTimeUtils; -import org.apache.paimon.utils.InstantiationUtil; -import org.apache.paimon.utils.Pair; -import org.apache.paimon.utils.Projection; -import org.apache.paimon.utils.RowDataToObjectArrayConverter; - -import java.io.ByteArrayOutputStream; -import java.io.FileNotFoundException; -import java.io.IOException; -import java.time.DateTimeException; -import java.time.LocalDate; -import java.time.LocalTime; -import java.time.ZoneId; -import java.time.format.DateTimeFormatter; -import java.util.ArrayList; -import java.util.Base64; -import java.util.HashMap; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Optional; -import java.util.regex.Pattern; -import java.util.stream.Collectors; -import javax.annotation.Nullable; - -public class PaimonUtil { - private static final Logger LOG = LogManager.getLogger(PaimonUtil.class); - private static final Base64.Encoder BASE64_ENCODER = java.util.Base64.getUrlEncoder().withoutPadding(); - private static final Pattern DIGITAL_REGEX = Pattern.compile("\\d+"); - private static final String SYS_TABLE_TYPE_AUDIT_LOG = "audit_log"; - private static final String SYS_TABLE_TYPE_BINLOG = "binlog"; - private static final String TABLE_READ_SEQUENCE_NUMBER_ENABLED = "table-read.sequence-number.enabled"; - private static final String PARTITION_LEGACY_NAME = "partition.legacy-name"; - - public static boolean isDigitalString(String value) { - return value != null && DIGITAL_REGEX.matcher(value).matches(); - } - - /** - * Extract the legacy partition name configuration from Paimon table options. - */ - public static boolean isLegacyPartitionName(Table paimonTable) { - return Boolean.parseBoolean( - paimonTable.options().getOrDefault(PARTITION_LEGACY_NAME, "true")); - } - - public static List read( - Table table, @Nullable int[] projection, @Nullable Predicate predicate, - Pair, String>... dynamicOptions) - throws IOException { - Map options = new HashMap<>(); - for (Pair, String> pair : dynamicOptions) { - options.put(pair.getKey().key(), pair.getValue()); - } - if (!options.isEmpty()) { - table = table.copy(options); - } - ReadBuilder readBuilder = table.newReadBuilder(); - if (projection != null) { - readBuilder.withProjection(projection); - } - if (predicate != null) { - readBuilder.withFilter(predicate); - } - RecordReader reader = - readBuilder.newRead().createReader(readBuilder.newScan().plan()); - InternalRowSerializer serializer = - new InternalRowSerializer( - projection == null - ? table.rowType() - : Projection.of(projection).project(table.rowType())); - List rows = new ArrayList<>(); - reader.forEachRemaining(row -> rows.add(serializer.copy(row))); - return rows; - } - - public static PaimonPartitionInfo generatePartitionInfo(List partitionColumns, - List paimonPartitions, boolean legacyPartitionName) { - - if (CollectionUtils.isEmpty(partitionColumns) || paimonPartitions.isEmpty()) { - return PaimonPartitionInfo.EMPTY; - } - - Map nameToPartitionItem = Maps.newHashMap(); - Map nameToPartition = Maps.newHashMap(); - PaimonPartitionInfo partitionInfo = new PaimonPartitionInfo(nameToPartitionItem, nameToPartition); - List types = partitionColumns.stream() - .map(Column::getType) - .collect(Collectors.toList()); - Map columnNameToType = partitionColumns.stream() - .collect(Collectors.toMap(Column::getName, Column::getType)); - - for (Partition partition : paimonPartitions) { - Map spec = partition.spec(); - StringBuilder sb = new StringBuilder(); - for (Map.Entry entry : spec.entrySet()) { - sb.append(entry.getKey()).append("="); - // When partition.legacy-name = true (default), Paimon stores DATE type as days since - // 1970-01-01 (epoch integer), so we need to convert the integer to a date string. - // When partition.legacy-name = false, the value is already a human read date string. - if (legacyPartitionName - && columnNameToType.getOrDefault(entry.getKey(), Type.NULL).isDateV2()) { - sb.append(DateTimeUtils.formatDate(Integer.parseInt(entry.getValue()))).append("/"); - } else { - sb.append(entry.getValue()).append("/"); - } - } - if (sb.length() > 0) { - sb.deleteCharAt(sb.length() - 1); - } - String partitionName = sb.toString(); - nameToPartition.put(partitionName, partition); - try { - // partition values return by paimon api, may have problem, - // to avoid affecting the query, we catch exceptions here - nameToPartitionItem.put(partitionName, toListPartitionItem(partitionName, types)); - } catch (Exception e) { - LOG.warn("toListPartitionItem failed, partitionColumns: {}, partitionValues: {}", - partitionColumns, partition.spec(), e); - } - } - return partitionInfo; - } - - public static ListPartitionItem toListPartitionItem(String partitionName, List types) - throws AnalysisException { - // Partition name will be in format: nation=cn/city=beijing - // parse it to get values "cn" and "beijing" - List partitionValues = HiveUtil.toPartitionValues(partitionName); - Preconditions.checkState(partitionValues.size() == types.size(), partitionName + " vs. " + types); - List values = Lists.newArrayListWithExpectedSize(types.size()); - for (String partitionValue : partitionValues) { - // null will in partition 'null' - // "null" will in partition 'null' - // NULL will in partition 'null' - // "NULL" will in partition 'NULL' - // values.add(new PartitionValue(partitionValue, "null".equals(partitionValue))); - values.add(new PartitionValue(partitionValue, false)); - } - PartitionKey key = PartitionKey.createListPartitionKeyWithTypes(values, types, true); - ListPartitionItem listPartitionItem = new ListPartitionItem(Lists.newArrayList(key)); - return listPartitionItem; - } - - private static Type paimonPrimitiveTypeToDorisType(org.apache.paimon.types.DataType dataType, - boolean enableVarbinaryMapping, boolean enableTimestampTzMapping) { - int tsScale = 3; // default - switch (dataType.getTypeRoot()) { - case BOOLEAN: - return Type.BOOLEAN; - case INTEGER: - return Type.INT; - case BIGINT: - return Type.BIGINT; - case FLOAT: - return Type.FLOAT; - case DOUBLE: - return Type.DOUBLE; - case SMALLINT: - return Type.SMALLINT; - case TINYINT: - return Type.TINYINT; - case VARCHAR: - int varcharLen = ((VarCharType) dataType).getLength(); - if (varcharLen > 65533) { - return ScalarType.createStringType(); - } - return ScalarType.createVarcharType(varcharLen); - case CHAR: - int charLen = ((CharType) dataType).getLength(); - if (charLen > 255) { - return ScalarType.createStringType(); - } - return ScalarType.createCharType(charLen); - case BINARY: - int binaryLen = ((BinaryType) dataType).getLength(); - return enableVarbinaryMapping ? ScalarType.createVarbinaryType(binaryLen) : Type.STRING; - case VARBINARY: - // Paimon VarBinaryType length is in [1, 2147483647] - int varbinaryLen = ((VarBinaryType) dataType).getLength(); - return enableVarbinaryMapping ? ScalarType.createVarbinaryType(varbinaryLen) : Type.STRING; - case DECIMAL: - DecimalType decimal = (DecimalType) dataType; - return ScalarType.createDecimalV3Type(decimal.getPrecision(), decimal.getScale()); - case DATE: - return ScalarType.createDateV2Type(); - case TIMESTAMP_WITHOUT_TIME_ZONE: - if (dataType instanceof org.apache.paimon.types.TimestampType) { - tsScale = ((org.apache.paimon.types.TimestampType) dataType).getPrecision(); - if (tsScale > 6) { - tsScale = 6; - } - } else if (dataType instanceof org.apache.paimon.types.LocalZonedTimestampType) { - tsScale = ((org.apache.paimon.types.LocalZonedTimestampType) dataType).getPrecision(); - if (tsScale > 6) { - tsScale = 6; - } - } - return ScalarType.createDatetimeV2Type(tsScale); - case TIMESTAMP_WITH_LOCAL_TIME_ZONE: - if (dataType instanceof org.apache.paimon.types.LocalZonedTimestampType) { - tsScale = ((org.apache.paimon.types.LocalZonedTimestampType) dataType).getPrecision(); - if (tsScale > 6) { - tsScale = 6; - } - } - if (enableTimestampTzMapping) { - return ScalarType.createTimeStampTzType(tsScale); - } - return ScalarType.createDatetimeV2Type(tsScale); - case ARRAY: - ArrayType arrayType = (ArrayType) dataType; - Type innerType = paimonPrimitiveTypeToDorisType(arrayType.getElementType(), enableVarbinaryMapping, - enableTimestampTzMapping); - return org.apache.doris.catalog.ArrayType.create(innerType, true); - case MAP: - MapType mapType = (MapType) dataType; - return new org.apache.doris.catalog.MapType( - paimonTypeToDorisType(mapType.getKeyType(), enableVarbinaryMapping, enableTimestampTzMapping), - paimonTypeToDorisType(mapType.getValueType(), enableVarbinaryMapping, - enableTimestampTzMapping)); - case ROW: - RowType rowType = (RowType) dataType; - List fields = rowType.getFields(); - return new org.apache.doris.catalog.StructType(fields.stream() - .map(field -> new org.apache.doris.catalog.StructField(field.name(), - paimonTypeToDorisType(field.type(), enableVarbinaryMapping, enableTimestampTzMapping))) - .collect(Collectors.toCollection(ArrayList::new))); - case TIME_WITHOUT_TIME_ZONE: - return Type.UNSUPPORTED; - default: - LOG.warn("Cannot transform unknown type: " + dataType.getTypeRoot()); - return Type.UNSUPPORTED; - } - } - - public static Type paimonTypeToDorisType(org.apache.paimon.types.DataType type, boolean enableVarbinaryMapping, - boolean enableTimestampTzMapping) { - return paimonPrimitiveTypeToDorisType(type, enableVarbinaryMapping, enableTimestampTzMapping); - } - - public static void updatePaimonColumnUniqueId(Column column, DataType dataType) { - List columns = column.getChildren(); - if (columns == null) { - return; - } - switch (dataType.getTypeRoot()) { - case ARRAY: - ArrayType arrayType = (ArrayType) dataType; - updatePaimonColumnUniqueId(columns.get(0), arrayType.getElementType()); - break; - case MAP: - MapType mapType = (MapType) dataType; - updatePaimonColumnUniqueId(columns.get(0), mapType.getKeyType()); - updatePaimonColumnUniqueId(columns.get(1), mapType.getValueType()); - break; - case ROW: - RowType rowType = (RowType) dataType; - for (int idx = 0; idx < columns.size(); idx++) { - updatePaimonColumnUniqueId(columns.get(idx), rowType.getFields().get(idx)); - } - break; - default: - return; - } - } - - public static void updatePaimonColumnUniqueId(Column column, DataField field) { - column.setUniqueId(field.id()); - updatePaimonColumnUniqueId(column, field.type()); - } - - public static TField getSchemaInfo(DataType dataType, boolean enableVarbinaryMapping, - boolean enableTimestampTzMapping) { - TField field = new TField(); - field.setIsOptional(dataType.isNullable()); - TNestedField nestedField = new TNestedField(); - switch (dataType.getTypeRoot()) { - case ARRAY: { - TArrayField listField = new TArrayField(); - org.apache.paimon.types.ArrayType paimonArrayType = (org.apache.paimon.types.ArrayType) dataType; - TFieldPtr fieldPtr = new TFieldPtr(); - fieldPtr.setFieldPtr(getSchemaInfo(paimonArrayType.getElementType(), enableVarbinaryMapping, - enableTimestampTzMapping)); - listField.setItemField(fieldPtr); - nestedField.setArrayField(listField); - field.setNestedField(nestedField); - - TColumnType tColumnType = new TColumnType(); - tColumnType.setType(TPrimitiveType.ARRAY); - field.setType(tColumnType); - break; - } - case MAP: { - TMapField mapField = new TMapField(); - org.apache.paimon.types.MapType mapType = (org.apache.paimon.types.MapType) dataType; - TFieldPtr keyField = new TFieldPtr(); - keyField.setFieldPtr( - getSchemaInfo(mapType.getKeyType(), enableVarbinaryMapping, enableTimestampTzMapping)); - mapField.setKeyField(keyField); - TFieldPtr valueField = new TFieldPtr(); - valueField.setFieldPtr( - getSchemaInfo(mapType.getValueType(), enableVarbinaryMapping, enableTimestampTzMapping)); - mapField.setValueField(valueField); - nestedField.setMapField(mapField); - field.setNestedField(nestedField); - - TColumnType tColumnType = new TColumnType(); - tColumnType.setType(TPrimitiveType.MAP); - field.setType(tColumnType); - break; - } - case ROW: { - RowType rowType = (RowType) dataType; - TStructField structField = getSchemaInfo(rowType.getFields(), enableVarbinaryMapping, - enableTimestampTzMapping); - nestedField.setStructField(structField); - field.setNestedField(nestedField); - - TColumnType tColumnType = new TColumnType(); - tColumnType.setType(TPrimitiveType.STRUCT); - field.setType(tColumnType); - break; - } - default: - field.setType(paimonPrimitiveTypeToDorisType(dataType, enableVarbinaryMapping, enableTimestampTzMapping) - .toColumnTypeThrift()); - break; - } - return field; - } - - public static TStructField getSchemaInfo(List paimonFields, boolean enableVarbinaryMapping, - boolean enableTimestampTzMapping) { - TStructField structField = new TStructField(); - for (DataField paimonField : paimonFields) { - TField childField = getSchemaInfo(paimonField.type(), enableVarbinaryMapping, enableTimestampTzMapping); - childField.setName(paimonField.name()); - childField.setId(paimonField.id()); - TFieldPtr fieldPtr = new TFieldPtr(); - fieldPtr.setFieldPtr(childField); - structField.addToFields(fieldPtr); - } - return structField; - } - - public static TSchema getSchemaInfo(TableSchema paimonTableSchema, boolean enableVarbinaryMapping, - boolean enableTimestampTzMapping) { - TSchema tSchema = new TSchema(); - tSchema.setSchemaId(paimonTableSchema.id()); - tSchema.setRootField( - getSchemaInfo(paimonTableSchema.fields(), enableVarbinaryMapping, enableTimestampTzMapping)); - return tSchema; - } - - public static TSchema getHistorySchemaInfo(ExternalTable targetTable, TableSchema sourceSchema, - boolean enableVarbinaryMapping, boolean enableTimestampTzMapping) { - TSchema tSchema = new TSchema(); - tSchema.setSchemaId(sourceSchema.id()); - tSchema.setRootField(getSchemaInfo(resolveHistorySchemaFields(targetTable, sourceSchema.fields()), - enableVarbinaryMapping, enableTimestampTzMapping)); - return tSchema; - } - - private static List resolveHistorySchemaFields(ExternalTable targetTable, List sourceFields) { - if (!(targetTable instanceof PaimonSysExternalTable)) { - return sourceFields; - } - - PaimonSysExternalTable sysTable = (PaimonSysExternalTable) targetTable; - boolean withSequenceNumber = isTableReadSequenceNumberEnabled(sysTable); - switch (sysTable.getSysTableType()) { - case SYS_TABLE_TYPE_AUDIT_LOG: - return buildAuditLogHistoryFields(sourceFields, withSequenceNumber); - case SYS_TABLE_TYPE_BINLOG: - return buildBinlogHistoryFields(sourceFields, withSequenceNumber); - default: - return sourceFields; - } - } - - private static List buildAuditLogHistoryFields(List sourceFields, - boolean withSequenceNumber) { - List fields = new ArrayList<>(sourceFields.size() + (withSequenceNumber ? 2 : 1)); - fields.add(SpecialFields.ROW_KIND); - if (withSequenceNumber) { - fields.add(SpecialFields.SEQUENCE_NUMBER); - } - fields.addAll(sourceFields); - return fields; - } - - private static List buildBinlogHistoryFields(List sourceFields, - boolean withSequenceNumber) { - List fields = new ArrayList<>(sourceFields.size() + (withSequenceNumber ? 2 : 1)); - fields.add(SpecialFields.ROW_KIND); - if (withSequenceNumber) { - fields.add(SpecialFields.SEQUENCE_NUMBER); - } - for (DataField sourceField : sourceFields) { - fields.add(sourceField.newType(new ArrayType(sourceField.type().nullable()))); - } - return fields; - } - - private static boolean isTableReadSequenceNumberEnabled(PaimonSysExternalTable sysTable) { - if (!SYS_TABLE_TYPE_AUDIT_LOG.equals(sysTable.getSysTableType()) - && !SYS_TABLE_TYPE_BINLOG.equals(sysTable.getSysTableType())) { - return false; - } - try { - String optionValue = sysTable.getTableProperties().get(TABLE_READ_SEQUENCE_NUMBER_ENABLED); - return Boolean.parseBoolean(optionValue); - } catch (Exception e) { - LOG.warn("Failed to parse table-read.sequence-number.enabled for Paimon system table {}: {}", - sysTable.getName(), e.getMessage()); - return false; - } - } - - public static List parseSchema(Table table, boolean enableVarbinaryMapping, - boolean enableTimestampTzMapping) { - List primaryKeys = table.primaryKeys(); - return parseSchema(table.rowType(), primaryKeys, enableVarbinaryMapping, enableTimestampTzMapping); - } - - public static List parseSchema(RowType rowType, List primaryKeys, boolean enableVarbinaryMapping, - boolean enableTimestampTzMapping) { - List resSchema = Lists.newArrayListWithCapacity(rowType.getFields().size()); - rowType.getFields().forEach(field -> { - resSchema.add(new Column(field.name().toLowerCase(), - PaimonUtil.paimonTypeToDorisType(field.type(), enableVarbinaryMapping, enableTimestampTzMapping), - primaryKeys.contains(field.name()), - null, - field.type().isNullable(), - field.description(), - true, - field.id())); - }); - return resSchema; - } - - public static String encodeObjectToString(T t) { - try { - byte[] bytes = InstantiationUtil.serializeObject(t); - return new String(BASE64_ENCODER.encode(bytes), java.nio.charset.StandardCharsets.UTF_8); - } catch (Exception e) { - throw new RuntimeException(e); - } - } - - /** - * Serialize DataSplit using Paimon's native binary format. - * This format is compatible with paimon-cpp reader. - * Uses standard Base64 encoding (not URL-safe) for BE compatibility. - */ - public static String encodeDataSplitToString(DataSplit split) { - try { - ByteArrayOutputStream baos = new ByteArrayOutputStream(); - DataOutputViewStreamWrapper out = new DataOutputViewStreamWrapper(baos); - split.serialize(out); - byte[] bytes = baos.toByteArray(); - return Base64.getEncoder().encodeToString(bytes); - } catch (IOException e) { - throw new RuntimeException("Failed to serialize DataSplit using Paimon native format", e); - } - } - - public static Map getPartitionInfoMap(Table table, BinaryRow partitionValues, String timeZone) { - Map partitionInfoMap = new HashMap<>(); - List partitionKeys = table.partitionKeys(); - RowType partitionType = table.rowType().project(partitionKeys); - RowDataToObjectArrayConverter toObjectArrayConverter = new RowDataToObjectArrayConverter( - partitionType); - Object[] partitionValuesArray = toObjectArrayConverter.convert(partitionValues); - for (int i = 0; i < partitionKeys.size(); i++) { - try { - String partitionValue = serializePartitionValue(partitionType.getFields().get(i).type(), - partitionValuesArray[i], timeZone); - partitionInfoMap.put(partitionKeys.get(i).toLowerCase(Locale.ROOT), partitionValue); - } catch (UnsupportedOperationException e) { - LOG.warn("Failed to serialize table {} partition value for key {}: {}", table.name(), - partitionKeys.get(i), e.getMessage()); - return null; - } - } - return partitionInfoMap; - } - - private static String serializePartitionValue(org.apache.paimon.types.DataType type, Object value, - String timeZone) { - switch (type.getTypeRoot()) { - case BOOLEAN: - case INTEGER: - case BIGINT: - case SMALLINT: - case TINYINT: - case DECIMAL: - case VARCHAR: - case CHAR: - if (value == null) { - return null; - } - return value.toString(); - case FLOAT: - if (value == null) { - return null; - } - return Float.toString((Float) value); - case DOUBLE: - if (value == null) { - return null; - } - return Double.toString((Double) value); - // case binary: - // case varbinary: should not supported, because if return string with utf8, - // the data maybe be corrupted - case DATE: - if (value == null) { - return null; - } - // Paimon date is stored as days since epoch - LocalDate date = LocalDate.ofEpochDay((Integer) value); - return date.format(DateTimeFormatter.ISO_LOCAL_DATE); - case TIME_WITHOUT_TIME_ZONE: - if (value == null) { - return null; - } - // Paimon time is stored as microseconds since midnight in utc - long micros = (Long) value; - LocalTime time = LocalTime.ofNanoOfDay(micros * 1000); - return time.format(DateTimeFormatter.ISO_LOCAL_TIME); - case TIMESTAMP_WITHOUT_TIME_ZONE: - if (value == null) { - return null; - } - // Paimon timestamp is stored as Timestamp type in utc - return ((Timestamp) value).toLocalDateTime().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME); - case TIMESTAMP_WITH_LOCAL_TIME_ZONE: - if (value == null) { - return null; - } - // Paimon timestamp with local time zone is stored as Timestamp type in utc - Timestamp timestamp = (Timestamp) value; - return timestamp.toLocalDateTime() - .atZone(ZoneId.of("UTC")) - .withZoneSameInstant(ZoneId.of(timeZone)) - .toLocalDateTime() - .format(DateTimeFormatter.ISO_LOCAL_DATE_TIME); - default: - throw new UnsupportedOperationException("Unsupported type for serializePartitionValue: " + type); - } - } - - /** - * Extracts the reference name (branch or tag name) from table scan parameters. - * - * @param scanParams the scan parameters containing reference name information - * @return the extracted reference name - * @throws IllegalArgumentException if the reference name is not properly specified - */ - public static String extractBranchOrTagName(TableScanParams scanParams) { - if (!scanParams.getMapParams().isEmpty()) { - if (!scanParams.getMapParams().containsKey(TableScanParams.PARAMS_NAME)) { - throw new IllegalArgumentException("must contain key 'name' in params"); - } - return scanParams.getMapParams().get(TableScanParams.PARAMS_NAME); - } else { - if (scanParams.getListParams().isEmpty() || scanParams.getListParams().get(0) == null) { - throw new IllegalArgumentException("must contain a branch/tag name in params"); - } - return scanParams.getListParams().get(0); - } - } - - static Snapshot getPaimonSnapshotByTimestamp(DataTable table, String timestamp, boolean isDigital) - throws UserException { - long timestampMillis = 0; - if (isDigital) { - timestampMillis = Long.parseLong(timestamp); - } else { - // Supported formats include：yyyy-MM-dd, yyyy-MM-dd HH:mm:ss, yyyy-MM-dd HH:mm:ss.SSS. - // use default local time zone. - timestampMillis = DateTimeUtils.parseTimestampData(timestamp, 3, TimeUtils.getTimeZone()).getMillisecond(); - if (timestampMillis < 0) { - throw new DateTimeException("can't parse time: " + timestamp); - } - } - Snapshot snapshot = table.snapshotManager().earlierOrEqualTimeMills(timestampMillis); - if (snapshot == null) { - Snapshot earliestSnapshot = table.snapshotManager().earliestSnapshot(); - throw new UserException( - String.format( - "There is currently no snapshot earlier than or equal to timestamp [%s], " - + "the earliest snapshot's timestamp is [%s]", - timestampMillis, - earliestSnapshot == null - ? "null" - : String.valueOf(earliestSnapshot.timeMillis()))); - } - return snapshot; - } - - static Snapshot getPaimonSnapshotBySnapshotId(DataTable table, String snapshotString) - throws UserException { - long snapshotId = Long.parseLong(snapshotString); - try { - Snapshot snapshot = table.snapshotManager().tryGetSnapshot(snapshotId); - return snapshot; - } catch (FileNotFoundException e) { - throw new UserException("can't find snapshot by id: " + snapshotId, e); - } - } - - static Snapshot getPaimonSnapshotByTag(DataTable table, String tagName) - throws UserException { - Optional tag = table.tagManager().get(tagName); - return tag.orElseThrow(() -> new UserException("can't find snapshot by tag: " + tagName)); - } - - - public static String resolvePaimonBranch(TableScanParams tableScanParams, Table baseTable) - throws UserException { - String branchName = extractBranchOrTagName(tableScanParams); - if (!(baseTable instanceof FileStoreTable)) { - throw new UserException("Table type should be FileStoreTable but got: " + baseTable.getClass().getName()); - } - - final FileStoreTable fileStoreTable = (FileStoreTable) baseTable; - if (!fileStoreTable.branchManager().branchExists(branchName)) { - throw new UserException("can't find branch: " + branchName); - } - return branchName; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtils.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtils.java deleted file mode 100644 index dc28c083ca10fa..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtils.java +++ /dev/null @@ -1,59 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.catalog.Env; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.mvcc.MvccSnapshot; - -import org.apache.paimon.table.Table; - -import java.util.Optional; - -public class PaimonUtils { - - public static Table getPaimonTable(ExternalTable dorisTable) { - return paimonExternalMetaCache(dorisTable).getPaimonTable(dorisTable); - } - - public static PaimonSnapshotCacheValue getLatestSnapshotCacheValue(ExternalTable dorisTable) { - return paimonExternalMetaCache(dorisTable).getSnapshotCache(dorisTable); - } - - public static PaimonSnapshotCacheValue getSnapshotCacheValue(Optional snapshot, - ExternalTable dorisTable) { - if (snapshot.isPresent() && snapshot.get() instanceof PaimonMvccSnapshot) { - return ((PaimonMvccSnapshot) snapshot.get()).getSnapshotCacheValue(); - } - return getLatestSnapshotCacheValue(dorisTable); - } - - public static PaimonSchemaCacheValue getSchemaCacheValue(ExternalTable dorisTable, - PaimonSnapshotCacheValue snapshotValue) { - return getSchemaCacheValue(dorisTable, snapshotValue.getSnapshot().getSchemaId()); - } - - public static PaimonSchemaCacheValue getSchemaCacheValue(ExternalTable dorisTable, long schemaId) { - return paimonExternalMetaCache(dorisTable) - .getPaimonSchemaCacheValue(dorisTable.getOrBuildNameMapping(), schemaId); - } - - private static PaimonExternalMetaCache paimonExternalMetaCache(ExternalTable table) { - return Env.getCurrentEnv().getExtMetaCacheMgr().paimon(table.getCatalog().getId()); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonMetricRegistry.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonMetricRegistry.java deleted file mode 100644 index 4904c71faab926..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonMetricRegistry.java +++ /dev/null @@ -1,72 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.profile; - -import org.apache.paimon.metrics.MetricGroup; -import org.apache.paimon.metrics.MetricGroupImpl; -import org.apache.paimon.metrics.MetricRegistry; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.Collection; -import java.util.Map; -import java.util.concurrent.ConcurrentHashMap; - -public class PaimonMetricRegistry implements MetricRegistry { - private static final Logger LOG = LoggerFactory.getLogger(PaimonMetricRegistry.class); - private static final String TABLE_TAG_KEY = "table"; - private final ConcurrentHashMap groups = new ConcurrentHashMap<>(); - - @Override - public MetricGroup createMetricGroup(String name, Map tags) { - MetricGroup group = new MetricGroupImpl(name, tags); - String table = tags == null ? "" : tags.getOrDefault(TABLE_TAG_KEY, ""); - groups.put(buildKey(name, table), group); - LOG.debug("Created metric group: {}:{}", name, table); - return group; - } - - public MetricGroup getGroup(String name, String table) { - String key = buildKey(name, table); - MetricGroup group = groups.get(key); - if (group == null) { - LOG.warn("MetricGroup not found: {}", key); - } - return group; - } - - public void removeGroup(String name, String table) { - groups.remove(buildKey(name, table)); - } - - public Collection getAllGroups() { - return groups.values(); - } - - public Map getAllGroupsAsMap() { - return new ConcurrentHashMap<>(groups); - } - - public void clear() { - groups.clear(); - } - - private static String buildKey(String name, String table) { - return name + ":" + table; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonScanMetricsReporter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonScanMetricsReporter.java deleted file mode 100644 index b76cf74dfda8e5..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonScanMetricsReporter.java +++ /dev/null @@ -1,152 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.profile; - -import org.apache.doris.catalog.TableIf; -import org.apache.doris.common.profile.RuntimeProfile; -import org.apache.doris.common.profile.SummaryProfile; -import org.apache.doris.common.util.DebugUtil; -import org.apache.doris.qe.ConnectContext; - -import org.apache.paimon.metrics.Counter; -import org.apache.paimon.metrics.Gauge; -import org.apache.paimon.metrics.Histogram; -import org.apache.paimon.metrics.HistogramStatistics; -import org.apache.paimon.metrics.Metric; -import org.apache.paimon.metrics.MetricGroup; -import org.apache.paimon.operation.metrics.ScanMetrics; - -import java.util.Map; -import java.util.concurrent.TimeUnit; - -public class PaimonScanMetricsReporter { - private static final double P95 = 0.95d; - - public static void report(TableIf table, String paimonTableName, PaimonMetricRegistry registry) { - if (registry == null || paimonTableName == null) { - return; - } - String resolvedTableName = paimonTableName; - MetricGroup group = registry.getGroup(ScanMetrics.GROUP_NAME, paimonTableName); - if (group == null) { - String prefix = ScanMetrics.GROUP_NAME + ":"; - for (Map.Entry entry : registry.getAllGroupsAsMap().entrySet()) { - String key = entry.getKey(); - if (!key.startsWith(prefix)) { - continue; - } - if (group != null) { - group = null; - break; - } - group = entry.getValue(); - resolvedTableName = key.substring(prefix.length()); - } - } - if (group == null) { - return; - } - Map metrics = group.getMetrics(); - if (metrics == null || metrics.isEmpty()) { - return; - } - - SummaryProfile summaryProfile = SummaryProfile.getSummaryProfile(ConnectContext.get()); - if (summaryProfile == null) { - return; - } - RuntimeProfile executionSummary = summaryProfile.getExecutionSummary(); - if (executionSummary == null) { - return; - } - - RuntimeProfile paimonGroup = executionSummary.getChildMap().get(SummaryProfile.PAIMON_SCAN_METRICS); - if (paimonGroup == null) { - paimonGroup = new RuntimeProfile(SummaryProfile.PAIMON_SCAN_METRICS); - executionSummary.addChild(paimonGroup, true); - } - - String displayName = table == null ? paimonTableName : table.getNameWithFullQualifiers(); - RuntimeProfile scanProfile = new RuntimeProfile("Table Scan (" + displayName + ")"); - appendDuration(scanProfile, metrics, ScanMetrics.LAST_SCAN_DURATION, "last_scan_duration"); - appendHistogram(scanProfile, metrics, ScanMetrics.SCAN_DURATION, "scan_duration"); - appendCounter(scanProfile, metrics, ScanMetrics.LAST_SCANNED_MANIFESTS, "last_scanned_manifests"); - appendCounter(scanProfile, metrics, ScanMetrics.LAST_SCAN_SKIPPED_TABLE_FILES, - "last_scan_skipped_table_files"); - appendCounter(scanProfile, metrics, ScanMetrics.LAST_SCAN_RESULTED_TABLE_FILES, - "last_scan_resulted_table_files"); - appendCounter(scanProfile, metrics, ScanMetrics.MANIFEST_HIT_CACHE, "manifest_hit_cache"); - appendCounter(scanProfile, metrics, ScanMetrics.MANIFEST_MISSED_CACHE, "manifest_missed_cache"); - paimonGroup.addChild(scanProfile, true); - registry.removeGroup(ScanMetrics.GROUP_NAME, resolvedTableName); - } - - private static void appendDuration(RuntimeProfile profile, Map metrics, String metricKey, - String profileKey) { - Long value = getLongValue(metrics.get(metricKey)); - if (value == null) { - return; - } - profile.addInfoString(profileKey, formatDuration(value)); - } - - private static void appendCounter(RuntimeProfile profile, Map metrics, String metricKey, - String profileKey) { - Long value = getLongValue(metrics.get(metricKey)); - if (value == null) { - return; - } - profile.addInfoString(profileKey, Long.toString(value)); - } - - private static void appendHistogram(RuntimeProfile profile, Map metrics, String metricKey, - String profileKey) { - Metric metric = metrics.get(metricKey); - if (!(metric instanceof Histogram)) { - return; - } - Histogram histogram = (Histogram) metric; - HistogramStatistics stats = histogram.getStatistics(); - if (stats == null) { - return; - } - String formatted = "count=" + histogram.getCount() - + ", mean=" + formatDuration(stats.getMean()) - + ", p95=" + formatDuration(stats.getQuantile(P95)) - + ", max=" + formatDuration(stats.getMax()); - profile.addInfoString(profileKey, formatted); - } - - private static Long getLongValue(Metric metric) { - if (metric instanceof Counter) { - return ((Counter) metric).getCount(); - } - if (metric instanceof Gauge) { - Object value = ((Gauge) metric).getValue(); - if (value instanceof Number) { - return ((Number) value).longValue(); - } - } - return null; - } - - private static String formatDuration(double nanos) { - long ms = TimeUnit.NANOSECONDS.toMillis(Math.round(nanos)); - return DebugUtil.getPrettyStringMs(ms); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java deleted file mode 100644 index 963904a4ff467d..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java +++ /dev/null @@ -1,201 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.source; - -import org.apache.doris.analysis.BinaryPredicate; -import org.apache.doris.analysis.CastExpr; -import org.apache.doris.analysis.CompoundPredicate; -import org.apache.doris.analysis.Expr; -import org.apache.doris.analysis.ExprToExprNameVisitor; -import org.apache.doris.analysis.FunctionCallExpr; -import org.apache.doris.analysis.InPredicate; -import org.apache.doris.analysis.IsNullPredicate; -import org.apache.doris.analysis.LiteralExpr; -import org.apache.doris.analysis.SlotRef; - -import org.apache.paimon.data.BinaryString; -import org.apache.paimon.predicate.Predicate; -import org.apache.paimon.predicate.PredicateBuilder; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.DataType; -import org.apache.paimon.types.RowType; - -import java.util.ArrayList; -import java.util.List; -import java.util.stream.Collectors; - - -public class PaimonPredicateConverter { - private final PredicateBuilder builder; - private final List fieldNames; - private final List paimonFieldTypes; - - public PaimonPredicateConverter(RowType rowType) { - this.builder = new PredicateBuilder(rowType); - this.fieldNames = rowType.getFields().stream().map(f -> f.name().toLowerCase()).collect(Collectors.toList()); - this.paimonFieldTypes = rowType.getFields().stream().map(DataField::type).collect(Collectors.toList()); - } - - public List convertToPaimonExpr(List conjuncts) { - List list = new ArrayList<>(conjuncts.size()); - for (Expr conjunct : conjuncts) { - Predicate predicate = convertToPaimonExpr(conjunct); - if (predicate != null) { - list.add(predicate); - } - } - return list; - } - - private Predicate convertToPaimonExpr(Expr dorisExpr) { - if (dorisExpr == null) { - return null; - } - if (dorisExpr instanceof CompoundPredicate) { - CompoundPredicate compoundPredicate = (CompoundPredicate) dorisExpr; - Predicate left = convertToPaimonExpr(compoundPredicate.getChild(0)); - Predicate right = convertToPaimonExpr(compoundPredicate.getChild(1)); - - switch (compoundPredicate.getOp()) { - case AND: { - if (left != null && right != null) { - return PredicateBuilder.and(left, right); - } - return null; - } - case OR: { - if (left != null && right != null) { - return PredicateBuilder.or(left, right); - } - return null; - } - default: - return null; - } - } else if (dorisExpr instanceof InPredicate) { - return doInPredicate((InPredicate) dorisExpr); - } else { - return binaryExprDesc(dorisExpr); - } - } - - private Predicate doInPredicate(InPredicate predicate) { - SlotRef slotRef = convertDorisExprToSlotRef(predicate.getChild(0)); - if (slotRef == null) { - return null; - } - String colName = slotRef.getColumnName(); - int idx = fieldNames.indexOf(colName); - DataType dataType = paimonFieldTypes.get(idx); - List valueList = new ArrayList<>(); - for (int i = 1; i < predicate.getChildren().size(); i++) { - if (!(predicate.getChild(i) instanceof LiteralExpr)) { - return null; - } - LiteralExpr literalExpr = convertDorisExprToLiteralExpr(predicate.getChild(i)); - Object value = dataType.accept(new PaimonValueConverter(literalExpr)); - if (value == null) { - return null; - } - valueList.add(value); - } - - if (predicate.isNotIn()) { - // not in - return builder.notIn(idx, valueList); - } else { - // in - return builder.in(idx, valueList); - } - } - - private Predicate binaryExprDesc(Expr dorisExpr) { - // Make sure the col slot is always first - SlotRef slotRef = convertDorisExprToSlotRef(dorisExpr.getChild(0)); - LiteralExpr literalExpr = convertDorisExprToLiteralExpr(dorisExpr.getChild(1)); - if (slotRef == null || literalExpr == null) { - return null; - } - String colName = slotRef.getColumnName(); - int idx = fieldNames.indexOf(colName); - DataType dataType = paimonFieldTypes.get(idx); - Object value = dataType.accept(new PaimonValueConverter(literalExpr)); - if (value == null) { - return null; - } - if (dorisExpr instanceof BinaryPredicate) { - BinaryPredicate.Operator op = ((BinaryPredicate) dorisExpr).getOp(); - switch (op) { - case EQ: - return builder.equal(idx, value); - case EQ_FOR_NULL: - return builder.isNull(idx); - case NE: - return builder.notEqual(idx, value); - case GE: - return builder.greaterOrEqual(idx, value); - case GT: - return builder.greaterThan(idx, value); - case LE: - return builder.lessOrEqual(idx, value); - case LT: - return builder.lessThan(idx, value); - default: - return null; - } - } else if (dorisExpr instanceof FunctionCallExpr) { - String name = dorisExpr.accept(ExprToExprNameVisitor.INSTANCE, null).toLowerCase(); - String s = value.toString(); - if (name.equals("like") && !s.startsWith("%") && s.endsWith("%")) { - return builder.startsWith(idx, BinaryString.fromString(s.substring(0, s.length() - 1))); - } - } else if (dorisExpr instanceof IsNullPredicate) { - if (((IsNullPredicate) dorisExpr).isNotNull()) { - return builder.isNotNull(idx); - } else { - return builder.isNull(idx); - } - } - return null; - } - - - public static SlotRef convertDorisExprToSlotRef(Expr expr) { - SlotRef slotRef = null; - if (expr instanceof SlotRef) { - slotRef = (SlotRef) expr; - } else if (expr instanceof CastExpr) { - if (expr.getChild(0) instanceof SlotRef) { - slotRef = (SlotRef) expr.getChild(0); - } - } - return slotRef; - } - - public LiteralExpr convertDorisExprToLiteralExpr(Expr expr) { - LiteralExpr literalExpr = null; - if (expr instanceof LiteralExpr) { - literalExpr = (LiteralExpr) expr; - } else if (expr instanceof CastExpr) { - if (expr.getChild(0) instanceof LiteralExpr) { - literalExpr = (LiteralExpr) expr.getChild(0); - } - } - return literalExpr; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java deleted file mode 100644 index 17a742b835a4fb..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java +++ /dev/null @@ -1,900 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.source; - -import org.apache.doris.analysis.TableScanParams; -import org.apache.doris.analysis.TupleDescriptor; -import org.apache.doris.catalog.TableIf; -import org.apache.doris.common.DdlException; -import org.apache.doris.common.MetaNotFoundException; -import org.apache.doris.common.UserException; -import org.apache.doris.common.util.FileFormatUtils; -import org.apache.doris.common.util.LocationPath; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.ExternalUtil; -import org.apache.doris.datasource.FileQueryScanNode; -import org.apache.doris.datasource.credentials.CredentialUtils; -import org.apache.doris.datasource.credentials.VendedCredentialsFactory; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; -import org.apache.doris.datasource.paimon.PaimonSysExternalTable; -import org.apache.doris.datasource.paimon.PaimonUtil; -import org.apache.doris.datasource.paimon.PaimonUtils; -import org.apache.doris.datasource.paimon.profile.PaimonMetricRegistry; -import org.apache.doris.datasource.paimon.profile.PaimonScanMetricsReporter; -import org.apache.doris.datasource.property.metastore.PaimonJdbcMetaStoreProperties; -import org.apache.doris.datasource.property.storage.StorageProperties; -import org.apache.doris.planner.PlanNodeId; -import org.apache.doris.planner.ScanContext; -import org.apache.doris.qe.SessionVariable; -import org.apache.doris.spi.Split; -import org.apache.doris.thrift.TExplainLevel; -import org.apache.doris.thrift.TFileFormatType; -import org.apache.doris.thrift.TFileRangeDesc; -import org.apache.doris.thrift.TPaimonDeletionFileDesc; -import org.apache.doris.thrift.TPaimonFileDesc; -import org.apache.doris.thrift.TPushAggOp; -import org.apache.doris.thrift.TTableFormatFileDesc; - -import com.google.common.annotations.VisibleForTesting; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.apache.paimon.data.BinaryRow; -import org.apache.paimon.predicate.Predicate; -import org.apache.paimon.schema.TableSchema; -import org.apache.paimon.table.Table; -import org.apache.paimon.table.source.DataSplit; -import org.apache.paimon.table.source.DeletionFile; -import org.apache.paimon.table.source.InnerTableScan; -import org.apache.paimon.table.source.RawFile; -import org.apache.paimon.table.source.ReadBuilder; -import org.apache.paimon.table.source.TableScan; - -import java.io.IOException; -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Optional; -import java.util.concurrent.ConcurrentHashMap; -import java.util.stream.Collectors; - -public class PaimonScanNode extends FileQueryScanNode { - private static final Logger LOG = LogManager.getLogger(PaimonScanNode.class); - - private static final long COUNT_WITH_PARALLEL_SPLITS = 10000; - // The keys of incremental read params for Paimon SDK - private static final String PAIMON_SCAN_SNAPSHOT_ID = "scan.snapshot-id"; - private static final String PAIMON_SCAN_MODE = "scan.mode"; - private static final String PAIMON_INCREMENTAL_BETWEEN = "incremental-between"; - private static final String PAIMON_INCREMENTAL_BETWEEN_SCAN_MODE = "incremental-between-scan-mode"; - private static final String PAIMON_INCREMENTAL_BETWEEN_TIMESTAMP = "incremental-between-timestamp"; - // The keys of incremental read params for Doris Statement - private static final String DORIS_START_SNAPSHOT_ID = "startSnapshotId"; - private static final String DORIS_END_SNAPSHOT_ID = "endSnapshotId"; - private static final String DORIS_START_TIMESTAMP = "startTimestamp"; - private static final String DORIS_END_TIMESTAMP = "endTimestamp"; - private static final String DORIS_INCREMENTAL_BETWEEN_SCAN_MODE = "incrementalBetweenScanMode"; - private static final String PAIMON_BINLOG_SYSTEM_TABLE_TYPE = "binlog"; - private static final String PAIMON_AUDIT_LOG_SYSTEM_TABLE_TYPE = "audit_log"; - - private enum SplitReadType { - JNI, - NATIVE, - } - - private class SplitStat { - SplitReadType type = SplitReadType.JNI; - private long rowCount = 0; - private Optional mergedRowCount = Optional.empty(); - private boolean rawFileConvertable = false; - private boolean hasDeletionVector = false; - - public void setType(SplitReadType type) { - this.type = type; - } - - public void setRowCount(long rowCount) { - this.rowCount = rowCount; - } - - public void setMergedRowCount(long mergedRowCount) { - this.mergedRowCount = Optional.of(mergedRowCount); - } - - public void setRawFileConvertable(boolean rawFileConvertable) { - this.rawFileConvertable = rawFileConvertable; - } - - public void setHasDeletionVector(boolean hasDeletionVector) { - this.hasDeletionVector = hasDeletionVector; - } - - @Override - public String toString() { - return "SplitStat [type=" + type - + ", rowCount=" + rowCount - + ", mergedRowCount=" + (mergedRowCount.isPresent() ? mergedRowCount.get() : "NONE") - + ", rawFileConvertable=" + rawFileConvertable - + ", hasDeletionVector=" + hasDeletionVector + "]"; - } - } - - private PaimonSource source = null; - private List predicates; - private int rawFileSplitNum = 0; - private int paimonSplitNum = 0; - private List splitStats = new ArrayList<>(); - private String serializedTable; - // Store PropertiesMap, including vended credentials or static credentials - // get them in doInitialize() to ensure internal consistency of ScanNode - private Map storagePropertiesMap; - private Map backendStorageProperties; - private Map backendPaimonOptions = Collections.emptyMap(); - - // The schema information involved in the current query process (including historical schema). - protected ConcurrentHashMap currentQuerySchema = new ConcurrentHashMap<>(); - - public PaimonScanNode(PlanNodeId id, - TupleDescriptor desc, - boolean needCheckColumnPriv, - SessionVariable sv, - ScanContext scanContext) { - super(id, desc, "PAIMON_SCAN_NODE", scanContext, needCheckColumnPriv, sv); - source = new PaimonSource(desc); - } - - @Override - protected void doInitialize() throws UserException { - super.doInitialize(); - long startTime = System.currentTimeMillis(); - serializedTable = PaimonUtil.encodeObjectToString(source.getPaimonTable()); - // Todo: Get the current schema id of the table, instead of using -1. - ExternalUtil.initSchemaInfo(params, -1L, source.getTargetTable().getColumns()); - PaimonExternalCatalog catalog = (PaimonExternalCatalog) source.getCatalog(); - storagePropertiesMap = VendedCredentialsFactory.getStoragePropertiesMapWithVendedCredentials( - catalog.getCatalogProperty().getMetastoreProperties(), - catalog.getCatalogProperty().getStoragePropertiesMap(), - source.getPaimonTable() - ); - backendStorageProperties = CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap); - backendPaimonOptions = getBackendPaimonOptions(); - if (getSummaryProfile() != null) { - getSummaryProfile().addExternalTableGetTableMetaTime(System.currentTimeMillis() - startTime); - } - } - - @VisibleForTesting - public void setSource(PaimonSource source) { - this.source = source; - } - - @Override - protected void convertPredicate() { - PaimonPredicateConverter paimonPredicateConverter = new PaimonPredicateConverter( - source.getPaimonTable().rowType()); - predicates = paimonPredicateConverter.convertToPaimonExpr(conjuncts); - } - - @Override - protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { - if (split instanceof PaimonSplit) { - setPaimonParams(rangeDesc, (PaimonSplit) split); - } - } - - @Override - protected Optional getSerializedTable() { - return Optional.of(serializedTable); - } - - @Override - public void createScanRangeLocations() throws UserException { - super.createScanRangeLocations(); - // Set paimon_predicate at ScanNode level to avoid redundant serialization in each split - String serializedPredicate = PaimonUtil.encodeObjectToString(predicates); - params.setPaimonPredicate(serializedPredicate); - setScanLevelPaimonOptions(); - } - - private void setScanLevelPaimonOptions() { - if (!backendPaimonOptions.isEmpty()) { - params.setPaimonOptions(backendPaimonOptions); - } - } - - private List getOrderedPathPartitionKeys() { - if (source == null) { - return Collections.emptyList(); - } - ExternalTable externalTable = source.getExternalTable(); - if (externalTable instanceof PaimonSysExternalTable - && !((PaimonSysExternalTable) externalTable).isDataTable()) { - return Collections.emptyList(); - } - return source.getPaimonTable().partitionKeys().stream() - .map(key -> key.toLowerCase(Locale.ROOT)) - .collect(Collectors.toList()); - } - - private void putHistorySchemaInfo(Long schemaId) { - if (currentQuerySchema.putIfAbsent(schemaId, Boolean.TRUE) == null) { - ExternalTable targetTable = source.getExternalTable(); - if (targetTable instanceof PaimonSysExternalTable) { - PaimonSysExternalTable sysTable = (PaimonSysExternalTable) targetTable; - if (!sysTable.isDataTable()) { - return; - } - } - - TableSchema tableSchema = PaimonUtils.getSchemaCacheValue(targetTable, schemaId).getTableSchema(); - params.addToHistorySchemaInfo(PaimonUtil.getHistorySchemaInfo(targetTable, tableSchema, - source.getCatalog().getEnableMappingVarbinary(), - source.getCatalog().getEnableMappingTimestampTz())); - } - } - - private void setPaimonParams(TFileRangeDesc rangeDesc, PaimonSplit paimonSplit) { - TTableFormatFileDesc tableFormatFileDesc = new TTableFormatFileDesc(); - tableFormatFileDesc.setTableFormatType(paimonSplit.getTableFormatType().value()); - TPaimonFileDesc fileDesc = new TPaimonFileDesc(); - org.apache.paimon.table.source.Split split = paimonSplit.getSplit(); - - String fileFormat = getFileFormat(paimonSplit.getPathString()); - if (split != null) { - // use jni reader or paimon-cpp reader - rangeDesc.setFormatType(TFileFormatType.FORMAT_JNI); - // Use Paimon native serialization for paimon-cpp reader - if (sessionVariable.isEnablePaimonCppReader() && split instanceof DataSplit) { - fileDesc.setPaimonSplit(PaimonUtil.encodeDataSplitToString((DataSplit) split)); - } else { - fileDesc.setPaimonSplit(PaimonUtil.encodeObjectToString(split)); - } - // Set table location for paimon-cpp reader - String tableLocation = source.getTableLocation(); - if (tableLocation != null) { - fileDesc.setPaimonTable(tableLocation); - } - rangeDesc.setSelfSplitWeight(paimonSplit.getSelfSplitWeight()); - } else { - // use native reader - if (fileFormat.equals("orc")) { - rangeDesc.setFormatType(TFileFormatType.FORMAT_ORC); - } else if (fileFormat.equals("parquet")) { - rangeDesc.setFormatType(TFileFormatType.FORMAT_PARQUET); - } else { - throw new RuntimeException("Unsupported file format: " + fileFormat); - } - - putHistorySchemaInfo(paimonSplit.getSchemaId()); - fileDesc.setSchemaId(paimonSplit.getSchemaId()); - } - fileDesc.setFileFormat(fileFormat); - // Hadoop conf is set at ScanNode level via params.properties in createScanRangeLocations(), - // no need to set it for each split to avoid redundant configuration - Optional optDeletionFile = paimonSplit.getDeletionFile(); - if (optDeletionFile.isPresent()) { - DeletionFile deletionFile = optDeletionFile.get(); - TPaimonDeletionFileDesc tDeletionFile = new TPaimonDeletionFileDesc(); - // convert the deletion file uri to make sure FileReader can read it in be - LocationPath locationPath = LocationPath.of(deletionFile.path(), storagePropertiesMap); - String path = locationPath.toStorageLocation().toString(); - tDeletionFile.setPath(path); - tDeletionFile.setOffset(deletionFile.offset()); - tDeletionFile.setLength(deletionFile.length()); - fileDesc.setDeletionFile(tDeletionFile); - } - if (paimonSplit.getRowCount().isPresent()) { - tableFormatFileDesc.setTableLevelRowCount(paimonSplit.getRowCount().get()); - } else { - // MUST explicitly set to -1, to be distinct from valid row count >= 0 - tableFormatFileDesc.setTableLevelRowCount(-1); - } - tableFormatFileDesc.setPaimonParams(fileDesc); - rangeDesc.unsetColumnsFromPath(); - rangeDesc.unsetColumnsFromPathKeys(); - rangeDesc.unsetColumnsFromPathIsNull(); - Map partitionValues = paimonSplit.getPaimonPartitionValues(); - List orderedPartitionKeys = getOrderedPathPartitionKeys(); - if (partitionValues != null && !orderedPartitionKeys.isEmpty()) { - List fromPathKeys = new ArrayList<>(); - List fromPathValues = new ArrayList<>(); - List fromPathIsNull = new ArrayList<>(); - for (String partitionKey : orderedPartitionKeys) { - if (!partitionValues.containsKey(partitionKey)) { - continue; - } - String partitionValue = partitionValues.get(partitionKey); - fromPathKeys.add(partitionKey); - fromPathValues.add(partitionValue != null ? partitionValue : ""); - fromPathIsNull.add(partitionValue == null); - } - if (!fromPathKeys.isEmpty()) { - rangeDesc.setColumnsFromPathKeys(fromPathKeys); - rangeDesc.setColumnsFromPath(fromPathValues); - rangeDesc.setColumnsFromPathIsNull(fromPathIsNull); - } - } - rangeDesc.setTableFormatParams(tableFormatFileDesc); - } - - @Override - protected List getDeleteFiles(TFileRangeDesc rangeDesc) { - List deleteFiles = new ArrayList<>(); - if (rangeDesc == null || !rangeDesc.isSetTableFormatParams()) { - return deleteFiles; - } - TTableFormatFileDesc tableFormatParams = rangeDesc.getTableFormatParams(); - if (tableFormatParams == null || !tableFormatParams.isSetPaimonParams()) { - return deleteFiles; - } - TPaimonFileDesc paimonParams = tableFormatParams.getPaimonParams(); - if (paimonParams == null || !paimonParams.isSetDeletionFile()) { - return deleteFiles; - } - TPaimonDeletionFileDesc deletionFile = paimonParams.getDeletionFile(); - if (deletionFile != null && deletionFile.isSetPath()) { - // Format: path [offset: offset, length: length] - deleteFiles.add(deletionFile.getPath()); - } - return deleteFiles; - } - - @Override - public List getSplits(int numBackends) throws UserException { - boolean forceJniScanner = sessionVariable.isForceJniScanner(); - // Paimon system tables need Paimon-side semantics: - // - binlog: pack/merge + array materialization - // - audit_log: rowkind / sequence-number projection - // TODO: Allow native reader after Doris native parquet/orc reader can materialize - // these system-table rows consistently with Paimon system-table semantics. - boolean forceJniForSystemTable = shouldForceJniForSystemTable(); - SessionVariable.IgnoreSplitType ignoreSplitType = SessionVariable.IgnoreSplitType - .valueOf(sessionVariable.getIgnoreSplitType()); - List splits = new ArrayList<>(); - List pushDownCountSplits = new ArrayList<>(); - long pushDownCountSum = 0; - - List paimonSplits = getPaimonSplitFromAPI(); - List dataSplits = new ArrayList<>(); - List nonDataSplits = new ArrayList<>(); - for (org.apache.paimon.table.source.Split split : paimonSplits) { - if (split instanceof DataSplit) { - dataSplits.add((DataSplit) split); - } else { - // Non-DataSplit types (e.g., from some system tables) will use JNI reader - nonDataSplits.add(split); - } - } - - // Handle non-DataSplit splits (typically from metadata system tables) - // These must use JNI reader as they can't be converted to raw files - for (org.apache.paimon.table.source.Split split : nonDataSplits) { - if (ignoreSplitType == SessionVariable.IgnoreSplitType.IGNORE_JNI) { - continue; - } - splits.add(new PaimonSplit(split)); - ++paimonSplitNum; - } - - boolean applyCountPushdown = getPushDownAggNoGroupingOp() == TPushAggOp.COUNT; - // Used to avoid repeatedly calculating partition info map for the same - // partition data. - // And for counting the number of selected partitions for this paimon table. - Map> partitionInfoMaps = new HashMap<>(); - boolean needPartitionMetadata = !getOrderedPathPartitionKeys().isEmpty(); - // if applyCountPushdown is true, we can't split the DataSplit - boolean hasDeterminedTargetFileSplitSize = false; - long targetFileSplitSize = 0; - for (DataSplit dataSplit : dataSplits) { - SplitStat splitStat = new SplitStat(); - splitStat.setRowCount(dataSplit.rowCount()); - - BinaryRow partitionValue = dataSplit.partition(); - Map partitionInfoMap = null; - if (needPartitionMetadata) { - partitionInfoMap = partitionInfoMaps.computeIfAbsent(partitionValue, k -> { - return PaimonUtil.getPartitionInfoMap( - source.getPaimonTable(), partitionValue, sessionVariable.getTimeZone()); - }); - } else { - partitionInfoMaps.put(partitionValue, null); - } - Optional> optRawFiles = dataSplit.convertToRawFiles(); - Optional> optDeletionFiles = dataSplit.deletionFiles(); - if (applyCountPushdown && dataSplit.mergedRowCountAvailable()) { - splitStat.setMergedRowCount(dataSplit.mergedRowCount()); - PaimonSplit split = new PaimonSplit(dataSplit); - split.setRowCount(dataSplit.mergedRowCount()); - if (partitionInfoMap != null) { - split.setPaimonPartitionValues(partitionInfoMap); - } - pushDownCountSplits.add(split); - pushDownCountSum += dataSplit.mergedRowCount(); - } else if (!forceJniScanner && !forceJniForSystemTable && supportNativeReader(optRawFiles)) { - if (ignoreSplitType == SessionVariable.IgnoreSplitType.IGNORE_NATIVE) { - continue; - } - if (!hasDeterminedTargetFileSplitSize) { - targetFileSplitSize = determineTargetFileSplitSize(dataSplits, isBatchMode()); - hasDeterminedTargetFileSplitSize = true; - } - splitStat.setType(SplitReadType.NATIVE); - splitStat.setRawFileConvertable(true); - List rawFiles = optRawFiles.get(); - for (int i = 0; i < rawFiles.size(); i++) { - RawFile file = rawFiles.get(i); - LocationPath locationPath = LocationPath.of(file.path(), storagePropertiesMap); - try { - List dorisSplits = fileSplitter.splitFile( - locationPath, - targetFileSplitSize, - null, - file.length(), - -1, - !applyCountPushdown, - Collections.emptyList(), - PaimonSplit.PaimonSplitCreator.DEFAULT); - for (Split dorisSplit : dorisSplits) { - PaimonSplit paimonSplit = (PaimonSplit) dorisSplit; - paimonSplit.setSchemaId(file.schemaId()); - paimonSplit.setPaimonPartitionValues(partitionInfoMap); - // try to set deletion file - if (optDeletionFiles.isPresent() && optDeletionFiles.get().get(i) != null) { - paimonSplit.setDeletionFile(optDeletionFiles.get().get(i)); - splitStat.setHasDeletionVector(true); - } - } - splits.addAll(dorisSplits); - ++rawFileSplitNum; - } catch (IOException e) { - throw new UserException("Paimon error to split file: " + e.getMessage(), e); - } - } - } else { - if (ignoreSplitType == SessionVariable.IgnoreSplitType.IGNORE_JNI) { - continue; - } - PaimonSplit jniSplit = new PaimonSplit(dataSplit); - jniSplit.setPaimonPartitionValues(partitionInfoMap); - splits.add(jniSplit); - ++paimonSplitNum; - } - - splitStats.add(splitStat); - } - - // if applyCountPushdown is true, calcute row count for count pushdown - if (applyCountPushdown && !pushDownCountSplits.isEmpty()) { - if (pushDownCountSum > COUNT_WITH_PARALLEL_SPLITS) { - int minSplits = sessionVariable.getParallelExecInstanceNum(scanContext.getClusterName()) - * numBackends; - pushDownCountSplits = pushDownCountSplits.subList(0, Math.min(pushDownCountSplits.size(), minSplits)); - } else { - pushDownCountSplits = Collections.singletonList(pushDownCountSplits.get(0)); - } - setPushDownCount(pushDownCountSum); - assignCountToSplits(pushDownCountSplits, pushDownCountSum); - splits.addAll(pushDownCountSplits); - } - - // We need to set the target size for all splits so that we can calculate the - // proportion of each split later. - splits.forEach(s -> s.setTargetSplitSize(sessionVariable.getFileSplitSize() > 0 - ? sessionVariable.getFileSplitSize() : sessionVariable.getMaxSplitSize())); - - this.selectedPartitionNum = partitionInfoMaps.size(); - return splits; - } - - @VisibleForTesting - Map getBackendPaimonOptions() { - if (source == null) { - return Collections.emptyMap(); - } - if (!(source.getCatalog() instanceof PaimonExternalCatalog)) { - return Collections.emptyMap(); - } - PaimonExternalCatalog catalog = (PaimonExternalCatalog) source.getCatalog(); - if (!(catalog.getCatalogProperty().getMetastoreProperties() instanceof PaimonJdbcMetaStoreProperties)) { - return Collections.emptyMap(); - } - PaimonJdbcMetaStoreProperties jdbcMetaStoreProperties = - (PaimonJdbcMetaStoreProperties) catalog.getCatalogProperty().getMetastoreProperties(); - return jdbcMetaStoreProperties.getBackendPaimonOptions(); - } - - @VisibleForTesting - boolean shouldForceJniForSystemTable() { - if (source == null) { - return false; - } - ExternalTable externalTable = source.getExternalTable(); - if (!(externalTable instanceof PaimonSysExternalTable)) { - return false; - } - PaimonSysExternalTable paimonSysExternalTable = (PaimonSysExternalTable) externalTable; - String sysTableType = paimonSysExternalTable.getSysTableType(); - return PAIMON_BINLOG_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType) - || PAIMON_AUDIT_LOG_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType); - } - - private long determineTargetFileSplitSize(List dataSplits, - boolean isBatchMode) { - if (sessionVariable.getFileSplitSize() > 0) { - return sessionVariable.getFileSplitSize(); - } - /** Paimon batch split mode will return 0. and FileSplitter - * will determine file split size. - */ - if (isBatchMode) { - return 0; - } - long result = sessionVariable.getMaxInitialSplitSize(); - long totalFileSize = 0; - boolean exceedInitialThreshold = false; - for (DataSplit dataSplit : dataSplits) { - Optional> rawFiles = dataSplit.convertToRawFiles(); - if (!supportNativeReader(rawFiles)) { - continue; - } - for (RawFile rawFile : rawFiles.get()) { - totalFileSize += rawFile.fileSize(); - if (!exceedInitialThreshold && totalFileSize - >= sessionVariable.getMaxSplitSize() * sessionVariable.getMaxInitialSplitNum()) { - exceedInitialThreshold = true; - } - } - } - result = exceedInitialThreshold ? sessionVariable.getMaxSplitSize() : result; - result = applyMaxFileSplitNumLimit(result, totalFileSize); - return result; - } - - @VisibleForTesting - public Map getIncrReadParams() throws UserException { - Map paimonScanParams = new HashMap<>(); - if (scanParams != null && scanParams.incrementalRead()) { - // Validate parameter combinations and get the result map - paimonScanParams = validateIncrementalReadParams(scanParams.getMapParams()); - } - return paimonScanParams; - } - - @VisibleForTesting - public List getPaimonSplitFromAPI() throws UserException { - long startTime = System.currentTimeMillis(); - try { - Table paimonTable = getProcessedTable(); - int[] projected = desc.getSlots().stream().mapToInt( - slot -> paimonTable.rowType() - .getFieldNames() - .stream() - .map(String::toLowerCase) - .collect(Collectors.toList()) - .indexOf(slot.getColumn().getName())) - .filter(i -> i >= 0) - .toArray(); - ReadBuilder readBuilder = paimonTable.newReadBuilder(); - TableScan scan = readBuilder.withFilter(predicates) - .withProjection(projected) - .newScan(); - PaimonMetricRegistry registry = new PaimonMetricRegistry(); - if (scan instanceof InnerTableScan) { - scan = ((InnerTableScan) scan).withMetricRegistry(registry); - } - List splits = scan.plan().splits(); - PaimonScanMetricsReporter.report(source.getTargetTable(), paimonTable.name(), registry); - if (!registry.getAllGroups().isEmpty()) { - registry.clear(); - } - return splits; - } finally { - if (getSummaryProfile() != null) { - getSummaryProfile().addExternalTableGetFileScanTasksTime(System.currentTimeMillis() - startTime); - } - } - } - - private String getFileFormat(String path) { - return FileFormatUtils.getFileFormatBySuffix(path).orElse(source.getFileFormatFromTableProperties()); - } - - @VisibleForTesting - public boolean supportNativeReader(Optional> optRawFiles) { - if (!optRawFiles.isPresent()) { - return false; - } - List files = optRawFiles.get().stream().map(RawFile::path).collect(Collectors.toList()); - for (String f : files) { - String splitFileFormat = getFileFormat(f); - if (!splitFileFormat.equals("orc") && !splitFileFormat.equals("parquet")) { - return false; - } - } - return true; - } - - @Override - public TFileFormatType getFileFormatType() throws DdlException, MetaNotFoundException { - return TFileFormatType.FORMAT_JNI; - } - - @Override - public List getPathPartitionKeys() throws DdlException, MetaNotFoundException { - return getOrderedPathPartitionKeys(); - } - - @Override - public TableIf getTargetTable() { - return desc.getTable(); - } - - @Override - protected Map getLocationProperties() { - return backendStorageProperties; - } - - @Override - public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { - StringBuilder sb = new StringBuilder(super.getNodeExplainString(prefix, detailLevel)); - sb.append(String.format("%spaimonNativeReadSplits=%d/%d\n", - prefix, rawFileSplitNum, (paimonSplitNum + rawFileSplitNum))); - - sb.append(prefix).append("predicatesFromPaimon:"); - if (predicates.isEmpty()) { - sb.append(" NONE\n"); - } else { - sb.append("\n"); - for (Predicate predicate : predicates) { - sb.append(prefix).append(prefix).append(predicate).append("\n"); - } - } - - if (detailLevel == TExplainLevel.VERBOSE) { - sb.append(prefix).append("PaimonSplitStats: \n"); - int size = splitStats.size(); - if (size <= 4) { - for (SplitStat splitStat : splitStats) { - sb.append(String.format("%s %s\n", prefix, splitStat)); - } - } else { - for (int i = 0; i < 3; i++) { - SplitStat splitStat = splitStats.get(i); - sb.append(String.format("%s %s\n", prefix, splitStat)); - } - int other = size - 4; - sb.append(prefix).append(" ... other ").append(other).append(" paimon split stats ...\n"); - SplitStat split = splitStats.get(size - 1); - sb.append(String.format("%s %s\n", prefix, split)); - } - } - return sb.toString(); - } - - private void assignCountToSplits(List splits, long totalCount) { - int size = splits.size(); - long countPerSplit = totalCount / size; - for (int i = 0; i < size - 1; i++) { - ((PaimonSplit) splits.get(i)).setRowCount(countPerSplit); - } - ((PaimonSplit) splits.get(size - 1)).setRowCount(countPerSplit + totalCount % size); - } - - @VisibleForTesting - public static Map validateIncrementalReadParams(Map params) throws UserException { - // Check if snapshot-based parameters exist - boolean hasStartSnapshotId = params.containsKey(DORIS_START_SNAPSHOT_ID) - && params.get(DORIS_START_SNAPSHOT_ID) != null; - boolean hasEndSnapshotId = params.containsKey(DORIS_END_SNAPSHOT_ID) - && params.get(DORIS_END_SNAPSHOT_ID) != null; - boolean hasIncrementalBetweenScanMode = params.containsKey(DORIS_INCREMENTAL_BETWEEN_SCAN_MODE) - && params.get(DORIS_INCREMENTAL_BETWEEN_SCAN_MODE) != null; - - // Check if timestamp-based parameters exist - boolean hasStartTimestamp = params.containsKey(DORIS_START_TIMESTAMP) - && params.get(DORIS_START_TIMESTAMP) != null; - boolean hasEndTimestamp = params.containsKey(DORIS_END_TIMESTAMP) && params.get(DORIS_END_TIMESTAMP) != null; - - // Check if any snapshot-based parameters are present - boolean hasSnapshotParams = hasStartSnapshotId || hasEndSnapshotId || hasIncrementalBetweenScanMode; - - // Check if any timestamp-based parameters are present - boolean hasTimestampParams = hasStartTimestamp || hasEndTimestamp; - - // Rule 2: The two groups are mutually exclusive - if (hasSnapshotParams && hasTimestampParams) { - throw new UserException( - "Cannot specify both snapshot-based parameters" - + "(startSnapshotId, endSnapshotId, incrementalBetweenScanMode) " - + "and timestamp-based parameters (startTimestamp, endTimestamp) at the same time"); - } - - // Validate snapshot-based parameters group - if (hasSnapshotParams) { - // Rule 3.1 & 3.2: DORIS_START_SNAPSHOT_ID is required - if (!hasStartSnapshotId) { - throw new UserException("startSnapshotId is required when using snapshot-based incremental read"); - } - - // Rule 3.3: DORIS_INCREMENTAL_BETWEEN_SCAN_MODE can only appear - // when both start and end snapshot IDs are specified - if (hasIncrementalBetweenScanMode && (!hasStartSnapshotId || !hasEndSnapshotId)) { - throw new UserException( - "incrementalBetweenScanMode can only be specified when" - + " both startSnapshotId and endSnapshotId are provided"); - } - - // Validate snapshot ID values - if (hasStartSnapshotId) { - try { - long startSId = Long.parseLong(params.get(DORIS_START_SNAPSHOT_ID)); - if (startSId < 0) { - throw new UserException("startSnapshotId must be greater than or equal to 0"); - } - } catch (NumberFormatException e) { - throw new UserException("Invalid startSnapshotId format: " + e.getMessage()); - } - } - - if (hasEndSnapshotId) { - try { - long endSId = Long.parseLong(params.get(DORIS_END_SNAPSHOT_ID)); - if (endSId < 0) { - throw new UserException("endSnapshotId must be greater than or equal to 0"); - } - } catch (NumberFormatException e) { - throw new UserException("Invalid endSnapshotId format: " + e.getMessage()); - } - } - - // Check if both snapshot IDs are present and validate their relationship - if (hasStartSnapshotId && hasEndSnapshotId) { - try { - long startSId = Long.parseLong(params.get(DORIS_START_SNAPSHOT_ID)); - long endSId = Long.parseLong(params.get(DORIS_END_SNAPSHOT_ID)); - if (startSId > endSId) { - throw new UserException("startSnapshotId must be less than or equal to endSnapshotId"); - } - } catch (NumberFormatException e) { - throw new UserException("Invalid snapshot ID format: " + e.getMessage()); - } - } - - // Validate DORIS_INCREMENTAL_BETWEEN_SCAN_MODE - if (hasIncrementalBetweenScanMode) { - String scanMode = params.get(DORIS_INCREMENTAL_BETWEEN_SCAN_MODE).toLowerCase(); - if (!scanMode.equals("auto") && !scanMode.equals("diff") - && !scanMode.equals("delta") && !scanMode.equals("changelog")) { - throw new UserException("incrementalBetweenScanMode must be one of: auto, diff, delta, changelog"); - } - } - } - - // Validate timestamp-based parameters group - if (hasTimestampParams) { - // Rule 4.1 & 4.2: DORIS_START_TIMESTAMP is required - if (!hasStartTimestamp) { - throw new UserException("startTimestamp is required when using timestamp-based incremental read"); - } - - // Validate timestamp values - if (hasStartTimestamp) { - try { - long startTS = Long.parseLong(params.get(DORIS_START_TIMESTAMP)); - if (startTS < 0) { - throw new UserException("startTimestamp must be greater than or equal to 0"); - } - } catch (NumberFormatException e) { - throw new UserException("Invalid startTimestamp format: " + e.getMessage()); - } - } - - if (hasEndTimestamp) { - try { - long endTS = Long.parseLong(params.get(DORIS_END_TIMESTAMP)); - if (endTS <= 0) { - throw new UserException("endTimestamp must be greater than 0"); - } - } catch (NumberFormatException e) { - throw new UserException("Invalid endTimestamp format: " + e.getMessage()); - } - } - - // Check if both timestamps are present and validate their relationship - if (hasStartTimestamp && hasEndTimestamp) { - try { - long startTS = Long.parseLong(params.get(DORIS_START_TIMESTAMP)); - long endTS = Long.parseLong(params.get(DORIS_END_TIMESTAMP)); - if (startTS >= endTS) { - throw new UserException("startTimestamp must be less than endTimestamp"); - } - } catch (NumberFormatException e) { - throw new UserException("Invalid timestamp format: " + e.getMessage()); - } - } - } - - // If no incremental parameters are provided at all, that's also invalid in this context - if (!hasSnapshotParams && !hasTimestampParams) { - throw new UserException( - "Invalid paimon incremental read params: at least one valid parameter group must be specified"); - } - - // Fill the result map based on parameter combinations - Map paimonScanParams = new HashMap<>(); - paimonScanParams.put(PAIMON_SCAN_SNAPSHOT_ID, null); - paimonScanParams.put(PAIMON_SCAN_MODE, null); - - if (hasSnapshotParams) { - paimonScanParams.put(PAIMON_SCAN_MODE, null); - if (hasStartSnapshotId && !hasEndSnapshotId) { - // Only startSnapshotId is specified - throw new UserException("endSnapshotId is required when using snapshot-based incremental read"); - } else if (hasStartSnapshotId && hasEndSnapshotId) { - // Both start and end snapshot IDs are specified - String startSId = params.get(DORIS_START_SNAPSHOT_ID); - String endSId = params.get(DORIS_END_SNAPSHOT_ID); - paimonScanParams.put(PAIMON_INCREMENTAL_BETWEEN, startSId + "," + endSId); - } - - // Add incremental between scan mode if present - if (hasIncrementalBetweenScanMode) { - paimonScanParams.put(PAIMON_INCREMENTAL_BETWEEN_SCAN_MODE, - params.get(DORIS_INCREMENTAL_BETWEEN_SCAN_MODE)); - } - } - - if (hasTimestampParams) { - String startTS = params.get(DORIS_START_TIMESTAMP); - String endTS = params.get(DORIS_END_TIMESTAMP); - - if (hasStartTimestamp && !hasEndTimestamp) { - // Only startTimestamp is specified - paimonScanParams.put(PAIMON_INCREMENTAL_BETWEEN_TIMESTAMP, startTS + "," + Long.MAX_VALUE); - } else if (hasStartTimestamp && hasEndTimestamp) { - // Both start and end timestamps are specified - paimonScanParams.put(PAIMON_INCREMENTAL_BETWEEN_TIMESTAMP, startTS + "," + endTS); - } - } - - return paimonScanParams; - } - - private Table getProcessedTable() throws UserException { - Table baseTable = source.getPaimonTable(); - TableScanParams theScanParams = getScanParams(); - if (source.getExternalTable() instanceof PaimonSysExternalTable) { - if (theScanParams != null) { - throw new UserException("Paimon system tables do not support scan params."); - } - if (getQueryTableSnapshot() != null) { - throw new UserException("Paimon system tables do not support time travel."); - } - } - if (theScanParams != null && getQueryTableSnapshot() != null) { - throw new UserException("Can not specify scan params and table snapshot at same time."); - } - - if (theScanParams != null && theScanParams.incrementalRead()) { - return baseTable.copy(getIncrReadParams()); - } - return baseTable; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSource.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSource.java deleted file mode 100644 index 43c6ef4170168c..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSource.java +++ /dev/null @@ -1,102 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.source; - -import org.apache.doris.analysis.TupleDescriptor; -import org.apache.doris.catalog.TableIf; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.mvcc.MvccSnapshot; -import org.apache.doris.datasource.mvcc.MvccUtil; -import org.apache.doris.datasource.paimon.PaimonExternalTable; -import org.apache.doris.datasource.paimon.PaimonSysExternalTable; -import org.apache.doris.thrift.TFileAttributes; - -import com.google.common.annotations.VisibleForTesting; -import org.apache.paimon.table.FileStoreTable; -import org.apache.paimon.table.Table; - -import java.util.Optional; - -public class PaimonSource { - private final ExternalTable paimonExtTable; - private final Table originTable; - private final TupleDescriptor desc; - - @VisibleForTesting - public PaimonSource() { - this.desc = null; - this.paimonExtTable = null; - this.originTable = null; - } - - public PaimonSource(TupleDescriptor desc) { - this.desc = desc; - this.paimonExtTable = (ExternalTable) desc.getTable(); - this.originTable = resolvePaimonTable(paimonExtTable); - } - - public TupleDescriptor getDesc() { - return desc; - } - - public Table getPaimonTable() { - return originTable; - } - - public TableIf getTargetTable() { - return paimonExtTable; - } - - public ExternalTable getExternalTable() { - return paimonExtTable; - } - - private Table resolvePaimonTable(ExternalTable table) { - Optional snapshot = MvccUtil.getSnapshotFromContext(table); - if (table instanceof PaimonExternalTable) { - return ((PaimonExternalTable) table).getPaimonTable(snapshot); - } - if (table instanceof PaimonSysExternalTable) { - return ((PaimonSysExternalTable) table).getSysPaimonTable(); - } - throw new IllegalArgumentException( - "Expected Paimon table but got " + table.getClass().getSimpleName()); - } - - public TFileAttributes getFileAttributes() throws UserException { - return new TFileAttributes(); - } - - public ExternalCatalog getCatalog() { - return paimonExtTable.getCatalog(); - } - - public String getFileFormatFromTableProperties() { - return originTable.options().getOrDefault("file.format", "parquet"); - } - - public String getTableLocation() { - if (originTable instanceof FileStoreTable) { - return ((FileStoreTable) originTable).location().toString(); - } - // Fallback to path option - return originTable.options().get("path"); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSplit.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSplit.java deleted file mode 100644 index 4a8808517b2176..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSplit.java +++ /dev/null @@ -1,159 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.source; - -import org.apache.doris.common.util.LocationPath; -import org.apache.doris.datasource.FileSplit; -import org.apache.doris.datasource.SplitCreator; -import org.apache.doris.datasource.TableFormatType; - -import org.apache.paimon.io.DataFileMeta; -import org.apache.paimon.table.source.DataSplit; -import org.apache.paimon.table.source.DeletionFile; -import org.apache.paimon.table.source.Split; - -import java.util.Collections; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.UUID; - -public class PaimonSplit extends FileSplit { - private static final LocationPath DUMMY_PATH = LocationPath.of("/dummyPath"); - // Paimon split - can be DataSplit or other Split types (e.g., from system tables) - private Split paimonSplit; - private TableFormatType tableFormatType; - private Optional optDeletionFile = Optional.empty(); - private Optional optRowCount = Optional.empty(); - private Optional schemaId = Optional.empty(); - private Map paimonPartitionValues = null; - - /** - * Constructor for Paimon splits. - * Handles both DataSplit (regular data tables) and other Split types (system tables). - */ - public PaimonSplit(Split paimonSplit) { - super(DUMMY_PATH, 0, 0, 0, 0, null, Collections.emptyList()); - this.paimonSplit = paimonSplit; - this.tableFormatType = TableFormatType.PAIMON; - - if (paimonSplit instanceof DataSplit) { - // For DataSplit, extract file info for path and weight calculation - DataSplit dataSplit = (DataSplit) paimonSplit; - List dataFileMetas = dataSplit.dataFiles(); - this.path = LocationPath.of("/" + dataFileMetas.get(0).fileName()); - this.selfSplitWeight = dataFileMetas.stream().mapToLong(DataFileMeta::fileSize).sum(); - } else { - // For non-DataSplit (e.g., system tables), use row count as weight - this.selfSplitWeight = paimonSplit.rowCount(); - } - } - - private PaimonSplit(LocationPath file, long start, long length, long fileLength, long modificationTime, - String[] hosts, List partitionList) { - super(file, start, length, fileLength, modificationTime, hosts, - partitionList == null ? Collections.emptyList() : partitionList); - this.tableFormatType = TableFormatType.PAIMON; - this.selfSplitWeight = length; - } - - @Override - public String getConsistentHashString() { - if (this.path == DUMMY_PATH) { - return UUID.randomUUID().toString(); - } - return getPathString(); - } - - /** - * Returns the underlying Paimon split. - * For JNI reader serialization. - */ - public Split getSplit() { - return paimonSplit; - } - - /** - * Returns the split as DataSplit if it's a DataSplit instance. - * Returns null if this is a non-DataSplit system table split. - */ - public DataSplit getDataSplit() { - return paimonSplit instanceof DataSplit ? (DataSplit) paimonSplit : null; - } - - public TableFormatType getTableFormatType() { - return tableFormatType; - } - - public void setTableFormatType(TableFormatType tableFormatType) { - this.tableFormatType = tableFormatType; - } - - public Optional getDeletionFile() { - return optDeletionFile; - } - - public void setDeletionFile(DeletionFile deletionFile) { - this.selfSplitWeight += deletionFile.length(); - this.optDeletionFile = Optional.of(deletionFile); - } - - public Optional getRowCount() { - return optRowCount; - } - - public void setRowCount(long rowCount) { - this.optRowCount = Optional.of(rowCount); - } - - public void setSchemaId(long schemaId) { - this.schemaId = Optional.of(schemaId); - } - - public Long getSchemaId() { - return schemaId.orElse(null); - } - - public void setPaimonPartitionValues(Map paimonPartitionValues) { - this.paimonPartitionValues = paimonPartitionValues; - } - - public Map getPaimonPartitionValues() { - return paimonPartitionValues; - } - - public static class PaimonSplitCreator implements SplitCreator { - - static final PaimonSplitCreator DEFAULT = new PaimonSplitCreator(); - - @Override - public org.apache.doris.spi.Split create(LocationPath path, - long start, - long length, - long fileLength, - long fileSplitSize, - long modificationTime, - String[] hosts, - List partitionValues) { - PaimonSplit split = new PaimonSplit(path, start, length, fileLength, - modificationTime, hosts, partitionValues); - split.setTargetSplitSize(fileSplitSize); - return split; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java deleted file mode 100644 index d490474489d59d..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java +++ /dev/null @@ -1,162 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.source; - -import org.apache.doris.analysis.BoolLiteral; -import org.apache.doris.analysis.DateLiteral; -import org.apache.doris.analysis.DecimalLiteral; -import org.apache.doris.analysis.FloatLiteral; -import org.apache.doris.analysis.IntLiteral; -import org.apache.doris.analysis.LiteralExpr; - -import org.apache.paimon.data.BinaryString; -import org.apache.paimon.data.Decimal; -import org.apache.paimon.data.Timestamp; -import org.apache.paimon.types.BigIntType; -import org.apache.paimon.types.BooleanType; -import org.apache.paimon.types.CharType; -import org.apache.paimon.types.DataType; -import org.apache.paimon.types.DataTypeDefaultVisitor; -import org.apache.paimon.types.DateType; -import org.apache.paimon.types.DecimalType; -import org.apache.paimon.types.DoubleType; -import org.apache.paimon.types.FloatType; -import org.apache.paimon.types.IntType; -import org.apache.paimon.types.SmallIntType; -import org.apache.paimon.types.TimestampType; -import org.apache.paimon.types.TinyIntType; -import org.apache.paimon.types.VarCharType; - -import java.math.BigDecimal; -import java.time.LocalDate; -import java.util.Calendar; -import java.util.TimeZone; - -/** - * Convert LiteralExpr to paimon value. - */ -public class PaimonValueConverter extends DataTypeDefaultVisitor { - private LiteralExpr expr; - - public PaimonValueConverter(LiteralExpr expr) { - this.expr = expr; - } - - public BinaryString visit(VarCharType varCharType) { - return BinaryString.fromString(expr.getStringValue()); - } - - public BinaryString visit(CharType charType) { - // Currently, Paimon does not support predicate push-down for char - // ref: org.apache.paimon.predicate.PredicateBuilder.convertJavaObject - return null; - } - - public Boolean visit(BooleanType booleanType) { - if (expr instanceof BoolLiteral) { - BoolLiteral boolLiteral = (BoolLiteral) expr; - return boolLiteral.getValue(); - } - return null; - } - - public Decimal visit(DecimalType decimalType) { - if (expr instanceof DecimalLiteral) { - DecimalLiteral decimalLiteral = (DecimalLiteral) expr; - BigDecimal value = decimalLiteral.getValue(); - return Decimal.fromBigDecimal(value, value.precision(), value.scale()); - } - return null; - } - - public Short visit(SmallIntType smallIntType) { - if (expr instanceof IntLiteral) { - IntLiteral intLiteral = (IntLiteral) expr; - return (short) intLiteral.getValue(); - } - return null; - } - - public Byte visit(TinyIntType tinyIntType) { - if (expr instanceof IntLiteral) { - IntLiteral intLiteral = (IntLiteral) expr; - return (byte) intLiteral.getValue(); - } - return null; - } - - - public Integer visit(IntType intType) { - if (expr instanceof IntLiteral) { - IntLiteral intLiteral = (IntLiteral) expr; - return (int) intLiteral.getValue(); - } - return null; - } - - public Long visit(BigIntType bigIntType) { - if (expr instanceof IntLiteral) { - IntLiteral intLiteral = (IntLiteral) expr; - return intLiteral.getValue(); - } - return null; - } - - // when a = 9.1,paimon can get data,doris can not get data - // when a > 9.1,paimon can not get data,doris can get data - // paimon is no problem,but we consistent with Doris internal table - // Therefore, comment out this code - public Float visit(FloatType floatType) { - return null; - } - - public Double visit(DoubleType doubleType) { - if (expr instanceof FloatLiteral) { - FloatLiteral floatLiteral = (FloatLiteral) expr; - return floatLiteral.getValue(); - } - return null; - } - - public Integer visit(DateType dateType) { - if (expr instanceof DateLiteral) { - DateLiteral dateLiteral = (DateLiteral) expr; - long l = LocalDate.of((int) dateLiteral.getYear(), (int) dateLiteral.getMonth(), (int) dateLiteral.getDay()) - .toEpochDay(); - return (int) l; - } - return null; - } - - public Timestamp visit(TimestampType timestampType) { - if (expr instanceof DateLiteral) { - DateLiteral dateLiteral = (DateLiteral) expr; - Calendar instance = Calendar.getInstance(TimeZone.getTimeZone("GMT")); - instance.set((int) dateLiteral.getYear(), (int) (dateLiteral.getMonth() - 1), (int) dateLiteral.getDay(), - (int) dateLiteral.getHour(), (int) dateLiteral.getMinute(), (int) dateLiteral.getSecond()); - return Timestamp - .fromEpochMillis(instance.getTimeInMillis() / 1000 * 1000 + dateLiteral.getMicrosecond() / 1000); - } - return null; - } - - @Override - protected Object defaultMethod(DataType dataType) { - return null; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java index 9bc77d543d3d59..b205257b8a3232 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java @@ -17,7 +17,6 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.property.storage.StorageProperties; import com.aliyun.datalake.metastore.common.DataLakeConfig; @@ -111,6 +110,6 @@ protected String getMetastoreType() { @Override public String getPaimonCatalogType() { - return PaimonExternalCatalog.PAIMON_DLF; + return "dlf"; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java index 8acf12b2056bc2..5762f97a9082b6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java @@ -18,7 +18,6 @@ package org.apache.doris.datasource.property.metastore; import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.property.storage.HdfsProperties; import org.apache.doris.datasource.property.storage.StorageProperties; @@ -79,6 +78,6 @@ protected String getMetastoreType() { @Override public String getPaimonCatalogType() { - return PaimonExternalCatalog.PAIMON_FILESYSTEM; + return "filesystem"; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java index e7e6689d3e3cab..f8d6404bdea367 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java @@ -18,7 +18,6 @@ package org.apache.doris.datasource.property.metastore; import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.foundation.property.ConnectorProperty; @@ -56,7 +55,7 @@ public class PaimonHMSMetaStoreProperties extends AbstractPaimonProperties { @Override public String getPaimonCatalogType() { - return PaimonExternalCatalog.PAIMON_HMS; + return "hms"; } protected PaimonHMSMetaStoreProperties(Map props) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java index 9bd9870718d597..a122dc020293dc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java @@ -19,7 +19,6 @@ import org.apache.doris.catalog.JdbcResource; import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.property.storage.HdfsProperties; import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.foundation.property.ConnectorProperty; @@ -97,7 +96,7 @@ protected PaimonJdbcMetaStoreProperties(Map props) { @Override public String getPaimonCatalogType() { - return PaimonExternalCatalog.PAIMON_JDBC; + return "jdbc"; } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java index 465fc873b7c5d5..48f9246150e165 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java @@ -17,7 +17,6 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.foundation.property.ConnectorProperty; import org.apache.doris.foundation.property.ParamRules; @@ -72,7 +71,7 @@ public void initNormalizeAndCheckProps() { @Override public String getPaimonCatalogType() { - return PaimonExternalCatalog.PAIMON_REST; + return "rest"; } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/NativeSysTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/NativeSysTable.java index 010a73d0319b9f..426438d7b88793 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/NativeSysTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/NativeSysTable.java @@ -32,8 +32,6 @@ * *

Subclasses must implement {@link #createSysExternalTable(ExternalTable)} to create * the appropriate system external table instance (e.g., PaimonSysExternalTable). - * - * @see PaimonSysTable */ public abstract class NativeSysTable extends SysTable { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PaimonSysTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PaimonSysTable.java deleted file mode 100644 index 7873cc0b95ec6a..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PaimonSysTable.java +++ /dev/null @@ -1,68 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.systable; - -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.paimon.PaimonExternalTable; -import org.apache.doris.datasource.paimon.PaimonSysExternalTable; - -import org.apache.paimon.table.system.SystemTableLoader; - -import java.util.Collections; -import java.util.Map; -import java.util.function.Function; -import java.util.stream.Collectors; - -/** - * System table type for Paimon system tables. - * - *

Paimon system tables are classified into two categories: - *

    - *
  • Data tables (e.g., binlog, audit_log, ro): Read actual ORC/Parquet data files, - * benefit from native vectorized readers
    • - *
  • Metadata tables (snapshots, partitions, etc.): Read metadata/manifest files, - * use JNI readers
    • - *
- * - *

All Paimon system tables use the native table execution path (FileQueryScanNode) - * instead of the TVF path (MetadataScanNode). - */ -public class PaimonSysTable extends NativeSysTable { - - /** - * All supported Paimon system tables (both data and metadata). - * Key is the system table name (e.g., "snapshots", "binlog"). - */ - public static final Map SUPPORTED_SYS_TABLES = Collections.unmodifiableMap( - SystemTableLoader.SYSTEM_TABLES.stream() - .map(PaimonSysTable::new) - .collect(Collectors.toMap(SysTable::getSysTableName, Function.identity()))); - - private PaimonSysTable(String tableName) { - super(tableName); - } - - @Override - public ExternalTable createSysExternalTable(ExternalTable sourceTable) { - if (!(sourceTable instanceof PaimonExternalTable)) { - throw new IllegalArgumentException( - "Expected PaimonExternalTable but got " + sourceTable.getClass().getSimpleName()); - } - return new PaimonSysExternalTable((PaimonExternalTable) sourceTable, getSysTableName()); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PluginDrivenSysTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PluginDrivenSysTable.java index 445184c37254aa..bc28fd8f0ea9f8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PluginDrivenSysTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/systable/PluginDrivenSysTable.java @@ -24,7 +24,7 @@ /** * Generic {@link NativeSysTable} for plugin-driven connectors. * - *

Unlike {@link PaimonSysTable} (which enumerates a fixed connector-specific set), instances of this + *

Unlike {@code PaimonSysTable} (which enumerates a fixed connector-specific set), instances of this * class are created on demand by {@link PluginDrivenExternalTable#getSupportedSysTables()} from the * names the connector SPI reports. {@link #createSysExternalTable(ExternalTable)} builds the transient * {@link PluginDrivenSysExternalTable} that the planner executes through the native table path.

diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/UserAuthentication.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/UserAuthentication.java index d44925946a3912..8f9c6d0a06de46 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/UserAuthentication.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/UserAuthentication.java @@ -29,7 +29,6 @@ import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.PluginDrivenSysExternalTable; import org.apache.doris.datasource.iceberg.IcebergSysExternalTable; -import org.apache.doris.datasource.paimon.PaimonSysExternalTable; import org.apache.doris.mysql.privilege.AccessControllerManager; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.qe.ConnectContext; @@ -55,10 +54,7 @@ public static void checkPermission(TableIf table, ConnectContext connectContext, } TableIf authTable = table; Set authColumns = columns; - if (table instanceof PaimonSysExternalTable) { - authTable = ((PaimonSysExternalTable) table).getSourceTable(); - authColumns = Collections.emptySet(); - } else if (table instanceof IcebergSysExternalTable) { + if (table instanceof IcebergSysExternalTable) { authTable = ((IcebergSysExternalTable) table).getSourceTable(); authColumns = Collections.emptySet(); } else if (table instanceof PluginDrivenSysExternalTable) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java index 8ba4b0c5a2c20d..8fbc0b09586259 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java @@ -48,9 +48,6 @@ import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergExternalCatalog; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; -import org.apache.doris.datasource.paimon.PaimonExternalDatabase; -import org.apache.doris.datasource.paimon.PaimonExternalTable; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.analyzer.UnboundSlot; import org.apache.doris.nereids.properties.OrderKey; @@ -75,13 +72,10 @@ import com.google.common.base.Strings; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; -import org.apache.paimon.partition.Partition; import java.util.ArrayList; -import java.util.Collections; import java.util.Comparator; import java.util.HashMap; -import java.util.LinkedHashSet; import java.util.List; import java.util.Map; import java.util.Optional; @@ -207,8 +201,7 @@ protected void validate(ConnectContext ctx) throws AnalysisException { // disallow unsupported catalog if (!(catalog.isInternalCatalog() || catalog instanceof HMSExternalCatalog - || catalog instanceof PluginDrivenExternalCatalog - || catalog instanceof PaimonExternalCatalog)) { + || catalog instanceof PluginDrivenExternalCatalog)) { throw new AnalysisException(String.format("Catalog of type '%s' is not allowed in ShowPartitionsCommand", catalog.getType())); } @@ -366,52 +359,6 @@ private boolean hasPartitionStatsCapability() { && connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_PARTITION_STATS); } - private ShowResultSet handleShowPaimonTablePartitions() throws AnalysisException { - PaimonExternalCatalog paimonCatalog = (PaimonExternalCatalog) catalog; - String db = tableName.getDb(); - String tbl = tableName.getTbl(); - - PaimonExternalDatabase database = (PaimonExternalDatabase) paimonCatalog.getDb(db) - .orElseThrow(() -> new AnalysisException("Paimon database '" + db + "' does not exist")); - PaimonExternalTable paimonTable = database.getTable(tbl) - .orElseThrow(() -> new AnalysisException("Paimon table '" + db + "." + tbl + "' does not exist")); - - Map partitionSnapshot = paimonTable.getPartitionSnapshot(Optional.empty()); - if (partitionSnapshot == null) { - partitionSnapshot = Collections.emptyMap(); - } - - LinkedHashSet partitionColumnNames = paimonTable - .getPartitionColumns(Optional.empty()) - .stream() - .map(Column::getName) - .collect(Collectors.toCollection(LinkedHashSet::new)); - String partitionColumnsStr = String.join(",", partitionColumnNames); - - List> rows = partitionSnapshot - .entrySet() - .stream() - .map(entry -> { - List row = new ArrayList<>(5); - row.add(entry.getKey()); - row.add(partitionColumnsStr); - row.add(String.valueOf(entry.getValue().recordCount())); - row.add(String.valueOf(entry.getValue().fileSizeInBytes())); - row.add(String.valueOf(entry.getValue().fileCount())); - return row; - }).collect(Collectors.toList()); - // sort by partition name - if (orderByPairs != null && orderByPairs.get(0).isDesc()) { - rows.sort(Comparator.comparing(x -> x.get(0), Comparator.reverseOrder())); - } else { - rows.sort(Comparator.comparing(x -> x.get(0))); - } - - rows = applyLimit(limit, offset, rows); - - return new ShowResultSet(getMetaData(), rows); - } - private ShowResultSet handleShowHMSTablePartitions() throws AnalysisException { HMSExternalCatalog hmsCatalog = (HMSExternalCatalog) catalog; List> rows = new ArrayList<>(); @@ -477,8 +424,6 @@ protected ShowResultSet handleShowPartitions(ConnectContext ctx, StmtExecutor ex return new ShowResultSet(getMetaData(), rows); } else if (catalog instanceof PluginDrivenExternalCatalog) { return handleShowPluginDrivenTablePartitions(); - } else if (catalog instanceof PaimonExternalCatalog) { - return handleShowPaimonTablePartitions(); } else { return handleShowHMSTablePartitions(); } @@ -502,10 +447,10 @@ public ShowResultSetMetaData getMetaData() { builder.addColumn(new Column("Partition", ScalarType.createVarchar(60))); builder.addColumn(new Column("Lower Bound", ScalarType.createVarchar(100))); builder.addColumn(new Column("Upper Bound", ScalarType.createVarchar(100))); - } else if (catalog instanceof PaimonExternalCatalog || hasPartitionStatsCapability()) { - // Legacy paimon catalog (pre-cutover) OR a plugin connector that declares - // SUPPORTS_PARTITION_STATS (paimon-after-cutover): 5-column rich result. Must match the - // row width built in handleShowPluginDrivenTablePartitions(). + } else if (hasPartitionStatsCapability()) { + // A plugin connector that declares SUPPORTS_PARTITION_STATS (paimon after cutover): + // 5-column rich result. Must match the row width built in + // handleShowPluginDrivenTablePartitions(). builder.addColumn(new Column("Partition", ScalarType.createVarchar(300))) .addColumn(new Column("PartitionKey", ScalarType.createVarchar(300))) .addColumn(new Column("RecordCount", ScalarType.createVarchar(300))) diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java index 85527090abb5d3..1c295128d2e25f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java @@ -26,7 +26,6 @@ import org.apache.doris.datasource.metacache.ExternalMetaCache; import org.apache.doris.datasource.metacache.MetaCacheEntry; import org.apache.doris.datasource.metacache.MetaCacheEntryStats; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.junit.After; import org.junit.Assert; @@ -78,10 +77,6 @@ public void testRouteByCatalogType() { new IcebergHMSExternalCatalog(2L, "iceberg", null, Collections.emptyMap(), ""), 2L); Assert.assertEquals(java.util.Collections.singletonList("iceberg"), icebergEngines); - List paimonEngines = metaCacheMgr.resolveCatalogEngineNamesForTest( - new PaimonExternalCatalog(3L, "paimon", null, Collections.emptyMap(), ""), 3L); - Assert.assertEquals(java.util.Collections.singletonList("paimon"), paimonEngines); - List dorisEngines = metaCacheMgr.resolveCatalogEngineNamesForTest( new RemoteDorisExternalCatalog(5L, "doris", null, Collections.emptyMap(), ""), 5L); Assert.assertEquals(java.util.Collections.singletonList("doris"), dorisEngines); @@ -138,7 +133,7 @@ public void testLifecycleRoutingOnlyTouchesSupportedEngine() throws Exception { RecordingExternalMetaCache iceberg = new RecordingExternalMetaCache( "iceberg", Collections.emptyList(), catalog -> catalog instanceof HMSExternalCatalog); RecordingExternalMetaCache paimon = new RecordingExternalMetaCache( - "paimon", Collections.emptyList(), catalog -> catalog instanceof PaimonExternalCatalog); + "paimon", Collections.emptyList(), catalog -> false); ExternalMetaCacheMgr metaCacheMgr = newManagerWithCaches(hive, hudi, iceberg, paimon); long catalogId = 8L; @@ -187,7 +182,7 @@ public void testMissingCatalogLifecycleOnlyTouchesInitializedEngine() throws Exc RecordingExternalMetaCache hive = new RecordingExternalMetaCache( "hive", Collections.singletonList("hms"), catalog -> catalog instanceof HMSExternalCatalog); RecordingExternalMetaCache paimon = new RecordingExternalMetaCache( - "paimon", Collections.emptyList(), catalog -> catalog instanceof PaimonExternalCatalog); + "paimon", Collections.emptyList(), catalog -> false); ExternalMetaCacheMgr metaCacheMgr = newManagerWithCaches(hive, paimon); long catalogId = 9L; diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCacheTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCacheTest.java deleted file mode 100644 index dc33354f6f93fd..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCacheTest.java +++ /dev/null @@ -1,123 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.datasource.metacache.MetaCacheEntryStats; - -import org.junit.Assert; -import org.junit.Test; - -import java.util.Collections; -import java.util.Map; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; - -public class PaimonExternalMetaCacheTest { - - @Test - public void testInvalidateTablePrecise() { - ExecutorService executor = Executors.newSingleThreadExecutor(); - try { - PaimonExternalMetaCache cache = new PaimonExternalMetaCache(executor); - long catalogId = 1L; - cache.initCatalog(catalogId, Collections.emptyMap()); - NameMapping t1 = new NameMapping(catalogId, "db1", "tbl1", "rdb1", "rtbl1"); - NameMapping t2 = new NameMapping(catalogId, "db1", "tbl2", "rdb1", "rtbl2"); - - org.apache.doris.datasource.metacache.MetaCacheEntry tableEntry = - cache.entry(catalogId, PaimonExternalMetaCache.ENTRY_TABLE, - NameMapping.class, PaimonTableCacheValue.class); - tableEntry.put(t1, new PaimonTableCacheValue(null, - () -> new PaimonSnapshotCacheValue(PaimonPartitionInfo.EMPTY, new PaimonSnapshot(1L, 1L, null)))); - tableEntry.put(t2, new PaimonTableCacheValue(null, - () -> new PaimonSnapshotCacheValue(PaimonPartitionInfo.EMPTY, new PaimonSnapshot(2L, 2L, null)))); - - cache.invalidateTable(catalogId, "db1", "tbl1"); - - Assert.assertNull(tableEntry.getIfPresent(t1)); - Assert.assertNotNull(tableEntry.getIfPresent(t2)); - } finally { - executor.shutdownNow(); - } - } - - @Test - public void testInvalidateDbAndStats() { - ExecutorService executor = Executors.newSingleThreadExecutor(); - try { - PaimonExternalMetaCache cache = new PaimonExternalMetaCache(executor); - long catalogId = 1L; - cache.initCatalog(catalogId, Collections.emptyMap()); - NameMapping db1Table = new NameMapping(catalogId, "db1", "tbl1", "rdb1", "rtbl1"); - NameMapping db2Table = new NameMapping(catalogId, "db2", "tbl1", "rdb2", "rtbl1"); - - org.apache.doris.datasource.metacache.MetaCacheEntry tableEntry = - cache.entry(catalogId, PaimonExternalMetaCache.ENTRY_TABLE, - NameMapping.class, PaimonTableCacheValue.class); - tableEntry.put(db1Table, new PaimonTableCacheValue(null, - () -> new PaimonSnapshotCacheValue(PaimonPartitionInfo.EMPTY, new PaimonSnapshot(1L, 1L, null)))); - tableEntry.put(db2Table, new PaimonTableCacheValue(null, - () -> new PaimonSnapshotCacheValue(PaimonPartitionInfo.EMPTY, new PaimonSnapshot(2L, 2L, null)))); - - org.apache.doris.datasource.metacache.MetaCacheEntry schemaEntry = - cache.entry(catalogId, PaimonExternalMetaCache.ENTRY_SCHEMA, - PaimonSchemaCacheKey.class, SchemaCacheValue.class); - PaimonSchemaCacheKey db1Schema = new PaimonSchemaCacheKey(db1Table, 1L); - PaimonSchemaCacheKey db2Schema = new PaimonSchemaCacheKey(db2Table, 2L); - schemaEntry.put(db1Schema, new SchemaCacheValue(Collections.emptyList())); - schemaEntry.put(db2Schema, new SchemaCacheValue(Collections.emptyList())); - - cache.invalidateDb(catalogId, "db1"); - - Assert.assertNull(tableEntry.getIfPresent(db1Table)); - Assert.assertNotNull(tableEntry.getIfPresent(db2Table)); - Assert.assertNull(schemaEntry.getIfPresent(db1Schema)); - Assert.assertNotNull(schemaEntry.getIfPresent(db2Schema)); - - Map stats = cache.stats(catalogId); - Assert.assertTrue(stats.containsKey(PaimonExternalMetaCache.ENTRY_TABLE)); - Assert.assertTrue(stats.containsKey(PaimonExternalMetaCache.ENTRY_SCHEMA)); - } finally { - executor.shutdownNow(); - } - } - - @Test - public void testSchemaStatsWhenSchemaCacheDisabled() { - ExecutorService executor = Executors.newSingleThreadExecutor(); - try { - PaimonExternalMetaCache cache = new PaimonExternalMetaCache(executor); - long catalogId = 1L; - Map properties = com.google.common.collect.Maps.newHashMap(); - properties.put(ExternalCatalog.SCHEMA_CACHE_TTL_SECOND, "0"); - cache.initCatalog(catalogId, properties); - - Map stats = cache.stats(catalogId); - MetaCacheEntryStats schemaStats = stats.get(PaimonExternalMetaCache.ENTRY_SCHEMA); - Assert.assertNotNull(schemaStats); - Assert.assertEquals(0L, schemaStats.getTtlSecond()); - Assert.assertTrue(schemaStats.isConfigEnabled()); - Assert.assertFalse(schemaStats.isEffectiveEnabled()); - } finally { - executor.shutdownNow(); - } - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java deleted file mode 100644 index ae0bb3b3af53e4..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java +++ /dev/null @@ -1,259 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.common.DdlException; -import org.apache.doris.common.UserException; -import org.apache.doris.nereids.parser.NereidsParser; -import org.apache.doris.nereids.trees.plans.commands.CreateTableCommand; -import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; -import org.apache.doris.nereids.trees.plans.logical.LogicalPlan; -import org.apache.doris.qe.ConnectContext; - -import com.google.common.collect.Maps; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.FileSystemCatalog; -import org.apache.paimon.catalog.Identifier; -import org.apache.paimon.hive.HiveCatalog; -import org.apache.paimon.table.Table; -import org.apache.paimon.types.BigIntType; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.DateType; -import org.apache.paimon.types.DecimalType; -import org.apache.paimon.types.DoubleType; -import org.apache.paimon.types.FloatType; -import org.apache.paimon.types.IntType; -import org.apache.paimon.types.TimestampType; -import org.apache.paimon.types.VarCharType; -import org.junit.Assert; -import org.junit.BeforeClass; -import org.junit.Test; -import org.junit.jupiter.api.Assertions; - -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.UUID; - -public class PaimonMetadataOpsTest { - public static String warehouse; - public static PaimonExternalCatalog paimonCatalog; - public static PaimonMetadataOps ops; - public static String dbName = "test_db"; - public static ConnectContext connectContext; - - @BeforeClass - public static void beforeClass() throws Throwable { - Path warehousePath = Files.createTempDirectory("test_warehouse_"); - warehouse = "file://" + warehousePath.toAbsolutePath() + "/"; - HashMap param = new HashMap<>(); - param.put("type", "paimon"); - param.put("paimon.catalog.type", "filesystem"); - param.put("warehouse", warehouse); - // Construct the legacy filesystem catalog directly. NOTE: "paimon" is in - // CatalogFactory.SPI_READY_TYPES, so CatalogFactory.createFromCommand now routes paimon through - // the connector-plugin SPI and returns a PluginDrivenExternalCatalog — it throws "No connector - // plugin loaded" when the plugin is not installed in connector_plugin_root (the case in a plain - // fe-core UT) and, even when loaded, is not castable to the legacy PaimonExternalCatalog. This - // test exercises the still-live legacy PaimonMetadataOps, so build the legacy catalog directly - // here rather than through the SPI-routed factory. - paimonCatalog = new PaimonFileExternalCatalog(1, "paimon", null, param, "comment"); - paimonCatalog.makeSureInitialized(); - // create db - ops = new PaimonMetadataOps(paimonCatalog, paimonCatalog.catalog); - ops.createDb(dbName, true, Maps.newHashMap()); - paimonCatalog.makeSureInitialized(); - - // context - connectContext = new ConnectContext(); - connectContext.setThreadLocalInfo(); - } - - @Test - public void testSimpleTable() throws Exception { - String tableName = getTableName(); - Identifier identifier = new Identifier(dbName, tableName); - String sql = "create table " + dbName + "." + tableName + " (id int) engine = paimon"; - createTable(sql); - Catalog catalog = ops.getCatalog(); - Table table = catalog.getTable(identifier); - List columnNames = new ArrayList<>(); - if (catalog instanceof HiveCatalog) { - columnNames.addAll(((HiveCatalog) catalog).loadTableSchema(identifier).fieldNames()); - } else if (catalog instanceof FileSystemCatalog) { - columnNames.addAll(((FileSystemCatalog) catalog).loadTableSchema(identifier).fieldNames()); - } - - if (!columnNames.isEmpty()) { - Assert.assertEquals(1, columnNames.size()); - } - Assert.assertEquals(0, table.partitionKeys().size()); - } - - @Test - public void testProperties() throws Exception { - String tableName = getTableName(); - Identifier identifier = new Identifier(dbName, tableName); - String sql = "create table " + dbName + "." + tableName + " (id int) engine = paimon properties(\"primary-key\"=id)"; - createTable(sql); - Catalog catalog = ops.getCatalog(); - Table table = catalog.getTable(identifier); - - List columnNames = new ArrayList<>(); - if (catalog instanceof HiveCatalog) { - columnNames.addAll(((HiveCatalog) catalog).loadTableSchema(identifier).fieldNames()); - } else if (catalog instanceof FileSystemCatalog) { - columnNames.addAll(((FileSystemCatalog) catalog).loadTableSchema(identifier).fieldNames()); - } - - if (!columnNames.isEmpty()) { - Assert.assertEquals(1, columnNames.size()); - } - Assert.assertEquals(0, table.partitionKeys().size()); - Assert.assertTrue(table.primaryKeys().contains("id")); - Assert.assertEquals(1, table.primaryKeys().size()); - } - - @Test - public void testType() throws Exception { - String tableName = getTableName(); - Identifier identifier = new Identifier(dbName, tableName); - String sql = "create table " + dbName + "." + tableName + " (" - + "c0 int, " - + "c1 bigint, " - + "c2 float, " - + "c3 double, " - + "c4 string, " - + "c5 date, " - + "c6 decimal(20, 10), " - + "c7 datetime" - + ") engine = paimon " - + "properties(\"primary-key\"=c0)"; - createTable(sql); - Catalog catalog = ops.getCatalog(); - Table table = catalog.getTable(identifier); - - List columns = new ArrayList<>(); - if (catalog instanceof HiveCatalog) { - columns.addAll(((HiveCatalog) catalog).loadTableSchema(identifier).fields()); - } else if (catalog instanceof FileSystemCatalog) { - columns.addAll(((FileSystemCatalog) catalog).loadTableSchema(identifier).fields()); - } - - if (!columns.isEmpty()) { - Assert.assertEquals(8, columns.size()); - Assert.assertEquals(new IntType().asSQLString(), columns.get(0).type().toString()); - Assert.assertEquals(new BigIntType().asSQLString(), columns.get(1).type().toString()); - Assert.assertEquals(new FloatType().asSQLString(), columns.get(2).type().toString()); - Assert.assertEquals(new DoubleType().asSQLString(), columns.get(3).type().toString()); - Assert.assertEquals(new VarCharType(VarCharType.MAX_LENGTH).asSQLString(), columns.get(4).type().toString()); - Assert.assertEquals(new DateType().asSQLString(), columns.get(5).type().toString()); - Assert.assertEquals(new DecimalType(20, 10).asSQLString(), columns.get(6).type().toString()); - Assert.assertEquals(new TimestampType().asSQLString(), columns.get(7).type().toString()); - } - - Assert.assertEquals(0, table.partitionKeys().size()); - Assert.assertTrue(table.primaryKeys().contains("c0")); - Assert.assertEquals(1, table.primaryKeys().size()); - } - - @Test - public void testPartition() throws Exception { - String tableName = "test04"; - Identifier identifier = new Identifier(dbName, tableName); - String sql = "create table " + dbName + "." + tableName + " (" - + "c0 int, " - + "c1 bigint, " - + "c2 float, " - + "c3 double, " - + "c4 string, " - + "c5 date, " - + "c6 decimal(20, 10), " - + "c7 datetime" - + ") engine = paimon " - + "partition by (" - + "c1 ) ()" - + "properties(\"primary-key\"=c0)"; - createTable(sql); - Catalog catalog = ops.getCatalog(); - Table table = catalog.getTable(identifier); - Assert.assertEquals(1, table.partitionKeys().size()); - Assert.assertTrue(table.primaryKeys().contains("c0")); - Assert.assertEquals(1, table.primaryKeys().size()); - } - - @Test - public void testBucket() throws Exception { - String tableName = getTableName(); - Identifier identifier = new Identifier(dbName, tableName); - String sql = "create table " + dbName + "." + tableName + " (" - + "c0 int, " - + "c1 bigint, " - + "c2 float, " - + "c3 double, " - + "c4 string, " - + "c5 date, " - + "c6 decimal(20, 10), " - + "c7 datetime" - + ") engine = paimon " - + "properties(\"primary-key\"=c0," - + "\"bucket\" = 4," - + "\"bucket-key\" = c0)"; - createTable(sql); - Catalog catalog = ops.getCatalog(); - Table table = catalog.getTable(identifier); - Assert.assertEquals("4", table.options().get("bucket")); - Assert.assertEquals("c0", table.options().get("bucket-key")); - } - - public void createTable(String sql) throws UserException { - LogicalPlan plan = new NereidsParser().parseSingle(sql); - Assertions.assertTrue(plan instanceof CreateTableCommand); - CreateTableInfo createTableInfo = ((CreateTableCommand) plan).getCreateTableInfo(); - createTableInfo.setIsExternal(true); - createTableInfo.analyzeEngine(); - ops.createTable(createTableInfo); - } - - public String getTableName() { - String s = "test_tb_" + UUID.randomUUID(); - return s.replaceAll("-", ""); - } - - @Test - public void testDropDB() { - try { - // create db success - ops.createDb("t_paimon", false, Maps.newHashMap()); - // drop db success - ops.dropDb("t_paimon", false, false); - } catch (Throwable t) { - Assert.fail(); - } - - try { - ops.dropDb("t_paimon", false, false); - Assert.fail(); - } catch (Throwable t) { - Assert.assertTrue(t instanceof DdlException); - Assert.assertTrue(t.getMessage().contains("database doesn't exist")); - } - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java deleted file mode 100644 index 4a2b609023bb93..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java +++ /dev/null @@ -1,146 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.catalog.Type; -import org.apache.doris.thrift.TPrimitiveType; -import org.apache.doris.thrift.schema.external.TFieldPtr; -import org.apache.doris.thrift.schema.external.TSchema; - -import org.apache.paimon.data.BinaryRow; -import org.apache.paimon.data.BinaryRowWriter; -import org.apache.paimon.data.BinaryString; -import org.apache.paimon.schema.TableSchema; -import org.apache.paimon.table.Table; -import org.apache.paimon.types.CharType; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.DataTypes; -import org.apache.paimon.types.VarCharType; -import org.junit.Assert; -import org.junit.Test; -import org.mockito.Mockito; - -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.Map; - -public class PaimonUtilTest { - private static final String TABLE_READ_SEQUENCE_NUMBER_ENABLED = "table-read.sequence-number.enabled"; - - @Test - public void testSchemaForVarcharAndChar() { - DataField c1 = new DataField(1, "c1", new VarCharType(32)); - DataField c2 = new DataField(2, "c2", new CharType(14)); - Type type1 = PaimonUtil.paimonTypeToDorisType(c1.type(), true, true); - Type type2 = PaimonUtil.paimonTypeToDorisType(c2.type(), true, true); - Assert.assertTrue(type1.isVarchar()); - Assert.assertEquals(32, type1.getLength()); - Assert.assertEquals(14, type2.getLength()); - } - - @Test - public void testGetPartitionInfoMapSupportsFloatingPointPartitions() { - DataField floatPartition = DataTypes.FIELD(0, "float_partition", DataTypes.FLOAT()); - DataField doublePartition = DataTypes.FIELD(1, "double_partition", DataTypes.DOUBLE()); - Table table = Mockito.mock(Table.class); - Mockito.when(table.name()).thenReturn("mock_table"); - Mockito.when(table.partitionKeys()).thenReturn(Arrays.asList("float_partition", "double_partition")); - Mockito.when(table.rowType()).thenReturn(DataTypes.ROW(floatPartition, doublePartition)); - - float floatValue = Math.nextUp(0.1F); - double doubleValue = Math.nextUp(0.1D); - BinaryRow partitionValues = new BinaryRow(2); - BinaryRowWriter writer = new BinaryRowWriter(partitionValues); - writer.writeFloat(0, floatValue); - writer.writeDouble(1, doubleValue); - writer.complete(); - - Map partitionInfoMap = PaimonUtil.getPartitionInfoMap(table, partitionValues, "UTC"); - - String serializedFloat = partitionInfoMap.get("float_partition"); - String serializedDouble = partitionInfoMap.get("double_partition"); - Assert.assertEquals(Float.toString(floatValue), serializedFloat); - Assert.assertEquals(Double.toString(doubleValue), serializedDouble); - Assert.assertEquals(Float.floatToIntBits(floatValue), - Float.floatToIntBits(Float.parseFloat(serializedFloat))); - Assert.assertEquals(Double.doubleToLongBits(doubleValue), - Double.doubleToLongBits(Double.parseDouble(serializedDouble))); - } - - @Test - public void testGetPartitionInfoMapUsesLowerCaseKeys() { - DataField mixedCasePartition = DataTypes.FIELD(0, "Dt", DataTypes.STRING()); - Table table = Mockito.mock(Table.class); - Mockito.when(table.name()).thenReturn("mock_table"); - Mockito.when(table.partitionKeys()).thenReturn(Collections.singletonList("Dt")); - Mockito.when(table.rowType()).thenReturn(DataTypes.ROW(mixedCasePartition)); - - BinaryRow partitionValues = BinaryRow.singleColumn(BinaryString.fromString("2026-05-26")); - - Map partitionInfoMap = PaimonUtil.getPartitionInfoMap(table, partitionValues, "UTC"); - - Assert.assertFalse(partitionInfoMap.containsKey("Dt")); - Assert.assertEquals("2026-05-26", partitionInfoMap.get("dt")); - } - - @Test - public void testBinlogHistorySchemaWithSequenceNumber() { - PaimonSysExternalTable binlogTable = Mockito.mock(PaimonSysExternalTable.class); - Mockito.when(binlogTable.getSysTableType()).thenReturn("binlog"); - Mockito.when(binlogTable.getTableProperties()).thenReturn( - Collections.singletonMap(TABLE_READ_SEQUENCE_NUMBER_ENABLED, "true")); - Mockito.when(binlogTable.getName()).thenReturn("mock_binlog"); - - List sourceFields = Arrays.asList( - new DataField(0, "id", DataTypes.INT()), - new DataField(1, "name", DataTypes.STRING())); - TableSchema sourceSchema = new TableSchema(1L, sourceFields, 1, Collections.emptyList(), - Collections.emptyList(), Collections.emptyMap(), ""); - TSchema historySchema = PaimonUtil.getHistorySchemaInfo(binlogTable, sourceSchema, true, true); - List fields = historySchema.getRootField().getFields(); - - Assert.assertEquals("rowkind", fields.get(0).getFieldPtr().getName()); - Assert.assertEquals("_SEQUENCE_NUMBER", fields.get(1).getFieldPtr().getName()); - Assert.assertEquals("id", fields.get(2).getFieldPtr().getName()); - Assert.assertEquals(TPrimitiveType.ARRAY, fields.get(2).getFieldPtr().getType().getType()); - Assert.assertEquals("name", fields.get(3).getFieldPtr().getName()); - Assert.assertEquals(TPrimitiveType.ARRAY, fields.get(3).getFieldPtr().getType().getType()); - } - - @Test - public void testAuditLogHistorySchemaWithoutSequenceNumber() { - PaimonSysExternalTable auditLogTable = Mockito.mock(PaimonSysExternalTable.class); - Mockito.when(auditLogTable.getSysTableType()).thenReturn("audit_log"); - Mockito.when(auditLogTable.getTableProperties()).thenReturn(Collections.emptyMap()); - Mockito.when(auditLogTable.getName()).thenReturn("mock_audit_log"); - - List sourceFields = Arrays.asList( - new DataField(0, "id", DataTypes.INT()), - new DataField(1, "name", DataTypes.STRING())); - TableSchema sourceSchema = new TableSchema(1L, sourceFields, 1, Collections.emptyList(), - Collections.emptyList(), Collections.emptyMap(), ""); - TSchema historySchema = PaimonUtil.getHistorySchemaInfo(auditLogTable, sourceSchema, true, true); - List fields = historySchema.getRootField().getFields(); - - Assert.assertEquals(3, fields.size()); - Assert.assertEquals("rowkind", fields.get(0).getFieldPtr().getName()); - Assert.assertEquals("id", fields.get(1).getFieldPtr().getName()); - Assert.assertEquals("name", fields.get(2).getFieldPtr().getName()); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java deleted file mode 100644 index c5ddc82a3f63df..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java +++ /dev/null @@ -1,658 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon.source; - -import org.apache.doris.analysis.TupleDescriptor; -import org.apache.doris.analysis.TupleId; -import org.apache.doris.common.ExceptionChecker; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.CatalogProperty; -import org.apache.doris.datasource.FileQueryScanNode; -import org.apache.doris.datasource.FileSplitter; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; -import org.apache.doris.datasource.paimon.PaimonExternalTable; -import org.apache.doris.datasource.paimon.PaimonFileExternalCatalog; -import org.apache.doris.datasource.paimon.PaimonSysExternalTable; -import org.apache.doris.datasource.property.metastore.MetastoreProperties; -import org.apache.doris.datasource.property.metastore.PaimonJdbcMetaStoreProperties; -import org.apache.doris.planner.PlanNodeId; -import org.apache.doris.planner.ScanContext; -import org.apache.doris.qe.SessionVariable; -import org.apache.doris.thrift.TFileRangeDesc; -import org.apache.doris.thrift.TFileScanRangeParams; - -import org.apache.paimon.data.BinaryRow; -import org.apache.paimon.io.DataFileMeta; -import org.apache.paimon.manifest.FileSource; -import org.apache.paimon.stats.SimpleStats; -import org.apache.paimon.table.Table; -import org.apache.paimon.table.source.DataSplit; -import org.apache.paimon.table.source.RawFile; -import org.junit.Assert; -import org.junit.Test; -import org.junit.runner.RunWith; -import org.mockito.ArgumentMatchers; -import org.mockito.Mock; -import org.mockito.Mockito; -import org.mockito.junit.MockitoJUnitRunner; - -import java.lang.reflect.Method; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Optional; - -@RunWith(MockitoJUnitRunner.class) -public class PaimonScanNodeTest { - @Mock - private SessionVariable sv; - - @Mock - private PaimonFileExternalCatalog paimonFileExternalCatalog; - - @Test - public void testSplitWeight() throws UserException { - - PaimonScanNode paimonScanNode = newTestNode(new PlanNodeId(1), new TupleId(3), sv); - - paimonScanNode.setSource(mockPaimonSourceWithPartitionKeys(Collections.emptyList())); - - DataFileMeta dfm1 = DataFileMeta.forAppend("f1.parquet", 64L * 1024 * 1024, 1L, SimpleStats.EMPTY_STATS, - 1L, 1L, 1L, Collections.emptyList(), null, FileSource.APPEND, - Collections.emptyList(), null, null, Collections.emptyList()); - BinaryRow binaryRow1 = BinaryRow.singleColumn(1); - DataSplit ds1 = DataSplit.builder() - .rawConvertible(true) - .withPartition(binaryRow1) - .withBucket(1) - .withBucketPath("file://b1") - .withDataFiles(Collections.singletonList(dfm1)) - .build(); - - DataFileMeta dfm2 = DataFileMeta.forAppend("f2.parquet", 32L * 1024 * 1024, 2L, SimpleStats.EMPTY_STATS, - 1L, 1L, 1L, Collections.emptyList(), null, FileSource.APPEND, - Collections.emptyList(), null, null, Collections.emptyList()); - BinaryRow binaryRow2 = BinaryRow.singleColumn(1); - DataSplit ds2 = DataSplit.builder() - .rawConvertible(true) - .withPartition(binaryRow2) - .withBucket(1) - .withBucketPath("file://b1") - .withDataFiles(Collections.singletonList(dfm2)) - .build(); - - - // Mock PaimonScanNode to return test data splits - PaimonScanNode spyPaimonScanNode = Mockito.spy(paimonScanNode); - Mockito.doReturn(new ArrayList() { - { - add(ds1); - add(ds2); - } - }).when(spyPaimonScanNode).getPaimonSplitFromAPI(); - - long maxInitialSplitSize = 32L * 1024L * 1024L; - long maxSplitSize = 64L * 1024L * 1024L; - // Ensure fileSplitter is initialized on the spy as doInitialize() is not called in this unit test - FileSplitter fileSplitter = new FileSplitter(maxInitialSplitSize, maxSplitSize, - 0); - try { - java.lang.reflect.Field field = FileQueryScanNode.class.getDeclaredField("fileSplitter"); - field.setAccessible(true); - field.set(spyPaimonScanNode, fileSplitter); - - java.lang.reflect.Field storagePropertiesField = - PaimonScanNode.class.getDeclaredField("storagePropertiesMap"); - storagePropertiesField.setAccessible(true); - storagePropertiesField.set(spyPaimonScanNode, Collections.emptyMap()); - } catch (NoSuchFieldException | IllegalAccessException e) { - throw new RuntimeException("Failed to inject test fields into PaimonScanNode", e); - } - - // Note: The original PaimonSource is sufficient for this test - // No need to mock catalog properties since doInitialize() is not called in this test - // Mock SessionVariable behavior - Mockito.when(sv.isForceJniScanner()).thenReturn(false); - Mockito.when(sv.getIgnoreSplitType()).thenReturn("NONE"); - Mockito.when(sv.getMaxInitialSplitSize()).thenReturn(maxInitialSplitSize); - Mockito.when(sv.getMaxSplitSize()).thenReturn(maxSplitSize); - - // native - mockNativeReader(spyPaimonScanNode); - List s1 = spyPaimonScanNode.getSplits(1); - PaimonSplit s11 = (PaimonSplit) s1.get(0); - PaimonSplit s12 = (PaimonSplit) s1.get(1); - Assert.assertEquals(2, s1.size()); - Assert.assertEquals(100, s11.getSplitWeight().getRawValue()); - Assert.assertNull(s11.getSplit()); - Assert.assertEquals(50, s12.getSplitWeight().getRawValue()); - Assert.assertNull(s12.getSplit()); - - // jni - mockJniReader(spyPaimonScanNode); - List s2 = spyPaimonScanNode.getSplits(1); - PaimonSplit s21 = (PaimonSplit) s2.get(0); - PaimonSplit s22 = (PaimonSplit) s2.get(1); - Assert.assertEquals(2, s2.size()); - Assert.assertNotNull(s21.getSplit()); - Assert.assertNotNull(s22.getSplit()); - Assert.assertEquals(100, s21.getSplitWeight().getRawValue()); - Assert.assertEquals(50, s22.getSplitWeight().getRawValue()); - } - - @Test - public void testValidateIncrementalReadParams() throws UserException { - // Test valid parameter combinations - - // 1. Only startSnapshotId - Map params1 = new HashMap<>(); - params1.put("startSnapshotId", "5"); - ExceptionChecker.expectThrowsWithMsg(UserException.class, - "endSnapshotId is required when using snapshot-based incremental read", - () -> PaimonScanNode.validateIncrementalReadParams(params1)); - - // 2. Both startSnapshotId and endSnapshotId - Map params = new HashMap<>(); - params.put("startSnapshotId", "1"); - params.put("endSnapshotId", "5"); - Map result = PaimonScanNode.validateIncrementalReadParams(params); - Assert.assertEquals("1,5", result.get("incremental-between")); - Assert.assertTrue(result.containsKey("scan.mode") && result.get("scan.mode") == null); - Assert.assertEquals(3, result.size()); - - // 3. startSnapshotId + endSnapshotId + incrementalBetweenScanMode - params.clear(); - params.put("startSnapshotId", "2"); - params.put("endSnapshotId", "8"); - params.put("incrementalBetweenScanMode", "diff"); - result = PaimonScanNode.validateIncrementalReadParams(params); - Assert.assertEquals("2,8", result.get("incremental-between")); - Assert.assertEquals("diff", result.get("incremental-between-scan-mode")); - Assert.assertTrue(result.containsKey("scan.mode") && result.get("scan.mode") == null); - Assert.assertEquals(4, result.size()); - - // 4. Only startTimestamp - params.clear(); - params.put("startTimestamp", "1000"); - result = PaimonScanNode.validateIncrementalReadParams(params); - Assert.assertEquals("1000," + Long.MAX_VALUE, result.get("incremental-between-timestamp")); - Assert.assertTrue(result.containsKey("scan.mode") && result.get("scan.mode") == null); - Assert.assertTrue(result.containsKey("scan.snapshot-id") && result.get("scan.snapshot-id") == null); - Assert.assertEquals(3, result.size()); - - // 5. Both startTimestamp and endTimestamp - params.clear(); - params.put("startTimestamp", "1000"); - params.put("endTimestamp", "2000"); - result = PaimonScanNode.validateIncrementalReadParams(params); - Assert.assertEquals("1000,2000", result.get("incremental-between-timestamp")); - Assert.assertTrue(result.containsKey("scan.mode") && result.get("scan.mode") == null); - Assert.assertTrue(result.containsKey("scan.snapshot-id") && result.get("scan.snapshot-id") == null); - Assert.assertEquals(3, result.size()); - - // Test invalid parameter combinations - - // 6. Test mutual exclusivity - both snapshot and timestamp params - params.clear(); - params.put("startSnapshotId", "1"); - params.put("startTimestamp", "1000"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for mutual exclusivity"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("Cannot specify both snapshot-based parameters")); - } - - // 7. Test snapshot params without required startSnapshotId - params.clear(); - params.put("endSnapshotId", "5"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when startSnapshotId is missing"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("startSnapshotId is required")); - } - - // 8. Test timestamp params without required startTimestamp - params.clear(); - params.put("endTimestamp", "2000"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when startTimestamp is missing"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("startTimestamp is required")); - } - - // 9. Test incrementalBetweenScanMode without endSnapshotId - params.clear(); - params.put("startSnapshotId", "1"); - params.put("incrementalBetweenScanMode", "auto"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when incrementalBetweenScanMode appears without endSnapshotId"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("incrementalBetweenScanMode can only be specified when both")); - } - - // 10. Test incrementalBetweenScanMode alone - params.clear(); - params.put("incrementalBetweenScanMode", "auto"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when incrementalBetweenScanMode appears alone"); - } catch (UserException e) { - Assert.assertTrue( - e.getMessage().contains("startSnapshotId is required when using snapshot-based incremental read")); - } - - // 11. Test invalid snapshot ID values < 0) - params.clear(); - params.put("startSnapshotId", "-1"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for startSnapshotId < 0"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("startSnapshotId must be greater than or equal to 0")); - } - - params.clear(); - params.put("startSnapshotId", "1"); - params.put("endSnapshotId", "-1"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for endSnapshotId < 0"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("endSnapshotId must be greater than or equal to 0")); - } - - // 12. Test start > end for snapshot IDs - params.clear(); - params.put("startSnapshotId", "6"); - params.put("endSnapshotId", "5"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when startSnapshotId > endSnapshotId"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("startSnapshotId must be less than or equal to endSnapshotId")); - } - - // 12.1. Test startSnapshotId == endSnapshotId (should be allowed, consistent with Spark Paimon behavior) - params.clear(); - params.put("startSnapshotId", "5"); - params.put("endSnapshotId", "5"); - result = PaimonScanNode.validateIncrementalReadParams(params); - Assert.assertEquals("5,5", result.get("incremental-between")); - Assert.assertTrue(result.containsKey("scan.mode") && result.get("scan.mode") == null); - Assert.assertEquals(3, result.size()); - - // 13. Test invalid timestamp values (< 0) - params.clear(); - params.put("startTimestamp", "-1"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for startTimestamp < 0"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("startTimestamp must be greater than or equal to 0")); - } - - params.clear(); - params.put("startTimestamp", "1000"); - params.put("endTimestamp", "0"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for endTimestamp ≤ 0"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("endTimestamp must be greater than 0")); - } - - // 14. Test start ≥ end for timestamps - params.clear(); - params.put("startTimestamp", "2000"); - params.put("endTimestamp", "2000"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when startTimestamp = endTimestamp"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("startTimestamp must be less than endTimestamp")); - } - - params.clear(); - params.put("startTimestamp", "3000"); - params.put("endTimestamp", "2000"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when startTimestamp > endTimestamp"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("startTimestamp must be less than endTimestamp")); - } - - // 15. Test invalid number format - params.clear(); - params.put("startSnapshotId", "invalid"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for invalid number format"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("Invalid startSnapshotId format")); - } - - params.clear(); - params.put("startTimestamp", "invalid"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for invalid timestamp format"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("Invalid startTimestamp format")); - } - - // 16. Test invalid incrementalBetweenScanMode values - params.clear(); - params.put("startSnapshotId", "1"); - params.put("endSnapshotId", "5"); - params.put("incrementalBetweenScanMode", "invalid"); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception for invalid scan mode"); - } catch (UserException e) { - Assert.assertTrue( - e.getMessage().contains("incrementalBetweenScanMode must be one of: auto, diff, delta, changelog")); - } - - // 17. Test valid incrementalBetweenScanMode values (case insensitive) - String[] validModes = {"auto", "AUTO", "diff", "DIFF", "delta", "DELTA", "changelog", "CHANGELOG"}; - for (String mode : validModes) { - params.clear(); - params.put("startSnapshotId", "1"); - params.put("endSnapshotId", "5"); - params.put("incrementalBetweenScanMode", mode); - result = PaimonScanNode.validateIncrementalReadParams(params); - Assert.assertEquals("1,5", result.get("incremental-between")); - Assert.assertEquals(mode, result.get("incremental-between-scan-mode")); - Assert.assertTrue(result.containsKey("scan.mode") && result.get("scan.mode") == null); - Assert.assertTrue(result.containsKey("scan.mode") && result.get("scan.mode") == null); - Assert.assertEquals(4, result.size()); - } - - // 18. Test no parameters at all - params.clear(); - try { - PaimonScanNode.validateIncrementalReadParams(params); - Assert.fail("Should throw exception when no parameters provided"); - } catch (UserException e) { - Assert.assertTrue(e.getMessage().contains("at least one valid parameter group must be specified")); - } - } - - @Test - public void testPaimonDataSystemTableForceJniEvenWhenNativeSupported() throws UserException { - PaimonScanNode paimonScanNode = newTestNode(new PlanNodeId(1), new TupleId(3), sv); - PaimonScanNode spyPaimonScanNode = Mockito.spy(paimonScanNode); - - DataFileMeta dfm = DataFileMeta.forAppend("f1.parquet", 64L * 1024 * 1024, 1L, SimpleStats.EMPTY_STATS, - 1L, 1L, 1L, Collections.emptyList(), null, FileSource.APPEND, - Collections.emptyList(), null, null, Collections.emptyList()); - BinaryRow binaryRow = BinaryRow.singleColumn(1); - DataSplit dataSplit = DataSplit.builder() - .rawConvertible(true) - .withPartition(binaryRow) - .withBucket(1) - .withBucketPath("file://b1") - .withDataFiles(Collections.singletonList(dfm)) - .build(); - - PaimonSource source = Mockito.mock(PaimonSource.class); - PaimonSysExternalTable binlogTable = Mockito.mock(PaimonSysExternalTable.class); - Mockito.when(binlogTable.getSysTableType()).thenReturn("binlog"); - Mockito.when(source.getExternalTable()).thenReturn(binlogTable); - spyPaimonScanNode.setSource(source); - - Mockito.doReturn(Collections.singletonList(dataSplit)).when(spyPaimonScanNode).getPaimonSplitFromAPI(); - Assert.assertTrue(spyPaimonScanNode.supportNativeReader(dataSplit.convertToRawFiles())); - - long maxInitialSplitSize = 32L * 1024L * 1024L; - long maxSplitSize = 64L * 1024L * 1024L; - FileSplitter fileSplitter = new FileSplitter(maxInitialSplitSize, maxSplitSize, 0); - try { - java.lang.reflect.Field field = FileQueryScanNode.class.getDeclaredField("fileSplitter"); - field.setAccessible(true); - field.set(spyPaimonScanNode, fileSplitter); - } catch (NoSuchFieldException | IllegalAccessException e) { - throw new RuntimeException("Failed to inject FileSplitter into PaimonScanNode test", e); - } - - Mockito.when(sv.isForceJniScanner()).thenReturn(false); - Mockito.when(sv.getIgnoreSplitType()).thenReturn("NONE"); - Mockito.when(sv.getMaxSplitSize()).thenReturn(maxSplitSize); - - Assert.assertTrue(spyPaimonScanNode.shouldForceJniForSystemTable()); - List splits = spyPaimonScanNode.getSplits(1); - Assert.assertEquals(1, splits.size()); - Assert.assertNotNull(((PaimonSplit) splits.get(0)).getSplit()); - - PaimonSysExternalTable auditLogTable = Mockito.mock(PaimonSysExternalTable.class); - Mockito.when(auditLogTable.getSysTableType()).thenReturn("audit_log"); - Mockito.when(source.getExternalTable()).thenReturn(auditLogTable); - - Assert.assertTrue(spyPaimonScanNode.shouldForceJniForSystemTable()); - List auditLogSplits = spyPaimonScanNode.getSplits(1); - Assert.assertEquals(1, auditLogSplits.size()); - Assert.assertNotNull(((PaimonSplit) auditLogSplits.get(0)).getSplit()); - } - - @Test - public void testDetermineTargetFileSplitSizeHonorsMaxFileSplitNum() throws Exception { - SessionVariable sv = new SessionVariable(); - sv.setMaxFileSplitNum(100); - PaimonScanNode node = newTestNode(new PlanNodeId(0), new TupleId(0), sv); - - PaimonSource source = Mockito.mock(PaimonSource.class); - Mockito.when(source.getFileFormatFromTableProperties()).thenReturn("parquet"); - node.setSource(source); - - RawFile rawFile = Mockito.mock(RawFile.class); - Mockito.when(rawFile.path()).thenReturn("file.parquet"); - Mockito.when(rawFile.fileSize()).thenReturn(10_000L * 1024L * 1024L); - - DataSplit dataSplit = Mockito.mock(DataSplit.class); - Mockito.when(dataSplit.convertToRawFiles()).thenReturn(Optional.of(Collections.singletonList(rawFile))); - - Method method = PaimonScanNode.class.getDeclaredMethod("determineTargetFileSplitSize", List.class, boolean.class); - method.setAccessible(true); - long target = (long) method.invoke(node, Collections.singletonList(dataSplit), false); - Assert.assertEquals(100L * 1024L * 1024L, target); - } - - @Test - public void testGetBackendPaimonOptionsForJdbcCatalog() throws Exception { - String driverUrl = "file:///tmp/postgresql-42.5.0.jar"; - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:postgresql://127.0.0.1:5442/postgres"); - props.put("warehouse", "s3://warehouse/path"); - props.put("paimon.jdbc.driver_url", driverUrl); - props.put("paimon.jdbc.driver_class", "org.postgresql.Driver"); - PaimonJdbcMetaStoreProperties jdbcMetaStoreProperties = - (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - - CatalogProperty catalogProperty = Mockito.mock(CatalogProperty.class); - Mockito.when(catalogProperty.getMetastoreProperties()).thenReturn(jdbcMetaStoreProperties); - - PaimonExternalCatalog catalog = Mockito.mock(PaimonExternalCatalog.class); - Mockito.when(catalog.getCatalogProperty()).thenReturn(catalogProperty); - - PaimonSource source = Mockito.mock(PaimonSource.class); - Mockito.when(source.getCatalog()).thenReturn(catalog); - - PaimonScanNode node = newTestNode(new PlanNodeId(0), new TupleId(0), sv); - node.setSource(source); - - Map backendOptions = node.getBackendPaimonOptions(); - Assert.assertEquals("org.postgresql.Driver", backendOptions.get("jdbc.driver_class")); - Assert.assertEquals(driverUrl, backendOptions.get("jdbc.driver_url")); - Assert.assertEquals(2, backendOptions.size()); - } - - @Test - public void testApplyBackendPaimonOptionsAtScanNodeLevel() throws Exception { - PaimonScanNode node = newTestNode(new PlanNodeId(0), new TupleId(0), sv); - PaimonSource source = Mockito.mock(PaimonSource.class); - Mockito.when(source.getTableLocation()).thenReturn("file:///warehouse"); - Table paimonTable = mockPaimonTableWithPartitionKeys(Collections.emptyList()); - Mockito.when(source.getPaimonTable()).thenReturn(paimonTable); - node.setSource(source); - - Map backendOptions = new HashMap<>(); - backendOptions.put("jdbc.driver_url", "file:///tmp/postgresql-42.5.0.jar"); - backendOptions.put("jdbc.driver_class", "org.postgresql.Driver"); - setField(FileQueryScanNode.class, node, "params", new TFileScanRangeParams()); - setField(PaimonScanNode.class, node, "backendPaimonOptions", backendOptions); - setField(PaimonScanNode.class, node, "storagePropertiesMap", Collections.emptyMap()); - - invokePrivateMethod(node, "setScanLevelPaimonOptions"); - - Assert.assertEquals(backendOptions, node.getFileScanRangeParams().getPaimonOptions()); - - TFileRangeDesc rangeDesc = new TFileRangeDesc(); - invokePrivateMethod(node, "setPaimonParams", - new Class[] {TFileRangeDesc.class, PaimonSplit.class}, - rangeDesc, new PaimonSplit(createDataSplit("scan_level.parquet"))); - Assert.assertFalse(rangeDesc.getTableFormatParams().getPaimonParams().isSetPaimonOptions()); - } - - @Test - public void testGetPathPartitionKeysReturnsTablePartitionKeys() throws Exception { - PaimonScanNode node = newTestNode(new PlanNodeId(0), new TupleId(0), sv); - PaimonSource source = Mockito.mock(PaimonSource.class); - Table table = Mockito.mock(Table.class); - PaimonSysExternalTable sysTable = Mockito.mock(PaimonSysExternalTable.class); - Mockito.when(source.getPaimonTable()).thenReturn(table); - Mockito.when(source.getExternalTable()).thenReturn(sysTable); - Mockito.when(table.partitionKeys()).thenReturn(Arrays.asList("Dt", "Region")); - Mockito.when(sysTable.isDataTable()).thenReturn(true); - node.setSource(source); - - Assert.assertEquals(Arrays.asList("dt", "region"), node.getPathPartitionKeys()); - } - - @Test - public void testGetPathPartitionKeysReturnsEmptyForMetadataSystemTable() throws Exception { - PaimonScanNode node = newTestNode(new PlanNodeId(0), new TupleId(0), sv); - PaimonSource source = Mockito.mock(PaimonSource.class); - PaimonSysExternalTable sysTable = Mockito.mock(PaimonSysExternalTable.class); - Mockito.when(source.getExternalTable()).thenReturn(sysTable); - Mockito.when(sysTable.isDataTable()).thenReturn(false); - node.setSource(source); - - Assert.assertEquals(Collections.emptyList(), node.getPathPartitionKeys()); - } - - @Test - public void testSetPaimonParamsUsesOrderedPartitionKeys() throws Exception { - PaimonScanNode node = newTestNode(new PlanNodeId(0), new TupleId(0), sv); - PaimonSource source = Mockito.mock(PaimonSource.class); - Table table = Mockito.mock(Table.class); - PaimonSysExternalTable sysTable = Mockito.mock(PaimonSysExternalTable.class); - Mockito.when(source.getPaimonTable()).thenReturn(table); - Mockito.when(source.getTableLocation()).thenReturn("file:///warehouse"); - Mockito.when(source.getExternalTable()).thenReturn(sysTable); - Mockito.when(sysTable.isDataTable()).thenReturn(true); - Mockito.when(table.partitionKeys()).thenReturn(Arrays.asList("Pt", "Dt")); - node.setSource(source); - - TFileRangeDesc rangeDesc = new TFileRangeDesc(); - rangeDesc.setColumnsFromPathKeys(Collections.singletonList("stale")); - rangeDesc.setColumnsFromPath(Collections.singletonList("old")); - rangeDesc.setColumnsFromPathIsNull(Collections.singletonList(false)); - Map partitionValues = new HashMap<>(); - partitionValues.put("dt", "2025-01-01"); - partitionValues.put("pt", "p1"); - PaimonSplit split = new PaimonSplit(createDataSplit("ordered.parquet")); - split.setPaimonPartitionValues(partitionValues); - - invokePrivateMethod(node, "setPaimonParams", - new Class[] {TFileRangeDesc.class, PaimonSplit.class}, rangeDesc, split); - - Assert.assertEquals(Arrays.asList("pt", "dt"), rangeDesc.getColumnsFromPathKeys()); - Assert.assertEquals(Arrays.asList("p1", "2025-01-01"), rangeDesc.getColumnsFromPath()); - Assert.assertEquals(Arrays.asList(false, false), rangeDesc.getColumnsFromPathIsNull()); - } - - private void mockJniReader(PaimonScanNode spyNode) { - Mockito.doReturn(false).when(spyNode).supportNativeReader(ArgumentMatchers.any(Optional.class)); - } - - private void mockNativeReader(PaimonScanNode spyNode) { - Mockito.doReturn(true).when(spyNode).supportNativeReader(ArgumentMatchers.any(Optional.class)); - } - - private PaimonScanNode newTestNode(PlanNodeId planNodeId, TupleId tupleId, SessionVariable sessionVariable) { - TupleDescriptor desc = new TupleDescriptor(tupleId); - PaimonExternalTable externalTable = Mockito.mock(PaimonExternalTable.class); - Table paimonTable = mockPaimonTableWithPartitionKeys(Collections.emptyList()); - Mockito.when(externalTable.getPaimonTable(ArgumentMatchers.any())).thenReturn(paimonTable); - desc.setTable(externalTable); - return new PaimonScanNode(planNodeId, desc, false, sessionVariable, ScanContext.EMPTY); - } - - private PaimonSource mockPaimonSourceWithPartitionKeys(List partitionKeys) { - PaimonSource source = Mockito.mock(PaimonSource.class); - Table paimonTable = mockPaimonTableWithPartitionKeys(partitionKeys); - Mockito.when(source.getPaimonTable()).thenReturn(paimonTable); - return source; - } - - private Table mockPaimonTableWithPartitionKeys(List partitionKeys) { - Table paimonTable = Mockito.mock(Table.class); - Mockito.when(paimonTable.partitionKeys()).thenReturn(partitionKeys); - return paimonTable; - } - - private void setField(Class clazz, Object target, String fieldName, Object value) throws Exception { - java.lang.reflect.Field field = clazz.getDeclaredField(fieldName); - field.setAccessible(true); - field.set(target, value); - } - - private Object invokePrivateMethod(Object target, String methodName, Class[] parameterTypes, Object... args) - throws Exception { - Method method = target.getClass().getDeclaredMethod(methodName, parameterTypes); - method.setAccessible(true); - return method.invoke(target, args); - } - - private Object invokePrivateMethod(Object target, String methodName) throws Exception { - return invokePrivateMethod(target, methodName, new Class[0]); - } - - private DataSplit createDataSplit(String fileName) { - DataFileMeta dataFileMeta = DataFileMeta.forAppend(fileName, 64L * 1024 * 1024, 1L, SimpleStats.EMPTY_STATS, - 1L, 1L, 1L, Collections.emptyList(), null, FileSource.APPEND, - Collections.emptyList(), null, null, Collections.emptyList()); - return DataSplit.builder() - .rawConvertible(true) - .withPartition(BinaryRow.singleColumn(1)) - .withBucket(1) - .withBucketPath("file://b1") - .withDataFiles(Collections.singletonList(dataFileMeta)) - .build(); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java index 664a38f7334bf7..f7517259fcc5ac 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java @@ -19,7 +19,6 @@ import org.apache.doris.catalog.JdbcResource; import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.paimon.options.CatalogOptions; @@ -46,7 +45,7 @@ public void testBasicJdbcProperties() throws Exception { jdbcProps.initNormalizeAndCheckProps(); jdbcProps.buildCatalogOptions(); - Assertions.assertEquals(PaimonExternalCatalog.PAIMON_JDBC, jdbcProps.getPaimonCatalogType()); + Assertions.assertEquals("jdbc", jdbcProps.getPaimonCatalogType()); Assertions.assertEquals("jdbc", jdbcProps.getCatalogOptions().get(CatalogOptions.METASTORE.key())); Assertions.assertEquals("jdbc:mysql://localhost:3306/paimon", jdbcProps.getCatalogOptions().get(CatalogOptions.URI.key())); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java index cbfa6a01c80012..0fefe365dfa9de 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java @@ -17,8 +17,6 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.datasource.paimon.PaimonExternalCatalog; - import org.apache.paimon.options.Options; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -38,7 +36,7 @@ public void testBasicRestProperties() { PaimonRestMetaStoreProperties restProps = new PaimonRestMetaStoreProperties(props); restProps.initNormalizeAndCheckProps(); - Assertions.assertEquals(PaimonExternalCatalog.PAIMON_REST, restProps.getPaimonCatalogType()); + Assertions.assertEquals("rest", restProps.getPaimonCatalogType()); Assertions.assertEquals("rest", restProps.getMetastoreType()); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextTest.java index 8db96ac9a6d4e6..14d07118182c8c 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextTest.java @@ -22,11 +22,11 @@ import org.apache.doris.catalog.TableIf; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.PluginDrivenExternalTable; +import org.apache.doris.datasource.PluginDrivenMvccExternalTable; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.hive.HMSExternalTable.DLAType; import org.apache.doris.datasource.iceberg.IcebergExternalTable; import org.apache.doris.datasource.mvcc.MvccSnapshot; -import org.apache.doris.datasource.paimon.PaimonExternalTable; import org.apache.doris.nereids.rules.analysis.PreloadExternalMetadata; import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; import org.apache.doris.qe.ConnectContext; @@ -497,7 +497,7 @@ public void testSkipIcebergPreloadWhenOnlyNonLatestRelationExists() { public void testPreloadPaimonLatestSnapshotBeforeLock() { ConnectContext connectContext = Mockito.mock(ConnectContext.class); TableIf internalTable = Mockito.mock(TableIf.class); - PaimonExternalTable paimonExternalTable = Mockito.mock(PaimonExternalTable.class); + PluginDrivenMvccExternalTable paimonExternalTable = Mockito.mock(PluginDrivenMvccExternalTable.class); DatabaseIf database = mockDatabase(); CatalogIf catalog = mockCatalog(); MvccSnapshot mvccSnapshot = Mockito.mock(MvccSnapshot.class); diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PaimonPredicateConverterTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PaimonPredicateConverterTest.java deleted file mode 100644 index fde1b6f74c244c..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/planner/PaimonPredicateConverterTest.java +++ /dev/null @@ -1,99 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.planner; - -import org.apache.doris.analysis.Expr; -import org.apache.doris.common.FeConstants; -import org.apache.doris.datasource.paimon.source.PaimonPredicateConverter; -import org.apache.doris.qe.StmtExecutor; -import org.apache.doris.utframe.TestWithFeService; - -import com.google.common.collect.Lists; -import org.apache.paimon.predicate.CompoundPredicate; -import org.apache.paimon.predicate.LeafPredicate; -import org.apache.paimon.predicate.Or; -import org.apache.paimon.predicate.Predicate; -import org.apache.paimon.types.DataField; -import org.apache.paimon.types.IntType; -import org.apache.paimon.types.RowType; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -import java.util.List; - -public class PaimonPredicateConverterTest extends TestWithFeService { - @Override - protected void runBeforeAll() throws Exception { - FeConstants.runningUnitTest = true; - // Create database `db1`. - createDatabase("db1"); - - String tbl1 = "create table db1.tbl1(" + "k1 int," + " k2 int," + " v1 int)" + " distributed by hash(k1)" - + " properties('replication_num' = '1');"; - createTables(tbl1); - } - - @Test - public void equal() throws Exception { - DataField paimonFieldK1 = new DataField(0, "k1", new IntType()); - DataField paimonFieldK2 = new DataField(1, "k2", new IntType()); - DataField paimonFieldV1 = new DataField(2, "v1", new IntType()); - RowType rowType = new RowType(Lists.newArrayList(paimonFieldK1, paimonFieldK2, paimonFieldV1)); - PaimonPredicateConverter converter = new PaimonPredicateConverter(rowType); - connectContext.getSessionVariable().setParallelResultSink(false); - - // k1=1 - String sql1 = "SELECT * from db1.tbl1 where k1 = 1"; - StmtExecutor stmtExecutor = new StmtExecutor(connectContext, sql1); - stmtExecutor.execute(); - Planner planner = stmtExecutor.planner(); - List fragments = planner.getFragments(); - List conjuncts = fragments.get(0).getPlanRoot().getChild(0).conjuncts; - List predicates = converter.convertToPaimonExpr(conjuncts); - Assertions.assertEquals(predicates.size(), 1); - Assertions.assertTrue(predicates.get(0) instanceof LeafPredicate); - LeafPredicate leafPredicate = (LeafPredicate) predicates.get(0); - Assertions.assertEquals(leafPredicate.fieldName(), "k1"); - - // k1=1 and k2=2 - sql1 = "SELECT * from db1.tbl1 where k1 = 1 and k2 = 2"; - stmtExecutor = new StmtExecutor(connectContext, sql1); - stmtExecutor.execute(); - planner = stmtExecutor.planner(); - fragments = planner.getFragments(); - conjuncts = fragments.get(0).getPlanRoot().getChild(0).conjuncts; - predicates = converter.convertToPaimonExpr(conjuncts); - Assertions.assertEquals(predicates.size(), 2); - - // k1 =1 or k2 = 2 - sql1 = "SELECT * from db1.tbl1 where k1 = 1 or k2 = 2"; - stmtExecutor = new StmtExecutor(connectContext, sql1); - stmtExecutor.execute(); - planner = stmtExecutor.planner(); - fragments = planner.getFragments(); - conjuncts = fragments.get(0).getPlanRoot().getChild(0).conjuncts; - predicates = converter.convertToPaimonExpr(conjuncts); - Assertions.assertEquals(predicates.size(), 1); - Assertions.assertTrue(predicates.get(0) instanceof CompoundPredicate); - CompoundPredicate predicate = (CompoundPredicate) predicates.get(0); - Assertions.assertTrue(predicate.function() instanceof Or); - Assertions.assertEquals(predicate.children().size(), 2); - Assertions.assertTrue(predicate.children().get(0) instanceof LeafPredicate); - Assertions.assertTrue(predicate.children().get(1) instanceof LeafPredicate); - } -} diff --git a/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md b/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md new file mode 100644 index 00000000000000..4cadbe1a6e6658 --- /dev/null +++ b/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md @@ -0,0 +1,183 @@ +# P5-T29 (B8) — paimon legacy removal from fe-core (design) + +> **Design-first, firsthand-verified.** Closure produced 2026-06-20 by two parallel re-grep + +> adversarial-verify workflows (`wf_a8bcfb20-405` Plan-A readiness, `wf_8a50af43-7a2` Plan-B +> feasibility) **plus a firsthand conflict-resolution pass** (the two workflows disagreed on whether +> `datasource/paimon/*` is dead; the import-level firsthand check settled it — see §0.1). This doc is +> the execution source for P5-T29. +> Mirrors **P4 #64300** (`73832991962`, "make fe-core odps-free"): delete files + clean reverse-refs + +> drop maven deps + `dependency:tree` verify. +> Sample design = [`P4-batchD-maxcompute-removal-design.md`](./P4-batchD-maxcompute-removal-design.md). + +--- + +## 0. Scope decisions (user-signed 2026-06-20) + +Two decisions were taken via AskUserQuestion after the feasibility dig: + +- **D-PB1 — metastore-props mechanism = B1 (strip SDK in place).** The 7 STILL-CONSUMED + `property/metastore/Paimon*` classes are **kept in fe-core** as thin SDK-free metastore-property + descriptors; their paimon-SDK use (confined to dead catalog-building methods) is stripped. NOT + physically relocated (B2 was rejected — it forces a generic `MetastoreProperties`-registry rework + + cross-loader re-basing for no marginal benefit toward dropping deps, and iceberg/hive keep their + metastore-props in fe-core *with* their engine SDK, so B1 makes paimon the clean outlier — parity-OK). +- **D-PB2 — sequencing = phased.** + - **Batch 1 (this doc, safe core):** delete the 33 DEAD files + reverse-ref cleanups + dead tests + + B1-strip the 6 metastore-props. **paimon maven deps STAY** (still needed by the one genuinely-LIVE + SDK class, `PaimonVendedCredentialsProvider`). + - **Batch 2 (later, docker-e2e-gated, separate):** migrate `PaimonVendedCredentialsProvider` out of + fe-core + rework the generic `VendedCredentialsFactory` paimon seam (shared with iceberg) + **drop + all 5 paimon maven deps**. This is the genuine cross-cutting piece; isolated for review/risk. + +End state after Batch 1+2 = fe-core fully paimon-SDK-free (zero `org.apache.paimon.*` imports), all 5 +paimon maven deps gone. + +### 0.1 Conflict resolution — `datasource/paimon/*` IS dead (firsthand) + +The Plan-B synth claimed `datasource/paimon/*` is LIVE (that `PluginDrivenMvccExternalTable` uses +`PaimonUtil`, sys-table classes use `PaimonSysTable`), which would block the dep drop. **Refuted by +firsthand import check:** the live generic `PluginDrivenMvccExternalTable` / `PluginDrivenExternalTable` +/ `PluginDrivenSysExternalTable` / `systable/PluginDrivenSysTable` / `systable/NativeSysTable` import +**none** of `datasource.paimon.*`, `systable.PaimonSysTable`, or `metacache.paimon.*` — those were +javadoc/comment references. The only live generic importer is `ExternalMetaCacheMgr:35` +(`PaimonExternalMetaCache`, the dead `paimon()`/register branch already on the cleanup list). Plan-A's +DEAD classification stands. + +### 0.2 The 31 paimon-SDK importers in fe-core, decomposed + +`grep -rln "import org.apache.paimon\." fe/fe-core/src/main` = 31 files: +- **~23 DEAD subtree files** → deleted in Batch 1 (§2). +- **6 metastore-props** (`AbstractPaimonProperties` + 5 flavors) → B1-stripped in Batch 1 (§4). SDK is + 100% in dead catalog-building methods; live duties (Kerberos `executionAuthenticator`, type) are SDK-free. +- **`PaimonVendedCredentialsProvider`** → genuinely LIVE (uses paimon REST SDK at runtime via generic + `VendedCredentialsFactory.getProviderType` `case PAIMON`). **Batch 2.** +- **`ShowPartitionsCommand`** → its lone SDK import (`org.apache.paimon.partition.Partition`) dies with + the dead `handleShowPaimonTablePartitions()` method removed in Batch 1 (§3). + +--- + +## 1. Batch 1 — DEAD file deletion set (33 files) + +> Counts firsthand-verified on `branch-catalog-spi` 2026-06-20. **NOTE: 33, not 34** — the Plan-doc +> ledger's "30" for `datasource/paimon/` double-counts `PaimonVendedCredentialsProvider` (LIVE, keep). + +**`datasource/paimon/` — 29 files** (the directory minus the LIVE `PaimonVendedCredentialsProvider.java`): +catalog/table/ops/util (11): `PaimonExternalCatalog`, `PaimonExternalCatalogFactory`, +`PaimonHMSExternalCatalog`, `PaimonFileExternalCatalog`, `PaimonRestExternalCatalog`, +`PaimonDLFExternalCatalog`, `PaimonExternalDatabase`, `PaimonExternalTable`, `PaimonSysExternalTable`, +`PaimonMetadataOps`, `PaimonExternalMetaCache`. Util (2): `PaimonUtil`, `PaimonUtils`. Cache/POJO (9): +`PaimonMvccSnapshot`, `PaimonSnapshot`, `PaimonSnapshotCacheValue`, `PaimonSchemaCacheKey`, +`PaimonSchemaCacheValue`, `PaimonTableCacheValue`, `PaimonPartition`, `PaimonPartitionInfo`, +`DorisToPaimonTypeVisitor`. profile/ (2): `profile/PaimonMetricRegistry`, `profile/PaimonScanMetricsReporter`. +source/ (5): `source/PaimonScanNode`, `source/PaimonSource`, `source/PaimonSplit`, +`source/PaimonPredicateConverter`, `source/PaimonValueConverter`. + +**`datasource/metacache/paimon/` — 3 files:** `PaimonTableLoader`, `PaimonPartitionInfoLoader`, +`PaimonLatestSnapshotProjectionLoader`. + +**`datasource/systable/` — 1 file:** `PaimonSysTable.java` (only consumer `PaimonExternalTable:395`, dead). + +**KEEP (LIVE, do NOT delete):** `datasource/paimon/PaimonVendedCredentialsProvider.java` — reached via +generic `VendedCredentialsFactory.getProviderType()` `case PAIMON` ← `CatalogProperty:182`. Batch 2 target. + +--- + +## 2. Batch 1 — reverse-reference cleanups (live files, sever compile-links to dead classes) + +| File | action | +|---|---| +| `datasource/ExternalCatalog.java` | delete `case PAIMON -> new PaimonExternalDatabase` switch arm + import (PluginDriven forces logType=PLUGIN) | +| `datasource/ExternalMetaCacheMgr.java` | delete `paimon()` accessor + the metacache-local `ENGINE_PAIMON` const + `register(new PaimonExternalMetaCache(...))` line + import | +| `datasource/metacache/ExternalMetaCacheRouteResolver.java` | delete `instanceof PaimonExternalCatalog` block + const + import | +| `catalog/Env.java` | delete `getType()==PAIMON_EXTERNAL_TABLE` legacy branch + 2 imports | +| `nereids/rules/analysis/UserAuthentication.java` | delete `instanceof PaimonSysExternalTable` else-if + import (live `PluginDrivenSysExternalTable` branch handles it) | +| `nereids/trees/plans/commands/ShowPartitionsCommand.java` | **surgical:** drop the 3 dead-class clauses (`instanceof PaimonExternalCatalog`) + `handleShowPaimonTablePartitions()` method + 3 imports (incl `org.apache.paimon.partition.Partition`). **KEEP `hasPartitionStatsCapability()` + the 5-col body.** | + +**KEEP — NOT reverse-refs to delete (LIVE, verified):** +- `credentials/VendedCredentialsFactory.java` `case PAIMON` — LIVE (Batch 2 target, not Batch 1). +- `persist/gson/GsonUtils.java` `registerCompatibleSubtype` **string** aliases (catalog/db/table) — upgrade-compat, string literals, zero compile-link. MUST KEEP (mirrors P4's kept `"MaxComputeExternalCatalog"`). +- `nereids/.../info/CreateTableInfo.ENGINE_PAIMON` — LIVE post-cutover engine name + distribution validation. KEEP. +- `PluginDrivenExternalTable` `case "paimon"` engine-name reporting, `TableType.PAIMON_EXTERNAL_TABLE` enum, `FileQueryScanNode.CACHEABLE_CATALOGS` `"paimon"` — all LIVE. KEEP. + +**Javadoc scrubs (would break strict checkstyle/javadoc after deletion):** +- `datasource/PluginDrivenSysExternalTable.java:34` `{@link ...PaimonSysExternalTable}` → re-point/plain. +- `datasource/systable/PluginDrivenSysTable.java:27` `{@link PaimonSysTable}` → re-point/plain. +- `datasource/systable/NativeSysTable.java:36` `@see PaimonSysTable` → drop/re-point. + +--- + +## 3. Batch 1 — dead tests + +**DELETE (SUT is a DEAD class) — 5:** `datasource/paimon/PaimonExternalMetaCacheTest`, +`datasource/paimon/source/PaimonScanNodeTest`, `planner/PaimonPredicateConverterTest` (legacy DUP converter), +`datasource/paimon/PaimonMetadataOpsTest`, `datasource/paimon/PaimonUtilTest`. + +**TRIM (dead class used only as fixture/mock) — 2:** `datasource/ExternalMetaCacheRouteResolverTest` +(replace `new PaimonExternalCatalog(...)` fixtures; tests LIVE `ExternalMetaCacheMgr`), +`nereids/StatementContextTest` (`testPreloadPaimonLatestSnapshotBeforeLock`: swap +`Mockito.mock(PaimonExternalTable.class)` → `PluginDrivenMvccExternalTable`). + +**KEEP (LIVE) — `datasource/paimon/PaimonVendedCredentialsProviderTest`** (SUT LIVE, Batch 2). + +--- + +## 4. Batch 1 — B1 strip the 6 metastore-props (paimon-SDK-free) + +**Strip from `AbstractPaimonProperties` + 5 flavors:** the `org.apache.paimon.*` imports; +abstract+impl `initializeCatalog(...)`; `buildCatalogOptions()`/`appendCatalogOptions()`/abstract +`appendCustomCatalogOptions()`; abstract+impl `getMetastoreType()` (zero callers outside pkg, firsthand); +the `Options catalogOptions` field + Lombok `getCatalogOptions()`; `appendUserHadoopConfig(Configuration)`; +`getCatalogOptionsMap()`; `normalizeS3Config()` (dead). In Jdbc also drop `getBackendPaimonOptions` + +`registerJdbcDriver`/`appendRawJdbcCatalogOptions`/`DriverShim` if unreferenced after. + +**Decouple from the deleted `PaimonExternalCatalog` constants:** `getPaimonCatalogType()` is dead-API +in MAIN (only dead-subtree callers) but is SDK-free and asserted by 5 metastore-props tests → **KEEP it, +inline its String-literal returns** (`"hms"`/`"filesystem"`/`"dlf"`/`"rest"`/`"jdbc"`) so it no longer +imports `PaimonExternalCatalog.PAIMON_*`. (Removing the dead-API method entirely is an optional follow-up; +out of Batch-1's minimal boundary.) Update the 2 tests asserting via `PaimonExternalCatalog.PAIMON_*` +(`PaimonJdbcMetaStorePropertiesTest:49`, `PaimonRestMetaStorePropertiesTest:41`) to assert the literal. + +**KEEP (all SDK-free, LIVE):** `@ConnectorProperty` fields (`warehouse` …); `Type.PAIMON` enum + +`register(Type.PAIMON, new PaimonPropertiesFactory())` (`MetastoreProperties:90`); `PaimonPropertiesFactory` +(no paimon imports); `initNormalizeAndCheckProps`/`checkRequiredProperties`; `getExecutionAuthenticator`/ +`initExecutionAuthenticator`/`initHdfsExecutionAuthenticator` (build `HadoopExecutionAuthenticator`, SDK-free); +`getPaimonCatalogType` (inlined literals). Examine `AbstractPaimonPropertiesTest` + the 5 flavor tests for +calls into stripped methods (e.g. `buildCatalogOptions`/`getCatalogOptionsMap`) and trim accordingly. + +--- + +## 5. Commit plan (each compiles independently; cycle-safe) + +The dead subtree and the metastore-props are **mutually dependent** (subtree calls `initializeCatalog`; +props reference `PaimonExternalCatalog.PAIMON_*`). **Additionally** the dead subtree calls a *removed* +reverse-ref symbol: `PaimonUtils:57` → `ExternalMetaCacheMgr.paimon()`. So — exactly as P4 #64300 found +("reverse-ref removal and file deletion must land as one compiling unit") — severing the reverse-refs and +deleting the dead files **cannot** be split. Batch 1 = **2 commits**: + +- **C1 (sever reverse-refs + delete dead, atomic):** §2 reverse-ref cleanups (6 files) + §2 javadoc + scrubs (3) + §4 decouple (inline `getPaimonCatalogType` literals in 5 flavors, drop their + `PaimonExternalCatalog` import) + §3 fixture-test trims (2) + 2 constant-test repoints + `git rm` the + 33 dead files (§1) + 5 dead test files (§3). After C1 the metastore-props keep their SDK catalog-building + methods (now caller-less) but still compile against the present paimon deps. + *Verify:* fe-core `test-compile` green; `datasource/paimon/` holds only `PaimonVendedCredentialsProvider`. + *(First attempt split this into prep-then-delete; the `PaimonUtils → paimon()` coupling broke the + intermediate compile — merged per P4 precedent.)* +- **C2 (B1 strip):** §4 strip SDK methods + imports from the 6 metastore-props + trim their tests' + assertions on the stripped catalog-building methods (`buildCatalogOptions`/`getCatalogOptions`/ + `getMetastoreType`). *Verify:* fe-core compiles; checkstyle 0; import-gate net; paimon connector UT + green; `grep org.apache.paimon fe-core/src/main` = only `PaimonVendedCredentialsProvider` remains (Batch 2). + +**Hard pre-commit (HANDOFF):** scrub `regression-test/conf/regression-conf.groovy` (plaintext key); +clean scratch (`.audit-scratch/`/`conf.cmy/`/`META-INF/`/`*.bak`). **Path-whitelist `git add`; NEVER `git add -A`.** +Each commit: `[P5-T29] ` + root cause + fix + tests + `Co-Authored-By: Claude Opus 4.8 (1M context) `. + +--- + +## 6. Verification gates (mirror P4 #64300) + +- [ ] fe-core `compile` BUILD SUCCESS + `testCompile` + checkstyle 0 (`validate` phase) per commit. +- [ ] `tools/check-connector-imports.sh` exit 0. +- [ ] paimon connector module UT green (`-pl :fe-connector-paimon -am package -Dassembly.skipAssembly=true`). +- [ ] After C3: `grep -rl "import org.apache.paimon\." fe/fe-core/src/main` = ONLY `PaimonVendedCredentialsProvider`. +- [ ] Batch 2 (separate): `dependency:tree | grep paimon` = removed set absent; live-e2e `enablePaimonTest=true`. +- [ ] regression-gated live-e2e (B9/P5-T30, user-run) after Batch 2 — 5-flavor read + sys-table + MTMV + DDL no regression. From 58160cbcb5662723a896cf3337fb65056d847830 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 20 Jun 2026 22:53:02 +0800 Subject: [PATCH 2/4] [P5-T29] docs: Batch 1 (C1) done; revised plan defers metastore-props strip to Batch 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records P5-T29 progress after the firsthand scope re-verification this session: - Batch 1 (C1) delete-dead + sever-reverse-refs is done (commit 7632a074e4b). - Corrects the old §D maven framing: neither Plan A nor B drops the 5 paimon deps by only touching the 7 metastore-props — 31 fe-core files import the SDK, ~23 of them the dead subtree C1 already deleted. The remaining SDK importers are the 6 metastore-props (strippable; SDK confined to dead catalog-building) and the genuinely-LIVE PaimonVendedCredentialsProvider (paimon REST SDK at runtime via the generic VendedCredentialsFactory). - User-signed: Plan B (fe-core fully paimon-free) + D-PB1 strip-in-place (keep the 7 thin metastore-props in fe-core, parity with iceberg/hive) + D-PB2 phased. - The B1 strip moved from Batch 1 to Batch 2 (it reshapes 6 live classes + trims 7 catalog-building test files + drops no dep by itself, so it belongs with the Batch-2 VendedCredentialsProvider migration + dep-drop). Adds tasks/designs/P5-T29-paimon-legacy-removal-design.md as the authoritative revised plan (full edit ledger). Updates HANDOFF/PROGRESS/connectors/task-table. Also folds in the prior session's uncommitted P5-T29 setup edits to these docs. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_011mTrPcvMZtFjsxWJM5TRnG --- plan-doc/HANDOFF.md | 309 +++++------------- plan-doc/PROGRESS.md | 32 +- plan-doc/connectors/paimon.md | 49 +-- plan-doc/tasks/P5-paimon-migration.md | 70 +++- .../P5-T29-paimon-legacy-removal-design.md | 50 +-- 5 files changed, 217 insertions(+), 293 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index dda48b4f74da8d..443b7eca8d96be 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -6,245 +6,106 @@ --- -# 🎯 下一个 session 的任务 — **P6-DEVIATIONS 余项 accept-as-deviation 签字 → 见 `task-list-P6-fixes.md` backlog 0 末项**（5 个 deviation→fix **全部完成**：A3 ✅ `5fa47c27eb8` / A2 ✅ `1935748d6c3` / B-MC2 ✅ `10284edbf88` / A1 ✅ `9d687145a28` / B-R2-be ✅ `60ed665c4dc`。**本批清零。** 下一步=把未转 fix 的剩余 MINOR/NIT 刻意偏离 + wave2 新增 + `PluginDrivenExternalCatalog:140` 吞 authenticator-wiring 异常逐条记入新建 `deviations-log.md`（含用户签字）；之后 B8 legacy 删除/元存储子线 P2-T04/05） - -> **进度（2026-06-19）**：P6 发现项按 `task-list-P6-fixes.md` 的 prioritized list 逐个修（单任务循环： -> design → 红队 → 实现 → impl 验证 → build+UT → commit）。 -> **✅ C1 (MinIO, MAJOR) `9967846ef64`** / **✅ C2 (HDFS XML, MAJOR) `e95128aed5b`** / **✅ R3-residual (MINOR) `44499f073e8`**（详见 git log + 各自 design/summary）。 -> **✅ A3 (NIT, profile-parity, deviation 1/5) `5fa47c27eb8`**：`PaimonScanRange` ctor 发 `paimon.self_split_weight` 的闸由 -> 值判断 `selfSplitWeight > 0` 改为 JNI 标记 `paimonSplit != null`——weight=0 的 JNI split（rowCount-0 系统表 split / -> fileSize-0 DataSplit）现也发 0，BE 不再读 -1 哨兵（profile counter `_max_time_split_weight_counter`）；= legacy -> `PaimonScanNode.setPaimonParams:274`（JNI 臂无条件、native 臂从不发）parity。新 `PaimonScanRangeSelfSplitWeightTest`（3， -> **RED→GREEN 两次独立跑验证**：未修代码 weight-0 测失败 1，修后 3/0）；283/0/0/1skip + checkstyle0 + import0；design 红队 -> `wf_3f2cd605-2a8`（9 候选→0-actionable on code）、impl-verify **APPROVE**；e2e gated 未跑。详见 -> `designs/FIX-A3-SELF-SPLIT-WEIGHT-{design,summary}.md`。 -> **✅ B-R2-be (NIT, intentional, deviation 5/5 = LAST, ⚠️NO PERF REGRESSION) `60ed665c4dc`**：schema-evolution -> 字典的 per-schema-id 读做 memo。**收窄方案被否（架构上连接器内做不到 + BE 崩风险）**：props 常先于 split 构建、 -> `getScanPlanProvider()` 每次 new provider 故 planScan 的 schema_id 到不了字典构建、引用集是 per-scan、通用桥不能收 -> `paimon.schema_id`、props 里重 plan=新 I/O、收窄漏发即 BE 硬崩（CI 969249）。**用户选 Option A=memo 读、保贪婪全集发射**： -> 字典发射**字节不变**（全 `listAllIds()`→永覆盖→零 BE 崩险），只把 per-schema-id 字段**读**走连接器级不可变 memo—— -> **复用 B-MC2 `PaimonSchemaAtMemo`**（已缓存同一 write-once 事实 `(handle,schemaId)→fields`）。新 package-private 4-arg -> provider ctor（2/3-arg 委托 fresh memo→~25 处不变）；`buildSchemaEvolutionParam` 收 handle + memo 包 loop（loader 走 -> **直读** `schemaManager.schema(id)` 非 `catalogOps.schemaAt`，real-table+fake-catalogOps 测不破）；`getScanPlanProvider()` -> 注入共享 memo。一致性（同 key 同值）经 write-once 不变式 + B-MC2 从不写 $ro/sys key 验证。+5 UT（memo-populated/ -> sentinel-HIT/byte-identical/force-jni/wiring）各 RED→GREEN；303/0/1skip；checkstyle0+import0；design 红队 -> `wf_222e1abd-655`（4 lens sound、荐复用、6 actionable 折叠）、impl-verify COMMIT_AS_IS（0 defect）；e2e gated 未跑。 -> 详见 `designs/FIX-B-R2-BE-SCHEMA-DICT-MEMO-{design,summary}.md`。 -> **✅ A1 (MINOR, FE 调度回归, deviation 4/5) `9d687145a28`**：把连接器算好的比例 split weight 接到 FE `FileSplit` -> 调度字段（`FederationBackendPolicy` 按大小分配，legacy parity；FE BE-分配 only，不改行/路由/BE读/结果）。SPI -> `ConnectorScanRange` 加默认 `getSelfSplitWeight()`/`getTargetSplitSize()`（哨兵 -1）；`PluginDrivenSplit` ctor -> 仅当 `weight>=0 && target>0` 时填字段（通用，别的连接器继承 -1→保持 `standard()` 无回退）；`PaimonScanRange` 携 -> `targetSplitSize`（Builder 默认 -1）+ `@Override` 两 getter；`PaimonScanPlanProvider` 算 `resolveSplitWeightDenominator` -> （= legacy `fileSplitSize>0 ? : max_file_split_size` 64MB）一次并把 `weightDenominator` 穿到每个 builder。**task-list 漏的 -> 关键缺口（靠追 legacy 抓出）**：native 范围从不设 `selfSplitWeight`（默认 0）——legacy 用 `length(+DV)`（`PaimonSplit:72,112`）， -> native 是默认路径,weight=0 会 clamp 0.01（均匀）使修复失效；故 `buildNativeRange` 现设 `length+DV`+denominator。FE-only -> （BE-thrift `paimon.self_split_weight` 仍 A3-gated on `paimonSplit`）。新 `PluginDrivenSplitWeightTest`(fe-core,5)+ -> `ConnectorScanRangeWeightDefaultsTest`(api)+5 `PaimonScanPlanProviderTest`+6 改签名调用点；各由独立 mutation -> **RED→GREEN 验证**（ctor-gate/native-weight/sentinel/denominator/swap）；api 44/0+paimon 298/0/1skip+fe-core 5/0; -> checkstyle0+import0；design 红队 `wf_c8345c28-ee6`（4 lens sound,6 actionable 已折叠）、impl-verify `wf_3381cfaa-205` -> （2 lens 全 COMMIT_AS_IS,0 actionable）；e2e gated 未跑。详见 `designs/FIX-A1-SPLIT-WEIGHT-{design,summary}.md`。 -> **✅ B-MC2 (NIT, CACHE-P1, deviation 3/5, ⚠️NO PERF REGRESSION) `10284edbf88`**：恢复 time-travel -> schema-at-snapshot 的跨查询二级缓存（SPI cutover CACHE-P1 丢弃）。新连接器侧 `PaimonSchemaAtMemo` -> （`ConcurrentHashMap`，loader 在锁外 + best-effort clear-on-overflow 界）， -> 由**长寿命 per-catalog `PaimonConnector`** 持有（REFRESH→onClose connector=null→重建→空 memo）并经**新 -> package-private 4-arg ctor** 注入每查询的 metadata（public 3-arg 委托一个 fresh per-instance memo→~15 处构造点不变）。 -> `getTableSchema(snapshot)` schemaId>=0 臂：`resolveTable` 在 loader 外**只调一次**（branch handle 的 getTable 仍 1/查询）， -> memo 只包 `schemaAt` 读，`ConnectorTableSchema` 每查询**重建**。**design 红队 MAJOR 已采纳**：缓存原始 -> `PaimonSchemaSnapshot`（key 的纯函数），**非**built `ConnectorTableSchema`（它嵌 live `coreOptions`→陈旧属性风险）。 -> `MemoKey`=抽取的 handle 身份(db,table,sysName,branch)+schemaId（不留 handle 引用→不钉住已加载的 paimon Table， -> 偏离红队「delegate to handle.equals」是为避免钉住 Table）。+3 `PaimonConnectorMetadataMvccTest` + 新 -> `PaimonSchemaAtMemoTest`(3)，各由独立 mutation 跑 **RED→GREEN 验证**（RED-1 memo禁/RED-2 key丢字段/RED-3 界禁）； -> 293/0/0/1skip + checkstyle0 + import0；design 红队 `wf_903bf4e9-3a4`、impl-verify `wf_67804f35-d5e` -> （2×COMMIT_AS_IS+1×FIX_THEN_COMMIT=仅 verifier 自留 scratch，产品码干净）；e2e gated 未跑。详见 -> `designs/FIX-B-MC2-SCHEMA-AT-MEMO-{design,summary}.md`。 -> **✅ A2 (MINOR, missing-port, deviation 2/5) `1935748d6c3`**：`appendExplainInfo` 反序列化已在 props 里的 -> `paimon.predicate`（即推给 SDK 的 `List`）重发 legacy `predicatesFromPaimon:` 块，置于 -> `paimonNativeReadSplits=` 与 VERBOSE `PaimonSplitStats` 之间（legacy 序 `PaimonScanNode:657-671`）。不重跑 converter -> （filter 不在 seam、provider 每次新建）；absent≠empty 跳过（保 exact-equality 旧测）；decode 失败 LOG.warn+跳过； -> 不改 SPI、无 BE 影响（`populateScanLevelParams` 逐键读，新键也到不了 BE）。4 新 `PaimonScanExplainTest`（**RED→GREEN -> 分跑**：未修 3 失→修后 0）；287/0/0/1skip + checkstyle0 + import0；design 红队 `wf_c67cb558-ff4`（13 候选→0-actionable -> on code，已折叠文档/测试细化）、impl-verify APPROVE。详见 `designs/FIX-A2-PREDICATES-FROM-PAIMON-{design,summary}.md`。 -> **✅ R3-residual (MINOR) 已完成**：去 `PluginDrivenScanNode.getNodeExplainString` 的 `"paimon".equals(getType())` -> gate，VERBOSE backends 块改无条件 emit（gate 变 `VERBOSE && !isBatchMode()`，与父 `FileScanNode` 完全一致）+ 重写假注释。 -> **红队纠正了 scope**（比 review 的「maxcompute」更广）：`SPI_READY_TYPES={jdbc,es,trino-connector,max_compute,paimon}` 全走 -> 此 node → paimon 不变、maxcompute/trino **恢复** pre-cutover 块、es/jdbc 获得**新增**（NPE-safe、合规则）VERBOSE 输出 -> （`PluginDrivenSplit extends FileSplit` 恒有 FileScanRange + `getDeleteFiles` null-guard；es/jdbc 渲染合成路径 -> `es://idx/shard`、`jdbc://virtual`）。新 `PluginDrivenScanNodeVerboseExplainTest`（3 测，**RED→GREEN 突变验证**： -> 重加 gate → 非-paimon 测变红）；45/0/0 `PluginDrivenScanNode*` UT + checkstyle 干净；**e2e gated/未跑**。 -> es_http `ES terminate_after:` gate 作**独立残留**留下（R3-LAYER-2，键 file-format-type 非 getType()，规则字面不违反）。 -> 设计/红队结论详见 `designs/FIX-R3-RESIDUAL-{design,summary}.md`（design 红队 3 lens finder→verifier `wf_3518653b-3cb`）。 -> **✅ R1-table (MINOR) 已完成**：`PluginDrivenExternalCatalog.createTable`（**通用桥**，全 SPI 连接器）去 `if (localExists)` -> 守卫 → 存在分支无条件报 `ERR_TABLE_EXISTS_ERROR`（MySQL **1050**/42S01），在 `metadata.createTable` 前短路。修「表只远端存在、 -> 本 FE 缓存缺」（陈旧缓存/他 FE/外部建）+ 无 IF NOT EXISTS 时丢 1050 退化成泛化 DdlException(errno 0)。精确 legacy parity -> （paimon `:195/:212` + maxcompute `:184/:195`，remote+local 两臂皆 1050）。es/jdbc/trino 对已存在表 CREATE 现报「already exists」(NIT)。 -> 改写 remote 测 + 强化 local 测加 errno 断言（**RED→GREEN 突变验证**：重加守卫→remote 测红）；26/0/0 DdlRouting + 12/0/0 Engine + -> checkstyle 干净；**e2e gated**。design 红队 `wf_19fd7785-165`（0 actionable）。详见 `designs/FIX-R1-TABLE-{design,summary}.md`。 -> **✅ C4 / R2-catalog / R3-catalog（3 MINOR，合一）已完成 `82b6de0de98`**：**C4** 透传 -> `Config.hive_metastore_client_timeout_second`（env key `hive_metastore_client_timeout_second` → `HmsMetaStoreProperties -> .toHiveConfOverrides(String)`，去硬编码 `"10"`；fe.conf 未设时 byte-parity，恢复 `HMSBaseProperties:204-208`）。**R2-catalog** -> 改 **warn-only**（非 strip，用户拍板）在 `PaimonConnectorProvider.validateProperties` 提示死键 `meta.cache.paimon.table.*`—— -> 经 `getMetaCacheEngine()=="default"`（PluginDriven 不 override）证实 plugin 路从不碰 `PaimonExternalMetaCache`，键确死； -> warn 落连接器（非 connector-agnostic 桥，report 引的位置=错层）。**R3-catalog** 改 **rethrow**（用户拍板，非仅加 catalog 名）—— -> `listDatabaseNames` 抛 `RuntimeException("Failed to list databases names, catalog name: ")` 与 legacy -> `PaimonMetadataOps:340` 完全一致（且所有其它连接器都 propagate），原先吞成 emptyList 且注释谎称 parity。280/0 paimon(+1 -> gated skip)+16/0+3/0+14/0+12/0；fe-core 编译过；checkstyle 0；import-check 干净；design+impl 两道红队均 0-actionable；e2e gated。 -> 详见 `designs/FIX-C4-R2-R3-CATALOG-{design,summary}.md`（design 红队 `wf_444e33b9-5c6`、impl 红队 `wf_b3d35e64-6b9`）。 -> **5 个 deviation→fix 全部完成**（A3 `5fa47c27eb8` / A2 `1935748d6c3` / B-MC2 `10284edbf88` / A1 `9d687145a28` -> / B-R2-be `60ed665c4dc`）。**下一个 = P6-DEVIATIONS 余项 accept-as-deviation 签字**：未转 fix 的剩余 MINOR/NIT -> 刻意偏离 + wave2 新增 + `PluginDrivenExternalCatalog:140` 吞 authenticator-wiring 异常 → 逐条记入新建 -> `deviations-log.md`（含用户签字）。**注**：B-R2-be 的「收窄」方案经分析架构不可行（见上方 ✅ B-R2-be 条 + 设计文档）， -> 已与用户确认改用 memo（Option A），亦应在 deviations-log 记一条「R2-be 收窄不可行→memo」的决策。之后才是 P6-fixes 批清零。 - -paimon connector 全功能路径 clean-room 对抗 review（6 维度 + 7 缺口线，2 波，零历史先验）**已完成**。 -报告：[`reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md`](./reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md)（未跟踪，待 vet+commit）。 -统计：**2 BLOCKER · 2 MAJOR · 16 MINOR · 10 NIT**（27 confirmed / 3 partial / 3 refuted）。方法：wave1 = 9 finder 线归 6 维度 -（read×2/write/ddl×2+config/replay/cache/residual），wave2 = 补 7 缺口线（show-partitions / partitions-TVF / 统计-ANALYZE / -@branch / MTMV / auth-UGI / config→BE），每线 finder→对抗 verifier；fresh subagent 仅喂代码+维度问题（成功挡住历史先验）。 - -**核心结论（详见报告）**： -- **2 BLOCKER 都是 B8 删除护栏、非运行时 bug**：R1 = legacy `property/metastore/Paimon*MetaStoreProperties` + `PaimonExternalCatalog` - **常量**仍 LIVE（cutover 的 `initPreExecutionAuthenticator`→Kerberos 装配经它）；R2 = `property/storage/{S3,OSS,COS,OBS,Minio}Properties` - 是**跨连接器共享**（~26 消费者 iceberg/hive/glue/dlf/storage-vault/load/cloud/policy）。→ **B8 不能整包删，必须分阶段**。 -- **2 MAJOR 是真活读路回归**（不挡 B8，应随 cutover 修）：**C1** = `minio.*`-keyed catalog 整条不可用（FE 建表 + BE 读，两波独立证实； - fe-filesystem 无 MinIO provider，S3 provider 不认 `minio.*`；2026-06-14 的 applyCanonicalMinioConfig 未进本分支）；**C2** = HDFS - `hadoop.config.resources` XML 未注入 FE 建表 Configuration（filesystem/jdbc flavor）→ XML-only HA 拓扑解析不到 nameservice。 - **C2 的 kerberos-by-alias 子项被 wave2 证伪**（per-FS Configuration 的 auth marker 非负载性：JVM-global `UGI.setConfiguration` 主导 SASL）→ 只修 XML。 -- **其余全 parity**：replay/GSON 干净（0 缺陷）、scan→BE 契约（历史 double-fill / `file_format=jni` / schema-evo `-1` bug 均已修）、 - write（无写路、两侧都 loud-reject）、cache pin 模型、SHOW PARTITIONS（critic 的 `VARCHAR(60→300)` 担忧被证伪：master 早已 300）、 - partitions-TVF、统计/ANALYZE（row-count 一致、column-stat 两侧空、ANALYZE 走 generic）、@branch、MTMV 新鲜度、auth/UGI（split-plan - 等不裹 `executeAuthenticated` 与 legacy 完全一致 → 非回归，了结 HANDOFF 旧 open item）。MINOR/NIT 多为 EXPLAIN/profile/错误码 - parity 或刻意更安全的偏离。 - -**下一步**：本轮是 review、**未改任何代码**（除报告本身 + 我修正了 writer 的计数）。发现项各自另起 fix task（见下方 backlog 0 + 报告 -§Coverage gaps & follow-ups 的 prioritized fix-task list）。**AGENT-PLAYBOOK 单任务循环：先 review 方案后实现**。 +# 🎯 下一个 session 的任务 — **P5-T29（批 B8）Batch 2：strip metastore-props SDK + 迁 `PaimonVendedCredentialsProvider` + 删 paimon maven 依赖（docker-gated）** + +> 📍 **完整修订计划（authoritative，含 firsthand 核实 + 用户签字 D-PB1/D-PB2 + 逐文件 edit 清单）见 +> [`tasks/designs/P5-T29-paimon-legacy-removal-design.md`](./tasks/designs/P5-T29-paimon-legacy-removal-design.md)**（本 session 2026-06-20 新建）。 +> 下文 §「P5-T29 scope ledger」是旧框架，部分被该 design doc **取代**（尤其 §B 常量「搬家」实际改为内联字面量、§D maven 决策见下「关键 scope 修正」）。 +> 样板 = **P4 #64300**（`73832991962`）；对照基线 = `reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md` §B8 ledger。 + +**✅ Batch 1（C1）已完成 + local-commit `7632a074e4b`（未 push）**：删 **33 dead 文件**（`datasource/paimon/*` 除 LIVE `PaimonVendedCredentialsProvider`、`metacache/paimon/*`、`systable/PaimonSysTable`）+ 清 **6 处 live reverse-ref**（`ExternalCatalog`/`ExternalMetaCacheMgr`/`ExternalMetaCacheRouteResolver`/`Env`[保 LIVE D-046 PLUGIN 分支]/`UserAuthentication`/`ShowPartitionsCommand`[保 `hasPartitionStatsCapability`+live `PAIMON_EXTERNAL_TABLE` 枚举]）+ **3 javadoc scrub** + **5 dead test 删** + 2 generic fixture test 修（`StatementContextTest` mock→`PluginDrivenMvccExternalTable`、`ExternalMetaCacheRouteResolverTest`）+ metastore-props `getPaimonCatalogType` 内联字面量（脱钩已删的 `PaimonExternalCatalog`，免「常量搬家」前置）。**fe-core test-compile BUILD SUCCESS + checkstyle 0 + 49 改动测试绿**；`datasource/paimon/` 现仅剩 `PaimonVendedCredentialsProvider`。`reverse-ref + 删文件须同一 commit`（P4 precedent：`PaimonUtils:57`→已删的 `ExternalMetaCacheMgr.paimon()`）。 + +**🔱 关键 scope 修正（本 session firsthand，推翻旧 §D 框架）**：方案 A/B 都**只碰 7 个 metastore-props**，都**不能单独删** 5 个 paimon maven 依赖——31 个 fe-core 文件 import `org.apache.paimon.*`，其中 ~23 是 Batch 1 已删的 dead 子树；剩 **6 metastore-props**（SDK 100% 在 dead catalog-building 方法→可 strip）+ **`PaimonVendedCredentialsProvider`**（genuinely LIVE，runtime 用 paimon REST SDK，挂在 generic `VendedCredentialsFactory.getProviderType` 的 `case PAIMON`，经 `CatalogProperty:182`）。**用户签 = Plan B（fe-core fully paimon-free）+ D-PB1 strip-in-place（不物理搬 7 类，与 iceberg/hive parity）+ D-PB2 phased**。strip 因「reshape 6 live 类 + trim 7 test 文件 + 单独不删 dep」从 Batch 1 **移到 Batch 2**（用户 2026-06-20 签）。 + +**Batch 2（下一步，docker-gated，design doc §4 有逐文件清单）**： +1. **B1-strip 6 metastore-props**（`AbstractPaimonProperties`+5 flavor）：删 `initializeCatalog`/`buildCatalogOptions`/`appendCatalogOptions`/`appendCustomCatalogOptions`/`getCatalogOptionsMap`/`getCatalogOptions`(catalogOptions 字段)/`getMetastoreType`/`appendUserHadoopConfig`/`normalizeS3Config`/Jdbc `getBackendPaimonOptions`+`registerJdbcDriver`+`DriverShim` + 全 `org.apache.paimon.*` import。**保 LIVE**：`warehouse` @ConnectorProperty、`executionAuthenticator`+`getExecutionAuthenticator`、`initExecutionAuthenticator`/`initHdfsExecutionAuthenticator`（`PluginDrivenExternalCatalog:137-138` 读，Kerberos doAs）、`initNormalizeAndCheckProps`/validation、`getPaimonCatalogType`（已内联）。**这些 strip 方法 0 live main caller**（只 test）。 +2. **trim 7 test**：`PaimonCatalogTest`（@Disabled 手测→直接删）、`AbstractPaimonPropertiesTest`（test-local subclass override 被删的抽象方法→修）、`Paimon{HMS,FileSystem,Jdbc,Rest,AliyunDLF}MetaStorePropertiesTest`（去 catalog-building 断言，保 validation/binding/type/auth）。 +3. **迁 `PaimonVendedCredentialsProvider` 出 fe-core** + 改 generic `VendedCredentialsFactory`（switch on `MetastoreProperties.Type.PAIMON`，与 iceberg 共享；需新 fe-core seam 让 plugin-loader 侧 provider 喂回，cross-loader）。**这是真正的 cross-cutting 件**，碰 generic/shared fe-core（iceberg 也在同 factory）。 +4. **删 5 paimon maven dep**（`fe-core/pom.xml` paimon-core/common/format/s3/jindo）+ 改 `:577` s3-transfer-manager 注释（iceberg-aws 仍需故 s3-transfer-manager 留）。**验**：`grep org.apache.paimon fe-core/src/main`=∅ + `dependency:tree|grep paimon`=∅ + checkstyle0 + import-gate净 + live-e2e `enablePaimonTest=true`（5-flavor 读+vended REST/DLF+Kerberos HMS+sys-table+MTMV+DDL 不回归）。 + +## P5-T29 scope ledger（已在 `branch-catalog-spi` firsthand 核实 2026-06-20） + +**这不是一次 `rm -rf datasource/paimon/`**：存在「STILL-CONSUMED 子树」与「常量耦合」前置，naive 删除会断编译。 + +### A. DEAD —— 可删（连同消费方的死分支/import） +- `fe/fe-core/.../datasource/paimon/`（**30 文件**，含 `source/`、`profile/`：catalog/factory/db/table、`PaimonExternalCatalog`、`PaimonExternalMetaCache`、`PaimonSysExternalTable`、legacy `source/PaimonScanNode`/`PaimonSplit`/`PaimonSource`、legacy 重复 `source/PaimonPredicateConverter`/`PaimonValueConverter`(P1-T02 推迟项，现可收) 等）。 +- `fe/fe-core/.../datasource/metacache/paimon/`（**3 文件**：`PaimonTableLoader`/`PaimonPartitionInfoLoader`/`PaimonLatestSnapshotProjectionLoader`）。 +- `fe/fe-core/.../datasource/systable/PaimonSysTable.java`（**1 文件**）。 +- **消费方死分支/import 清理**（文件保留，只删 paimon 分支）：`ExternalMetaCacheMgr`（`paimon()`/`ENGINE_PAIMON` 路由 + `PaimonExternalMetaCache` 返回）、`metacache/ExternalMetaCacheRouteResolver`（`ENGINE_PAIMON` 注册）、`catalog/Env`（getDdlStmt 等 legacy 分支若有残留）、`nereids/rules/analysis/UserAuthentication`、`nereids/.../ShowPartitionsCommand`、`credentials/VendedCredentialsFactory`、`ExternalCatalog`（`buildDbForInit` 死分支）。逐个 grep 确认是死分支再删。 +- **死测试**：`ExternalMetaCacheRouteResolverTest`、`planner/PaimonPredicateConverterTest`（测 legacy 重复转换器）、`StatementContextTest`（paimon 用法）等——按编译失败/语义死亡逐个判。 + +### B. 硬前置（删 datasource/paimon/ **之前**必做，否则断编译/checkstyle） +1. **迁出 `PaimonExternalCatalog` 常量**：`PAIMON_FILESYSTEM`/`PAIMON_HMS`（及其它被引常量）被 **5 个 STILL-CONSUMED** `property/metastore/Paimon*MetaStoreProperties` 类 `import` 引用（已核实）。须先把这些常量搬到一个存活的家（metastore-props 模块内的常量持有者 / `fe-kerberos` / 新常量类），再删 `datasource/paimon/PaimonExternalCatalog`。 +2. **scrub 悬空 javadoc** `{@link PaimonSysTable}`（如 `PluginDrivenSysTable`、`NativeSysTable` 里的 `@link`）否则 strict checkstyle/javadoc 挂。 +3. **保 load-bearing dispatch ordering**（PluginDriven 分支须先于任何 legacy 分支）。 +4. **`ENGINE_PAIMON` 区分**：`metacache` 两处是 DEAD（删）；但 `nereids/.../info/CreateTableInfo.ENGINE_PAIMON`（`:123`，被 `:790/:937/:967/:1150` 用作**翻闸后的 engine 名 + distribution 校验**）是 **LIVE，保留**。 + +### C. STILL-CONSUMED —— **不在 P5-T29 删除范围**（删了会断 cutover 的 Kerberos 装配） +- `fe/fe-core/.../datasource/property/metastore/Paimon*MetaStoreProperties`（HMS/DLF/Rest/Jdbc/FileSystem，5）+ `AbstractPaimonProperties` + `PaimonPropertiesFactory`（共 7）。这些是 cutover `initPreExecutionAuthenticator`→Kerberos 装配 **LIVE** 路径（P6 review R1）。它们的测试 `Paimon*MetaStorePropertiesTest` 同样保留。 +- 这些类属 **metastore-storage-refactor 子线**（D-016 那条线也不碰），不在主线 B8 scope。 + +### D. Maven 依赖（用户明确点名「相关 maven 依赖」）—— ⚠️ **核心冲突，须先定决策** +`fe/fe-core/pom.xml` 现含 5 个 paimon 依赖：`paimon-core`、`paimon-common`、`paimon-format`、`paimon-s3`、`paimon-jindo`（`:543-563`）+ s3 aws-bundle 注释（`:576`，与 iceberg-aws 共享）。 +- **关键事实**：C 项的 STILL-CONSUMED `property/metastore/Paimon*` 类 **直接 import `org.apache.paimon.*` SDK**（已核实 6 文件）。⇒ **只要这些类留在 fe-core，fe-core 就不可能像 P4(odps-free) 那样做到完全 paimon-free。** +- **可能可删**：`paimon-format`/`-s3`/`-jindo`（legacy reader/格式/对象存储 IO 路径专用，随 `datasource/paimon/source` 删除而无消费方）。 +- **可能保留**：`paimon-core`/`-common`（被 STILL-CONSUMED metastore-props 的 `Options`/`CatalogContext` 等用）。 +- 真实可删集合须由下一 session 经 `dependency:tree | grep paimon` + fe-core 编译验证敲定。 +- **🔱 开放决策（建议下一 session 先 AskUserQuestion）**：P5-T29 是否把 `property/metastore/Paimon*` 一并迁出 fe-core（→ metastore SPI 模块/连接器），从而让 fe-core 完全 paimon-free？ + - **方案 A（推荐，对齐 master plan B8 / D-016 scope）**：保留 STILL-CONSUMED metastore-props 在 fe-core，**只删 DEAD 子树 + 部分 maven 依赖**（fe-core 保留 paimon-core/common）。surface 小、与已签 B8 scope 一致。 + - **方案 B（更大，越界子线）**：连带迁出 metastore-props，fe-core 完全 paimon-free（对齐 P4 终态）。但这碰 metastore-storage-refactor 子线领域，scope/风险更高，宜单独立项或与子线 P2-T05 合并。 + +### E. 守门 / 验证（mirror P4 #64300） +- fe-core 编译 BUILD SUCCESS + checkstyle 0 + import-gate 净（`tools/check-connector-imports.sh`）。 +- 连接器测试仍绿（删 legacy 不应触连接器）。 +- `dependency:tree` 验证 paimon-core 在 FE classpath **恰一份**（R-004/R-007 `NoClassDefFound`/SDK 单例守）。 +- regression-gated live-e2e（`enablePaimonTest=true`，用户跑）= 删除后 5-flavor 读 + sys-table + MTMV + DDL 不回归。 +- 逐子树删 + 每批跑编译，参 master plan [§3.9/§4 playbook 第 13 步](./00-connector-migration-master-plan.md)。 --- -# 🔭 主线 backlog（P6 review 已出报告，按此排） - -0. **修复 P6 发现项**（报告 §Coverage gaps & follow-ups → prioritized fix-task list；每个独立 fix task； - 逐项进度见 `task-list-P6-fixes.md`）： - - ✅ **C1 MinIO**（MAJOR）— **DONE `9967846ef64`**（minio.* 别名进共享 fe-filesystem-s3 + 保留 tuning 默认；28/0/0 UT）。 - - ✅ **C2 HDFS XML**（MAJOR）— **DONE `e95128aed5b`**（`HdfsFileSystemProperties implements HadoopStorageProperties`； - FE `toHadoopConfigurationMap()` 返 **defaults-free** 图避免多后端 `fs.s3a.*` clobber，BE `toMap()` 仍 defaults-laden； - DLF=DV-036、disable-cache=DV-037；28/0+279/0/1skip+glue test）。 - - ✅ **R3 residual**（MINOR）— **DONE**：去 `PluginDrivenScanNode.getNodeExplainString` 的 `"paimon".equals(getType())` - gate，VERBOSE backends 块无条件 emit（与父 `FileScanNode` gate 一致）+ 重写假注释。红队纠正 scope=全 5 个 SPI 连接器 - （paimon 不变 / maxcompute+trino 恢复 / es+jdbc 新增 NPE-safe 输出）；新 UT 3 测 RED→GREEN；45/0/0 + checkstyle 干净。 - es_http gate 留作 R3-LAYER-2 独立残留。详见 `designs/FIX-R3-RESIDUAL-{design,summary}.md`。 - - ✅ **R1 table**（MINOR）— **DONE `44499f073e8` 之后**：`PluginDrivenExternalCatalog.createTable`（通用桥）去 `if (localExists)` - 守卫 → 存在分支无条件报 `ERR_TABLE_EXISTS_ERROR`(1050)，在 `metadata.createTable` 前短路；精确 legacy parity（paimon+maxcompute - remote+local 两臂皆 1050）；改写 remote 测 + 强化 local 测 errno 断言（RED→GREEN）；26/0/0+12/0/0+checkstyle 干净；红队 0 actionable。 - 详见 `designs/FIX-R1-TABLE-{design,summary}.md`。 - - ✅ **C4 / R2-catalog / R3-catalog**（3 MINOR，合一）— **DONE `82b6de0de98`**：C4 透传 - `Config.hive_metastore_client_timeout_second`（去硬编码 `"10"`，fe.conf 未设 byte-parity）；R2-catalog **warn-only** - （非 strip，用户拍板）提示死键 `meta.cache.paimon.table.*`（`getMetaCacheEngine()=="default"` 证实 plugin 路不碰 - `PaimonExternalMetaCache`，键确死）；R3-catalog **rethrow**（用户拍板）`RuntimeException` 带 catalog 名，与 legacy - `PaimonMetadataOps:340` 一致（原吞成 emptyList）。280/0+16/0+3/0+14/0+12/0；checkstyle 0；两道红队 0-actionable。 - 详见 `designs/FIX-C4-R2-R3-CATALOG-{design,summary}.md`。 - - ✅ **A3**（NIT profile-parity）— **DONE `5fa47c27eb8`**：`PaimonScanRange` ctor `self_split_weight` 闸 `>0`→ - `paimonSplit != null`（emit-iff-JNI = legacy `PaimonScanNode:274` parity）；weight-0 JNI 现发 0；新 UT 3 RED→GREEN； - 283/0/0/1skip。详见 `designs/FIX-A3-SELF-SPLIT-WEIGHT-{design,summary}.md`。 - - ✅ **A2**（MINOR missing-port）— **DONE `1935748d6c3`**：`appendExplainInfo` 反序列化 `paimon.predicate` 重发 legacy - `predicatesFromPaimon:`（置于 `paimonNativeReadSplits=` 与 VERBOSE `PaimonSplitStats` 间，legacy 序）；不重跑 converter； - 4 新 UT RED→GREEN；287/0/0/1skip。详见 `designs/FIX-A2-PREDICATES-FROM-PAIMON-{design,summary}.md`。 - - ✅ **B-MC2**（NIT CACHE-P1，⚠️NO PERF REGRESSION）— **DONE `10284edbf88`**：连接器侧 `PaimonSchemaAtMemo` - （`ConcurrentHashMap`，loader 锁外 + clear-on-overflow 界）由 per-catalog `PaimonConnector` 持有并注入每查询 metadata； - `getTableSchema(snapshot)` schemaId>=0 臂 memo 只包 `schemaAt` 读、`ConnectorTableSchema` 每查询重建（红队 MAJOR：缓存 - 原始 `PaimonSchemaSnapshot` 非 built schema 以保 live coreOptions）；+3 Mvcc UT + 新 `PaimonSchemaAtMemoTest`(3) 各 RED→GREEN； - 293/0/0/1skip。详见 `designs/FIX-B-MC2-SCHEMA-AT-MEMO-{design,summary}.md`（design 红队 `wf_903bf4e9-3a4`、impl-verify `wf_67804f35-d5e`）。 - - ✅ **A1**（MINOR FE 调度回归）— **DONE `9d687145a28`**：SPI `ConnectorScanRange` 加 `getSelfSplitWeight`/ - `getTargetSplitSize`（哨兵 -1）；`PluginDrivenSplit` ctor 仅 `weight>=0 && target>0` 填 FileSplit 权重； - `PaimonScanRange` 携 `targetSplitSize`（默认 -1）；`PaimonScanPlanProvider` 算 denominator（legacy 64MB 公式） - 穿到每 builder + **native 范围补 `selfSplitWeight=length+DV`**（task-list 漏的关键缺口，native 是默认路径否则均匀）。 - fe-core 5 + api 1 + 连接器 5 新 UT 各 RED→GREEN；298/0/1skip+5/0+44/0；两道红队全过。详见 `designs/FIX-A1-SPLIT-WEIGHT-*.md`。 - - ✅ **B-R2-be**（NIT intentional，⚠️NO PERF REGRESSION）— **DONE `60ed665c4dc`**：收窄方案架构不可行→ - 用户选 Option A=memo per-schema-id 读、保贪婪全集发射（字节不变→零 BE 崩险）；复用 B-MC2 `PaimonSchemaAtMemo`； - +5 UT RED→GREEN；303/0/1skip；红队+impl-verify 全过。详见 `designs/FIX-B-R2-BE-SCHEMA-DICT-MEMO-*.md`。 - - **5 个 deviation→fix 全部完成（本批清零）；下一个 = 下面这条 P6-DEVIATIONS 余项签字。** - - **P6-DEVIATIONS 余项（5 项之后，本批最后一项）**：未转 fix 的剩余 MINOR/NIT 刻意偏离 + wave2 新增 + - `PluginDrivenExternalCatalog:140` 吞 authenticator-wiring 异常（R3/R4/R5/R6 residual 属 B8 清理、2 BLOCKER 属 B8 护栏， - 均不在此）。逐条记入新建 `deviations-log.md` accept-as-deviation（含用户签字）。 -1. **B8 legacy 删除（review 已解锁；须分阶段，按报告 §B8 deletion readiness 的 DEAD vs STILL-CONSUMED ledger）**： - - **可删（DEAD，成单元同删）**：`datasource/paimon/*`（PaimonExternalCatalog/Factory、ExternalDatabase/Table、HMS/DLF/File/Rest 子类、 - SysExternalTable、MetaCache 等）、`systable/PaimonSysTable`、`metacache/paimon/*` + `ExternalMetaCacheMgr.paimon()/ENGINE_PAIMON`、 - `ShowPartitionsCommand`/`Env`/`ExternalCatalog.buildDbForInit`/`UserAuthentication`/`ExternalMetaCacheRouteResolver` 的死 legacy 分支+import。 - - **删除前置（硬）**：① 先把 `PaimonExternalCatalog` 的常量（`PAIMON_FILESYSTEM`/`PAIMON_HMS`）迁出到 metastore-props 模块（5 个 live 类 import 它）； - ② scrub 悬空 javadoc `{@link PaimonSysTable}`（`PluginDrivenSysTable:27`、`NativeSysTable:36`）否则 strict checkstyle/javadoc 挂； - ③ 保 load-bearing dispatch ordering（`ShowPartitionsCommand` PluginDriven 分支先于 legacy）。 - - **不可删（STILL-CONSUMED）**：`property/metastore/Paimon*MetaStoreProperties`+`PaimonPropertiesFactory`+`AbstractPaimonProperties`（cutover - Kerberos 装配 LIVE，R1）、`property/storage/{S3,OSS,COS,OBS,Minio}Properties`（跨连接器共享，R2）。**B8 scope 不含这两树。** - - 逐子树删 + 每批跑 fe-core 编译 + 连接器测 + regression-gated。与元存储子线 D-016 一致（那两包不碰）。 -2. **元存储子线收尾**（[`metastore-storage-refactor/`](./metastore-storage-refactor/)）：P2-T04（剩 paimon pom + gate； - ✅ `MetaStoreProviders` ServiceLoader 已改 2-arg 显式 loader `MetaStoreProvider.class.getClassLoader()`（`2612af5e88f`）—— - CI external **973270**（commit `13d3876`）实证:1-arg TCCL 在插件 child loader 下 `registered providers: []`，全 37 个 - paimon 家族测试 CREATE CATALOG 挂；单测单 classpath 不可区分 1-arg/2-arg（44/0 绿），真闸仍在 P2-T05 docker）→ P2-T05（docker - 5-flavor 真闸 + vended(REST/DLF) + Kerberos HMS + storage 等价，合并原 P1-T06；`enablePaimonTest=true`）。 -3. **D-057 re-scope**（第三轮报告 §D.3）：deferred `TablePartitionValues:162` prune-path sentinel residue **不影响 - paimon**（MVCC override 绕过）→ re-scope 到非-MVCC 插件连接器（maxcompute/es/jdbc）。 -4. **accepted-deviation 用户签字**（task-list「NOT in this fix scope」）：~10 MINOR + ~12 NIT + C-1 observability + - uncheckedFallbacks（REFRESH cache invalidation / partitions-TVF auth / split-plan RPC 在 `executeAuthenticated` 外 / - `PluginDrivenExternalCatalog:140` 吞 authenticator-wiring 异常）。逐条 accept-as-deviation 或转 fix。 +# 🔭 主线 backlog（按优先级） + +1. **P5-T29（B8）删 legacy + maven 依赖** —— 见上 §头条。**这是 P5 阶段最后一块主体工作。** +2. **P5-T30（B9）post-cutover 回归**：SHOW PARTITIONS + partitions TVF / DROP·CREATE DB·TABLE / no-ENGINE CREATE / edit-log replay / MTMV 增量刷 / sys-table / session-TZ 谓词不丢行。大部分是 live-e2e（B7 翻闸前置的硬门，随 #64446 合入即应已由用户跑过 5-flavor live-e2e；删 legacy 后须再跑一遍确认无回归）。可与 P5-T29 的 E 项验证合并。 +3. **元存储子线收尾**（[`metastore-storage-refactor/`](./metastore-storage-refactor/)）：P2-T04（剩 paimon pom + gate；`MetaStoreProviders` 已改 2-arg 显式 loader `2612af5e88f`）→ P2-T05（docker 5-flavor 真闸 + vended(REST/DLF) + Kerberos HMS + storage 等价，`enablePaimonTest=true`）。**与 P5-T29 的 C 项（STILL-CONSUMED metastore-props）领域重叠**——若 P5-T29 取方案 B，须与本子线协调。 +4. **accepted-deviation 用户签字残项**：P6 review 未转 fix 的剩余 MINOR/NIT 刻意偏离 + `PluginDrivenExternalCatalog:140` 吞 authenticator-wiring 异常 + uncheckedFallbacks（REFRESH cache invalidation / partitions-TVF auth / split-plan RPC 在 `executeAuthenticated` 外）。逐条记入 `deviations-log.md` accept-as-deviation（含用户签字）。可与 P5-T29 穿插。 +5. **D-057 re-scope**：deferred `TablePartitionValues:162` prune-path sentinel residue **不影响 paimon**（MVCC override 绕过）→ re-scope 到非-MVCC 插件连接器（maxcompute/es/jdbc）。 +6. **后续阶段**：P6 iceberg / P7 hive(+HMS) / P8 收尾（删 SPI_READY_TYPES、删 instanceof）——见 master plan §3.7–3.9。 --- # 📦 仓库 / 进度状态 -- **HEAD = `60ed665c4dc`**（FIX-B-R2-be schema-dict memo；前序 `9d687145a28` A1、`10284edbf88` B-MC2、`1935748d6c3` A2、`5fa47c27eb8` A3、`82b6de0de98` C4/R2/R3、`f652b40d210` R1-table、`44499f073e8` R3-residual、`e95128aed5b` C2 HDFS XML、`9967846ef64` C1 MinIO）。当前分支 **`catalog-spi-07-paimon`**（非 master）； - remote `master-catalog-spi-07-paimon`（= PR [#64445](https://github.com/apache/doris/pull/64445) head）仍在 `82b6de0de98`， - **本地领先：A3 起的 deviation fix commits 尚未 push** → 待本批 deviation fix 做完，session 收尾一次性 - force-with-lease push + PR 评论 `run buildall`（见 §Commit 须知 / memory `catalog-spi-07-paimon-branch-pr-workflow`）。 -- **主线（P0–P5）**：paimon connector SPI cutover + round-3 clean-room review 的 4 个 user-approved fix 全完成 - （FIX-1 `c376aba1264` rest-vended-uri / FIX-2 `2e845e88bf9` jni-file-format / FIX-3 `f08bc22b9bd` incr-scan-reset / - FIX-4 `f0210b51871` feconf-storage-parity）。详见 `task-list-P5-rereview3-fixes.md` + `reviews/P5-paimon-rereview3-2026-06-12.md`。 -- **元存储/storage 子线**（独立目录，本 session 推进）：storage 收口到 `fe-filesystem-api` typed（P1）+ 新建 - `fe-connector-metastore-{api,spi}` + `fe-kerberos`（P2-T01..T03，paimon 已 cutover 到共享 metastore SPI）+ - **fe-property 模块已物理删除**（P1-T07，0 消费者孤儿）。剩 P2-T04/T05（见 backlog）。**注**：fe-core - `datasource.property.{storage,metastore}` 两包仍在（子线 D-016 不碰；B8 才考虑删其 paimon-only 部分）。 -- ⚠️ `regression-test/conf/regression-conf.groovy` 仍 modified 未 commit 且含**明文 Aliyun key** → commit 前继续 - path-whitelist，**严禁 `git add -A`**；`regression-conf.groovy.bak` 同理排除。 -- 未 commit/未跟踪：scratch（`.audit-scratch/` `conf.cmy/` `META-INF/`）；`reviews/P5-paimon-rereview3-2026-06-12.md` - （第三轮 review 报告）；**`reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md`（本轮全路径 clean-room review 报告，502 行，本 session 产物）**。 - HANDOFF.md 本身已更新（review 完成态）。三者未跟踪——下次方便时 vet + path-whitelist commit 或保留本地。 +- **当前分支 = `branch-catalog-spi`**（开发主分支）。HEAD 近端：`38e7140ce56`（#64446 P5 迁移+翻闸）← `e9c5b3e70ce`（修编译）。 + P0–P5(迁移+翻闸) + P3 hybrid + P4 全部已合入本分支。 +- **P5 状态**：B0–B7 全完成并合入 #64446；**仅剩 B8（P5-T29 删 legacy）+ B9（P5-T30 回归）**。 +- **legacy 仍在 fe-core**（待 P5-T29 删）：`datasource/paimon/`(30) + `metacache/paimon/`(3) + `systable/PaimonSysTable`(1) + 8 处反向引用文件 + paimon maven 依赖(5)。STILL-CONSUMED `property/metastore/Paimon*`(7) **保留**。 +- ⚠️ `regression-test/conf/regression-conf.groovy` 若仍 modified 且含**明文 Aliyun key** → commit 前继续 path-whitelist，**严禁 `git add -A`**；`regression-conf.groovy.bak` 同理排除。 +- 未跟踪 scratch：`.audit-scratch/` `conf.cmy/` `META-INF/` `*.bak` 等——commit 前清，勿 add。 +- `reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md`（B8 readiness ledger 来源）若仍未跟踪，下次方便时 vet + commit 或保留本地。 ## 🗺️ 代码脚手架 -- **Plugin connector**：`fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/` - （`PaimonConnector` / `PaimonConnectorProvider` / 存储+HiveConf 装配 `PaimonCatalogFactory`[现 cutover 到 - `MetaStoreProviders.bind` + 薄 `assembleHiveConf`] / scan `PaimonScanPlanProvider` / @incr `PaimonIncrementalScanParams`）。 -- **共享 SPI / 叶子**：`fe/fe-connector/fe-connector-{api,spi}/` + `fe-connector-metastore-{api,spi}/`（metastore 解析器 + - `MetaStoreProvider` SPI/ServiceLoader）+ 顶层叶子 `fe/fe-kerberos/`（kerberos facts）+ `fe/fe-filesystem/`（typed - storage，含 `-hdfs` BE model）。 -- **fe-core 桥**：`fe/fe-core/.../connector/DefaultConnectorContext.java`、`.../datasource/PluginDriven*.java`、 - `.../fs/FileSystem{Factory,PluginManager}.java`；nereids scan-node 分发。 -- **Legacy 对照基准（＝ review 对照 + B8 删除目标）**：fe-core `.../datasource/paimon/`、 - `.../datasource/property/storage/` 下 `{OSS,COS,OBS,S3,Minio}Properties`、`.../property/metastore/HMSBaseProperties`。 -- **BE 消费端**：`be/src/format/table/`（`paimon_cpp_reader.cpp`、`paimon_reader.cpp`、`partition_column_filler.h`）。 +- **Plugin connector（终态归宿）**：`fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/` + （`PaimonConnector` / `PaimonConnectorProvider` / `PaimonCatalogFactory`[`MetaStoreProviders.bind` + 薄 `assembleHiveConf`] / `PaimonScanPlanProvider` / `PaimonConnectorMetadata` / `PaimonCatalogOps` seam / `PaimonTableResolver` / `PaimonSchemaBuilder` / `PaimonTypeMapping` / `PaimonIncrementalScanParams`）。 +- **fe-core 通用桥（保留）**：`connector/DefaultConnectorContext`、`datasource/PluginDriven*`（含 `PluginDrivenMvccExternalTable`/`PluginDrivenSysExternalTable`/`PluginDrivenSysTable`/`NativeSysTable`）、`fs/FileSystem{Factory,PluginManager}`、nereids scan-node 分发。 +- **B8 删除目标（legacy 对照基准）**：fe-core `datasource/paimon/`、`metacache/paimon/`、`systable/PaimonSysTable`。 +- **STILL-CONSUMED（B8 不碰）**：fe-core `datasource/property/metastore/Paimon*` + `AbstractPaimonProperties` + `PaimonPropertiesFactory`；`datasource/property/storage/{S3,OSS,COS,OBS,Minio}Properties`（跨连接器共享）。 +- **BE 消费端（不在 FE 范围）**：`be/src/format/table/`（`paimon_cpp_reader.cpp`、`paimon_reader.cpp`、`partition_column_filler.h`）。 ## ⚠️ Commit 须知（任何 `git add` 前必读） -- **硬前置**：scrub `regression-test/conf/regression-conf.groovy`（明文 key）+ 清 scratch（`.audit-scratch/` `conf.cmy/` - `META-INF/` `*.bak`）。**path-whitelist `git add`，严禁 `git add -A`。** -- 每个 fix 独立 commit；message = `fix: ` / `[Pn-Tnn] ` + 根因 + 解法 + 测试，末尾带 - `Co-Authored-By: Claude Opus 4.8 (1M context) `。fix commit 带其 design doc（repo 惯例）。 -- **收尾推送惯例**（见 memory `catalog-spi-07-paimon-branch-pr-workflow`）：push `catalog-spi-07-paimon`(ff) + - **force-with-lease** `master-catalog-spi-07-paimon`（PR #64445 head）+ 在 PR #64445 评论 `run buildall`。⚠️ 两分支 - 历史曾发散；force 前先 fetch 对比、用 `--force-with-lease`。⚠️ remote URL 明文嵌 GitHub PAT（`git remote -v` 会打印）。 +- **硬前置**：scrub `regression-test/conf/regression-conf.groovy`（明文 key）+ 清 scratch（`.audit-scratch/` `conf.cmy/` `META-INF/` `*.bak`）。**path-whitelist `git add`，严禁 `git add -A`。** +- 每个 fix/删除批独立 commit；message = `[P5-T29] ` + 根因 + 解法 + 测试，末尾带 + `Co-Authored-By: Claude Opus 4.8 (1M context) `。删除 PR 带其 design doc（repo 惯例，参 P4 #64300 的 `P4-batchD-maxcompute-removal-design.md`）。 +- **PR 流程**：P5-T29 在 `branch-catalog-spi` 上做（或开 feature 分支 off 它），走**新 PR**（mirror P4 #64300，base = `branch-catalog-spi`）。 + ⚠️ 历史的 `catalog-spi-07-paimon` 分支 + PR #64445 force-push 流程**已作废**（那条线已并入 #64446）。memory `catalog-spi-07-paimon-branch-pr-workflow` 据此过时，勿再 force-push 那两个分支。 ## ⚙️ 操作须知（复用） -- maven 绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -Dmaven.build.cache.enabled=false - -DfailIfNoTests=false`；验证读 surefire XML + `BUILD SUCCESS`（memory `doris-build-verify-gotchas`）。**漏 `-am` → - `could not resolve … ${revision}` 假错**。paimon 模块需 `-am package -Dassembly.skipAssembly=true`（shade jar 携带 - HiveConf）。**checkstyle 在 `validate` phase（编译前）跑**。 +- maven 绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -Dmaven.build.cache.enabled=false -DfailIfNoTests=false`；验证读 surefire XML + `BUILD SUCCESS`（memory `doris-build-verify-gotchas`）。**漏 `-am` → `could not resolve … ${revision}` 假错**。paimon 连接器模块需 `-am package -Dassembly.skipAssembly=true`（shade jar 携带 HiveConf）。**checkstyle 在 `validate` phase（编译前）跑**。 - 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`（仅允许 `org.apache.doris.{thrift,connector,extension,filesystem}`）。 - cwd 跨 Bash 调用持久，`cd` 破相对路径 → 一律绝对路径。 -- 测试 harness：`PaimonCatalogFactoryTest`（纯 Map→Configuration/HiveConf）/`PaimonScanPlanProviderTest`(real-table - `FileSystemCatalog`)/`PaimonIncrementalScanParamsTest`/`RecordingConnectorContext`/`RecordingPaimonCatalogOps`/ - `FakePaimonTable`（`.copy` 是 no-op recorder，reset/merge fail-before 须 real table）/ metastore-spi 的 - `*MetaStorePropertiesTest` / `DefaultConnectorContextNormalizeUriTest`(fe-core)。live-e2e CI-gated - （`enablePaimonTest` 默认 false）→ 注明 gated，勿谎称跑过。 +- 测试 harness：`PaimonCatalogFactoryTest` / `PaimonScanPlanProviderTest`(real-table `FileSystemCatalog`) / `PaimonIncrementalScanParamsTest` / `RecordingConnectorContext` / `RecordingPaimonCatalogOps` / `FakePaimonTable`（`.copy` no-op recorder）/ metastore-spi 的 `*MetaStorePropertiesTest`。live-e2e CI-gated（`enablePaimonTest` 默认 false）→ 注明 gated，勿谎称跑过。 ## 🧠 给下一个 agent 的 meta -- **本轮是 review、不是改码**：先出 review 报告，发现项各自另起 fix task；**review 须 clean-room、零历史先验**（见上「关键约束」）。 -- **review 必须先于 B8**（legacy ＝ 对照基线）；B8 scope 须经 review dim-6 确认真 dead（别误删仍被 hive/hudi/iceberg 消费的类）。 -- **改 handle/分区/scan/storage/auth 流必 grep 全调用方 + 确认实际实例类（base vs MVCC 子类）**；storage/auth 装配注意 raw - `hadoop.*`/`fs.*` passthrough 跑最后会 clobber 之前 authoritative 设置（FIX-4 4d/4e 亲证）。 -- **design red-team（写码前）+ impl verification（写码后）两道**历史证有效（修复阶段照用，但 review 阶段保持 clean-room）。 +- **P5-T29 的 D 项（maven）须先与用户对齐 scope（方案 A vs B）再动手**——这是 fe-core 能否完全 paimon-free 的分叉，影响 metastore 子线。 +- **删除前必 grep 全调用方 + 确认实际实例类（base vs MVCC 子类 / DEAD vs STILL-CONSUMED）**；逐子树删 + 每批跑 fe-core 编译，别一次性大删。 +- **改 storage/auth 装配注意** raw `hadoop.*`/`fs.*` passthrough 跑最后会 clobber 之前 authoritative 设置（FIX-4 4d/4e 亲证）——B8 不应碰这些，但若动 metastore-props（方案 B）须警惕。 - **元存储子线**细节不在本文件——读 `metastore-storage-refactor/HANDOFF.md`。 diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index a6a9b6f842ca25..aa860a6d28c0cb 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-06-10** | 当前阶段：**P4 maxcompute 完成 ✅（已合入），P5 paimon B0–B4 已落地（测基建/flavor/normal-read/DDL/sys-tables+MVCC；下一 = B5 MTMV 桥）**——P4 full-adopter 迁移 + live 翻闸 + legacy 删除全部完成并合入 `branch-catalog-spi`：**#64253**（T01–T06 连接器全适配 + `CatalogFactory.SPI_READY_TYPES += "max_compute"`）+ **#64300**（T07–T09 删 20 fe-core 文件 + 清反向引用 + MCUtils 下沉 be-java-extensions，fe-core 依赖树**彻底无 odps**，HEAD `e96037cf6aa`）；upstream PR **#64119**（MaxCompute 连接校验）功能已迁连接器 SPI 并随 #64300 合入。前序 P0/P1/P2（#63582/#63641/#64096）+ P3 hybrid（#64143）均已合入。**P5 paimon B0–B4 已落地 2026-06-10**（recon+设计 2026-06-09；B0 测基建 / B1 flavor 装配 / B2 normal-read / B3 DDL metadata / B4 sys-tables E7 + MVCC E5；签字 D-037/D-038/D7/**D-039**(E7=live SysTable 机制非 RFC §10)；连接器 124 绿 + fe-core PluginDriven*Test 100 绿、checkstyle/import-gate 0、**未提交**；下一 = B5 MTMV 桥，翻闸 B7 gated on B5+live-e2e）。| 项目总进度：**~33%**（按 §一 进度条加权：P0+P1+P2+P4 满 + P3 hybrid 45% + P5 设计 ~5%，约 8.0/25 周） +> 最后更新：**2026-06-20** | 当前阶段：**P5 paimon 迁移 + 翻闸已合入 ✅，仅剩 P5-T29 删 legacy + maven 依赖（B8）+ P5-T30 回归（B9）**——P5 全量（B0–B7 = 测基建/flavor/normal-read/DDL/sys-tables+MVCC/MTMV桥/时间旅行/**翻闸** + P6 全路径 clean-room review 的全部 deviation fix）squash-合入 `branch-catalog-spi`：**#64446 / `38e7140ce56`**（"migrate to catalog SPI + cutover"，+ `e9c5b3e70ce` 修编译）。paimon 现已在 `SPI_READY_TYPES`，FE 走 SPI 路径。**下一 = P5-T29**（删 fe-core `datasource/paimon/`(30)+`metacache/paimon/`(3)+`systable/PaimonSysTable`+8 处反向引用+paimon maven 依赖；**硬前置**=迁出 `PaimonExternalCatalog` 常量；**STILL-CONSUMED `property/metastore/Paimon*`(7) 保留**；详见 [tasks/P5 §P5-T29 执行计划](./tasks/P5-paimon-migration.md)）。前序 P0/P1/P2（#63582/#63641/#64096）+ P3 hybrid（#64143）+ P4（#64253/#64300）均已合入。| 项目总进度：**~42%**（按 §一 进度条加权：P0+P1+P2+P4 满 + P3 hybrid 45% + P5 ~95%，约 10.75/25 周） > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -14,12 +14,12 @@ | **P2** | trino-connector 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 已合入 `branch-catalog-spi`（#64096，squash `0793f032662`；T12 回归推迟 DV-003）| [tasks/P2](./tasks/P2-trino-connector-migration.md) | | P3 | hudi 迁移 | 2 周 | ▰▰▰▰▰▱▱▱▱▱ 45% | ✅ hybrid（D-019）批 A–D 已合入 `branch-catalog-spi`（**#64143** squash `5c240dc7a34`）；批 E（live cutover）并入 P7 | [tasks/P3](./tasks/P3-hudi-migration.md) | | **P4** | maxcompute 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（**#64253** T01–T06 适配+翻闸 + **#64300** T07–T09 删 legacy/odps-free；含 #64119 校验迁移）| [tasks/P4](./tasks/P4-maxcompute-migration.md) | -| **P5** | paimon 迁移 | 3 周 | ▰▰▰▰▱▱▱▱▱▱ 45% | 🚧 **B0–B4 已落地**（测基建/flavor/normal-read/DDL/sys-tables+MVCC；D-037/D-038/D7/D-039 签字，未提交）；下一 = B5 MTMV 桥；翻闸 B7 gated on B5 + live-e2e | [tasks/P5](./tasks/P5-paimon-migration.md) + [recon](./research/p5-paimon-migration-recon.md) | +| **P5** | paimon 迁移 | 3 周 | ▰▰▰▰▰▰▰▰▰▱ 95% | 🚧 **迁移 + 翻闸已合入 `branch-catalog-spi`（#64446 `38e7140ce56`）**；仅剩 **B8 = P5-T29 删 legacy + maven 依赖** + B9 回归 | [tasks/P5](./tasks/P5-paimon-migration.md) + [recon](./research/p5-paimon-migration-recon.md) | | P6 | iceberg 迁移 | 5 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P7 | hive (+HMS) 迁移 | 6 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P8 | 收尾清理 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | -**全局进度：~34%**（25 周计划中已完成约 8.5 周：P0+P1+P2+P4 满 + P3 hybrid 45% + P5 45%；按 §一 进度条加权） +**全局进度：~42%**（25 周计划中已完成约 10.75 周：P0+P1+P2+P4 满 + P3 hybrid 45% + P5 ~95%；按 §一 进度条加权） --- @@ -34,7 +34,7 @@ | trino-connector | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/trino-connector.md) | | hudi | 🟡（D-005 区分符 + D-020 模型 dispatch 已设计；实现批 E）| 🟨 55%（读路径 dormant + 批 C 测试基线）| ❌（gate 关）| ❌ | 0/0（寄生 hms）| **25%** | [详情](./connectors/hudi.md) | | maxcompute | ✅ | ✅ 100% | ✅ **已合入 #64253** | ✅ **#64300 已删** | ✅ 0/0 | **100%** | [详情](./connectors/maxcompute.md) | -| paimon | ✅（D-037/D-038/D-039）| 🟨 70%（read+DDL+sys-tables+MVCC连接器侧；MTMV桥 B5 待）| ❌（gate 关）| ❌ | 0/10 | **45%** | [详情](./connectors/paimon.md) | +| paimon | ✅ | ✅ 100%（迁移+翻闸已合入 #64446）| ✅ **已入 SPI_READY_TYPES** | ⏳ **P5-T29 待删** | 🟡（热区翻闸已清；infra 死引用 8 处待 T29）| **95%** | [详情](./connectors/paimon.md) | | iceberg | 🟡 | 🟥 10% | ❌ | ❌ | 0/19 | **5%** | [详情](./connectors/iceberg.md) | | hive (+hms) | 🟡 | 🟥 20% | ❌ | ❌ | 0/31 | **10%** | [详情](./connectors/hive.md) | @@ -44,14 +44,15 @@ > 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 -### P5 — paimon 迁移（🚧 B0–B4 已落地 2026-06-10，D-037/D-038/D7/D-039 已签字；下一 = B5 MTMV 桥） +### P5 — paimon 迁移（✅ 迁移+翻闸已合入 #64446；🎯 下一 = **P5-T29 删 legacy + maven 依赖**） -> 策略 = **full adopter + 翻闸**（复用 P4 样板，非 P3 hybrid）。recon `research/p5-paimon-migration-recon.md` + 设计 `tasks/P5-paimon-migration.md`（30 TODO / B0–B9 批 + old→new 映射 + 批次依赖图）。覆盖 5 功能区：普通读/系统表/procedure/DDL/mtmv。 +> 策略 = **full adopter + 翻闸**（复用 P4 样板）。B0–B7 全完成并 squash-合入 `branch-catalog-spi`（**#64446 / `38e7140ce56`** + `e9c5b3e70ce` 修编译）：测基建/flavor/normal-read/DDL/sys-tables+MVCC(E7/E5)/MTMV桥(E10)/时间旅行(AS-OF/tag/branch/@incr)/**翻闸** + P6 全路径 clean-room review 的全部 deviation fix（C1 MinIO/C2 HDFS XML/R1-table/R3-residual/C4+R2+R3-catalog/A1/A2/A3/B-MC2/B-R2-be）。paimon 已在 `SPI_READY_TYPES`。 > -> **已签字决策**：**D-037**=flavor(hms/filesystem/dlf/rest/jdbc) 走单 Catalog + `createCatalog` flavor switch（MC 一致，**非** backend 模块——5 个 backend 模块是空壳）；**D-038**=MTMV/MVCC 桥 P5 内实现（fe-core `PaimonPluginDrivenExternalTable`），翻闸(B7) gated on 它(B5)，禁静默读 latest。 -> **校正先验**（recon + 对抗复审证伪）：① 「6 flavor 工厂已重组」假（backend 模块空壳，连接器走单 Catalog stub）；② 「FE 分发全缺」假（DROP/CREATE·DROP DB/SHOW PARTITIONS/TVF 已部分预接，残留=连接器 `listPartitions*`）；③ 「Base64 blocker」假（BE 有 STD fallback，真风险=pin paimon-core 三方版本对齐）。procedure 区=**零可迁 doc-only**。 -> **关键 SPI 缺口**：E7 sys-table hook（greenfield 须新增 default-no-op）、E10 MTMV（无面，经 fe-core 子类桥）、E5 MVCC（首个真消费者，须 wire）、E6 vended（REST flavor 需，可延后）；删 fe-core 重复 `PaimonPredicateConverter`（**P1-T02 推迟项**）+ 清 10 处反向 `instanceof`。 -> **前置风险**：R-004（classloader SDK 单例）、R-007（FE/BE 共享 jar）、R-012（snapshotId 类型）。**最高 correctness 风险**：MTMV 单-pin 不变式 + `lastFileCreationTime()` 跨 flavor 可靠性（须 live 验）。**关联决策**：D-037、D-038、D-005、D-006。 +> **🎯 P5-T29（B8）= 本阶段最后一块主体工作**：删 fe-core `datasource/paimon/`(30) + `metacache/paimon/`(3) + `systable/PaimonSysTable`(1) + 清 8 处反向引用死分支 + **删 paimon maven 依赖**。 +> **硬前置**：迁出 `PaimonExternalCatalog.PAIMON_FILESYSTEM/_HMS` 常量（被 5 个 STILL-CONSUMED metastore-props 引用）；scrub 悬空 javadoc `{@link PaimonSysTable}`；保 dispatch ordering；`CreateTableInfo.ENGINE_PAIMON` 是 LIVE 保留。 +> **STILL-CONSUMED 不删**：`property/metastore/Paimon*`(7，cutover Kerberos 装配 LIVE，P6 R1)；`property/storage/*Properties`（跨连接器共享，P6 R2）。 +> **⚠️ maven 核心冲突**：STILL-CONSUMED metastore-props 直接 import paimon SDK → **fe-core 不可能像 P4 完全 paimon-free**（除非方案 B 连带迁出 metastore-props，越界 metastore 子线）。须先定方案 A（推荐，部分删）vs B。 +> **样板 = P4 #64300**；scope ledger + checklist 详见 [tasks/P5 §P5-T29 执行计划](./tasks/P5-paimon-migration.md)。**风险**：R-004（classloader SDK 单例）、R-007（FE/BE 共享 jar）→ 删后验 paimon-core FE classpath 恰一份。 ### P4 — maxcompute 迁移（✅ 已完成并合入：**#64253** T01–T06 适配+翻闸 + **#64300** T07–T09 删 legacy/odps-free；含 #64119 校验迁移） @@ -149,6 +150,7 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-06-20（阶段里程碑 · P5 迁移+翻闸合入 + 文档对账）** ✅ **P5 paimon B0–B7 全完成并 squash-合入 `branch-catalog-spi`** —— **PR #64446 / `38e7140ce56`**（"[refactor](catalog) P5 paimon: migrate to catalog SPI + cutover"）+ `e9c5b3e70ce`（修编译）。涵盖 B5 MTMV 桥（通用 `PluginDrivenMvccExternalTable`，D-040/041/042）、B5b 时间旅行全 parity（AS-OF/tag/branch/@incr，D-043/044）、B6 procedure no-op、**B7 翻闸**（入 `SPI_READY_TYPES` + GSON 原子 compat + D-045/046/047 restore SHOW PARTITIONS/SHOW CREATE）、**P6 全路径 clean-room review**（报告 `reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md`：2 BLOCKER=B8 删除护栏、2 MAJOR=C1 MinIO/C2 HDFS XML 已修，余 parity）+ 全部 deviation fix（C1/C2/R1-table/R3-residual/C4+R2+R3-catalog/A1/A2/A3/B-MC2/B-R2-be）。**仅剩 P5-T29（B8 删 legacy + maven 依赖）+ P5-T30（B9 回归）**。本 session = 对账 stale 跟踪文档（PROGRESS 停 B4、tasks/P5 停 B5b、HANDOFF 停历史工作分支）→ 全部刷到「迁移+翻闸已合入、下一步 P5-T29」状态，0 产线代码；P5-T29 scope ledger（DEAD/硬前置/STILL-CONSUMED/maven 方案 A/B）已 firsthand 核实并写入 [tasks/P5 §P5-T29 执行计划](./tasks/P5-paimon-migration.md)。**下一 session = P5-T29**（建议先 AskUserQuestion 定 maven scope A/B）。 - **2026-06-10（实现里程碑 · P5 B0–B4）** ✅ **P5 paimon B0–B4 已落地**（连接器+fe-core，**未提交**，用户控时机）：B0 测基建+parity baseline / B1 flavor 装配(全 5 flavor) / B2 normal-read / B3 DDL metadata / **B4 sys-tables E7 + MVCC E5（本 session，T16-T20）**。B4 = subagent-driven（understand workflow 纠偏 2 处 → 用户签 **D-039**「E7 复用 live `SysTableResolver` 机制，非 RFC §10」[DV-023]；T20 MVCC inert until B5）+ 5 dispatch（implement→双审→fix-loop）+ 3-lens final holistic（PARITY/SCOPE READY + 1 ADVERSARIAL BLOCKER「`PluginDrivenScanNode.create` 绕 seam 丢 forceJni→binlog/audit_log 静默错行」**已修**）。另核出并修 B2 遗留缺陷 [DV-024]（普通 paimon plugin 表 BE 描述符 SCHEMA_TABLE→应 HIVE_TABLE）。**验证**：连接器 124/0/0/1 绿、fe-core PluginDriven*Test 100 绿、checkstyle/import-gate 0、无 cutover/B5 泄漏、唯一 fe-connector-api 改动=T16 两 default no-op。**下一 = B5 MTMV 桥**（接活 E5：`PluginDrivenExternalTable`→MvccTable + `beginQuerySnapshot` 调用 + `ConnectorMvccSnapshotAdapter` 构造）。 - **2026-06-09（设计里程碑 · P5 kickoff）** ✅ **P5 paimon recon + 设计完成**（0 产线代码）：14-agent code-grounded recon + cross-cut 对抗复审，产 [recon](./research/p5-paimon-migration-recon.md)（5 功能区旧实现 + E1–E10 SPI 状态 + 跨切面风险 + MC 一致性 11 约定）+ [设计 doc](./tasks/P5-paimon-migration.md)（old→new 映射 + 30 TODO/B0–B9 + 验收 + 批次依赖图）。**用户签字 D-037**（flavor=单 Catalog + `createCatalog` switch，**非** backend 模块）/ **D-038**（MTMV/MVCC 桥 P5 内实现，翻闸 gated on B5，禁静默读 latest）。**证伪 3 先验**：backend 模块空壳（连接器走单 Catalog stub）、FE 分发部分已预接（残留=连接器 listPartitions）、Base64 非 blocker（BE 有 STD fallback）。procedure 区=零可迁 doc-only（expire_snapshots=iceberg、CALL migrate_table=Spark 两假阳性）。**下一 = B0 测试基建 + parity baseline 起分批实现**。 - **2026-06-09（阶段里程碑 · P4 完成）** ✅ **P4 maxcompute 迁移全部完成并合入 `branch-catalog-spi`** —— **#64253**（T01–T06 连接器 full 适配 + live 翻闸 `SPI_READY_TYPES += "max_compute"`）+ **#64300**（T07–T09 删 20 fe-core legacy 文件 + 清反向引用 + MCUtils 下沉 be-java-extensions，`fe-core dependency:tree | grep odps`=∅，HEAD `e96037cf6aa`）。upstream PR **#64119**（MaxCompute 连接校验）功能已迁连接器 SPI（`validateMaxComputeConnection`/`checkOperationSupported`，连接器 UT 101/0/0/1）并随 #64300 squash 合入（`git log -S` 证）。fe-core **彻底无 odps**（代码 + 依赖树）。本 session = 交接文档同步（PROGRESS + HANDOFF 第 19 次），0 产线代码；**下一 session = P5 paimon 迁移 kickoff**（recon + 设计 + 批次计划，复用 P4 full-adopter 写 SPI 样板）。 @@ -214,8 +216,8 @@ | 类型 | 总数 | 最新条目 | 文档 | |---|---|---|---| -| **决策**（D-NNN） | 39 | D-039（P5-D8 paimon B4 E7=复用 live SysTable 机制，非 RFC §10）；D-038（P5-D2 MTMV/MVCC 桥 P5 内实现）；D-037（P5-D1 flavor=单 Catalog）| [decisions-log.md](./decisions-log.md) | -| **偏差**（DV-NNN） | 24 | DV-024（P5-B4 修 B2 遗留：普通 paimon plugin 表 BE 描述符 SCHEMA_TABLE→HIVE_TABLE）；DV-023（RFC §10 E7 设计被 P5-B4 取代）；DV-022（P4-T09 fe-common 去 odps）| [deviations-log.md](./deviations-log.md) | +| **决策**（D-NNN） | 57 | D-057（最新，见 log：prune-path sentinel residue re-scope 非-MVCC 连接器）；D-045/046/047（P5-B7 restore SHOW PARTITIONS 5 列 / SHOW CREATE LOCATION+PROPERTIES / Hybrid SPI）；D-040–044（P5-B5/B5b MTMV+时间旅行）| [decisions-log.md](./decisions-log.md) | +| **偏差**（DV-NNN） | 37 | DV-036/037（P6-C2 HDFS XML：DLF / disable-cache）；DV-024（P5-B4 BE 描述符 SCHEMA_TABLE→HIVE_TABLE）；DV-023（RFC §10 E7 被 P5-B4 取代）| [deviations-log.md](./deviations-log.md) | | **风险**（R-NNN） | 14 | R-014（thrift sink 选择灵活性） | [risks.md](./risks.md) | --- @@ -224,9 +226,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：**P5 paimon B4（sys-tables E7 + MVCC E5，T16-T20，连接器+fe-core，未提交）** —— understand workflow（6-agent）纠偏 2 处 → 用户签 **D-039**（E7 复用 live SysTable 机制非 RFC §10）+ T20 留 B4（inert until B5）；subagent-driven 5 dispatch（implement→双审→fix-loop）+ 3-lens final holistic（PARITY/SCOPE READY + 1 ADVERSARIAL BLOCKER `PluginDrivenScanNode.create` 绕 seam 丢 forceJni **已修**）；另修 B2 遗留缺陷 [DV-024]。验证：连接器 124 绿、fe-core 100 绿、checkstyle/import-gate 0。同步 decisions-log(+D-039) + deviations-log(+DV-023/DV-024) + RFC §10 脚注 + tasks/P5 + 本 PROGRESS + connectors/paimon + HANDOFF（覆盖）+ auto-memory。 -- **下一个 session 应做**：**B5 MTMV 桥**（gated on B4，现满足）—— T21 GAP-LISTPART-AT-SNAPSHOT / T22 fe-core `PaimonPluginDrivenExternalTable` implements MTMVRelatedTableIf+MTMVBaseTableIf+MvccTable + `loadSnapshot` / T23 子类 MTMV 方法 / T24 rehome `PaimonMvccSnapshot` / T25 isPartitionInvalid parity。**关键**：B5 须把 B4 inert 的 E5 接活（调 `beginQuerySnapshot` + 构造 `ConnectorMvccSnapshotAdapter` + 接 scan-params/snapshot 到 `PluginDrivenScanNode` 令 T19 sys-guard 生效）。**B6**（procedure doc no-op）可穿插。继 B7 翻闸（gated on B5+live-e2e）→ B8 删 legacy → B9 回归。详见 [tasks/P5](./tasks/P5-paimon-migration.md) 批次依赖图。 -- **是否需要 handoff**：**是**——本场已**覆盖** rewrite [HANDOFF.md](./HANDOFF.md)（P5 B4 完成 + D-039 + 下一步 B5）。 +- **本 session 已完成**：**文档对账（0 产线代码）** —— 发现 P5 迁移+翻闸已随 #64446 合入 `branch-catalog-spi`，但跟踪文档严重 stale（PROGRESS 停 B4、tasks/P5 停 B5b、HANDOFF 停历史工作分支 `catalog-spi-07-paimon`）。在 `branch-catalog-spi` firsthand 核实 P5-T29 删除 scope（DEAD `datasource/paimon/`(30)+`metacache/paimon/`(3)+`systable/PaimonSysTable`；硬前置=迁出 `PaimonExternalCatalog` 常量；STILL-CONSUMED `property/metastore/Paimon*`(7) 保留；maven 5 依赖 + 「fe-core 不可完全 paimon-free」冲突）。同步刷新 HANDOFF（覆盖）+ 本 PROGRESS + tasks/P5（含新 §P5-T29 执行计划 checklist）+ connectors/paimon。 +- **下一个 session 应做**：**P5-T29（B8 删 legacy + maven 依赖）**——见 [tasks/P5 §P5-T29 执行计划](./tasks/P5-paimon-migration.md) 的 A–E checklist。**建议先 AskUserQuestion 定 maven scope 方案 A（推荐，部分删，fe-core 保 paimon-core/common）vs B（连带迁出 metastore-props，完全 paimon-free，越界 metastore 子线）**。样板 = P4 #64300。继 B9 回归（可与 T29 §E 验证合并）。 +- **是否需要 handoff**：**是**——本场已**覆盖** rewrite [HANDOFF.md](./HANDOFF.md)（P5 迁移+翻闸合入 #64446 + 下一步 P5-T29 scope ledger）。 - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） --- diff --git a/plan-doc/connectors/paimon.md b/plan-doc/connectors/paimon.md index 181e2903737b61..f9f9c2a23b1e6c 100644 --- a/plan-doc/connectors/paimon.md +++ b/plan-doc/connectors/paimon.md @@ -10,29 +10,30 @@ | **fe-connector 模块** | `fe/fe-connector/fe-connector-paimon/` | | **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/` | | **共享依赖** | `fe-connector-hms`（paimon-HMS-flavor 用） | -| **计划迁移阶段** | **P5**（B0–B4 已落地 2026-06-10，未提交；下一 = B5 MTMV 桥）| -| **当前状态** | 🚧 B0–B4 完成（测基建/flavor/normal-read/DDL/sys-tables+MVCC 连接器侧）；D-037/D-038/D7/D-039 签字；B5 MTMV 桥 待 | -| **完成度** | 70%（连接器侧 read+DDL+sys-tables(E7)+MVCC(E5) 全实现 + fe-core 通用 sys 机制；MTMV 桥(B5)+翻闸(B7)+删 legacy(B8)+回归(B9) 待）| +| **计划迁移阶段** | **P5**（B0–B7 迁移+翻闸已合入 `branch-catalog-spi` #64446 `38e7140ce56`；下一 = **P5-T29 删 legacy**）| +| **当前状态** | ✅ 迁移 + 翻闸已合入（paimon 入 `SPI_READY_TYPES`，FE 走 SPI 路径）；仅剩 **B8 = P5-T29 删 fe-core legacy + maven 依赖** + B9 回归 | +| **完成度** | 95%（B0–B7 全实现并合入：read/DDL/sys-tables(E7)/MVCC(E5)/MTMV桥(E10)/时间旅行/翻闸 + P6 review deviation fix；剩删 legacy(B8/P5-T29)+回归(B9)）| | **主 owner** | @morningman / TBD | --- ## 迁移 Playbook 进度 +> 全部已合入 #64446，除步骤 13（= P5-T29）。 | 步骤 | 状态 | 备注 | |---|---|---| -| 1 | 🟡 | fe-core 22 个顶层 + `source/`（5 个）+ `profile/`（2 个）| -| 2 | 🟡 | fe-connector 10 个文件，scan/predicate/handle 完整 | -| 3 | ⏳ | 反向 instanceof：10 处 | -| 4 | 🟡 | ConnectorMetadata 仅 read 实现；flavor 装配=单 Catalog + `createCatalog` flavor switch（D-037，**非** backend 模块——5 个 `fe-connector-paimon-backend-*` 是空壳）| -| 5 | ⏳ | | +| 1 | ✅ | fe-core legacy 已盘点（`datasource/paimon/` 30 + `metacache/paimon/` 3 + `systable/PaimonSysTable`）| +| 2 | ✅ | fe-connector 全功能完整（scan/predicate/handle/DDL/sys-table/MVCC/MTMV/时间旅行）| +| 3 | ✅ | 反向 instanceof 已盘点（热区 + infra 死引用）| +| 4 | ✅ | ConnectorMetadata 全实现；flavor 装配=单 Catalog + `createCatalog` flavor switch（D-037，**非** backend 模块——5 个 `fe-connector-paimon-backend-*` 是空壳）| +| 5 | ✅ | validateProperties + preCreateValidation 全 flavor | | 6 | ✅ | META-INF/services 已注册 | -| 7 | ⏳ | | -| 8-9 | ⏳ | | -| 10 | ⏳ | 清理 10 处反向 instanceof | -| 11 | ⏳ | PhysicalPlanTranslator 删 `PAIMON_EXTERNAL_TABLE` 分支 | -| 12 | ⏳ | 0 个测试 | -| 13 | ⏳ | 删 `datasource/paimon/` | +| 7 | ✅ | `SPI_READY_TYPES += "paimon"`（翻闸已合入 #64446）| +| 8-9 | ✅ | GSON 原子转 `registerCompatibleSubtype` + db/table compat | +| 10 | 🟡 | 热区 instanceof 已清（翻闸）；**infra 死引用 8 处待 P5-T29** | +| 11 | ✅ | PhysicalPlanTranslator 删 `PAIMON` 分支（翻闸已合入）| +| 12 | ✅ | 连接器 UT ~300+ 绿 + fe-core PluginDriven* 测 | +| 13 | ⏳ | **删 `datasource/paimon/` = P5-T29（下一 session）** | --- @@ -44,23 +45,23 @@ | E2 Procedures | ❌ 不需要 | **零可迁**：fe-core 无 paimon procedure（expire_snapshots=iceberg、CALL migrate_table=Spark，皆非 paimon）| doc-only no-op | | E3 MetaInvalidator | 🟡 | paimon-HMS-flavor 需要 | 复用 `fe-connector-hms` | | E4 Transactions | ✅ 需要 | | -| E5 MvccSnapshot | ✅ 需要 | **B4 连接器侧已实现**（`beginQuerySnapshot/getSnapshotAt/getSnapshotById` + caps；**inert until B5** wires fe-core MvccTable 消费方）| 首个 E5 消费者 | -| E6 VendedCredentials | ✅ 需要 | `PaimonVendedCredentialsProvider` 待迁 | | -| E7 SysTables | ✅ 需要 | **B4 已实现**（D-039：复用 live `SysTableResolver` 机制，非 RFC §10 [DV-023]）：连接器 `listSupportedSysTables`+`getSysTableHandle`；fe-core 通用 `PluginDrivenSysExternalTable`+`PluginDrivenSysTable`（报 PLUGIN_EXTERNAL_TABLE）；forceJni binlog/audit_log；`buildTableDescriptor`→HIVE_TABLE | greenfield SPI，未来 iceberg/hudi 复用 | +| E5 MvccSnapshot | ✅ 需要 | ✅ **已合入 #64446**（B5 wire 通用 `PluginDrivenMvccExternalTable`→MvccTable 消费 `beginQuerySnapshot`）| 首个 E5 消费者 | +| E6 VendedCredentials | ✅ 需要 | ✅ 已迁（REST flavor）| | +| E7 SysTables | ✅ 需要 | ✅ **已合入**（D-039：复用 live `SysTableResolver`，非 RFC §10 [DV-023]）：连接器 `listSupportedSysTables`+`getSysTableHandle`；fe-core 通用 `PluginDrivenSysExternalTable`+`PluginDrivenSysTable`（报 PLUGIN_EXTERNAL_TABLE）；forceJni binlog/audit_log；`buildTableDescriptor`→HIVE_TABLE | greenfield SPI，未来 iceberg/hudi 复用 | | E8 ColumnStatistics | 🟡 | snapshot summary 已含部分 | 可选 | | E9 Delete/Merge sink | 🟡 | merge-on-read 路径 | | -| E10 listPartitions | ✅ 需要 | **B2 连接器侧已实现**（`listPartitionNames/listPartitions/listPartitionValues`）；FE 消费 + `partition_columns` key 翻 = B5 前置 | | -| **MTMV（无 E 号）** | ✅ 需要 | **SPI 完全无面（须新增 + fe-core `PaimonPluginDrivenExternalTable` 桥）**；paimon 是唯一带 MTMV 的 adopter | D-038（P5 内实现）| +| E10 listPartitions | ✅ 需要 | ✅ **已合入**（连接器 `listPartitionNames/listPartitions/listPartitionValues` + FE 消费 + `partition_columns` key 翻，B5）| | +| **MTMV（无 E 号）** | ✅ 需要 | ✅ **已合入 #64446**：通用 **`PluginDrivenMvccExternalTable`**（capability-selected，源无关，D-042）+ 时间旅行全 parity（AS-OF/tag/branch/@incr，D-043/044）| D-038（P5 内实现）| --- ## 已知特殊性 - **flavor 装配（D-037=单 Catalog）**：6 flavor（hms/filesystem/dlf/rest/jdbc + base）经 `PaimonConnector.createCatalog` 内 flavor switch on `paimon.catalog.type`（MC 一致，拷常量/conf/**每-flavor authenticator** 入模块）。⚠️ 5 个 `fe-connector-paimon-backend-*` 模块只是**空壳**（gitignore `.flattened-pom.xml`，零 src），**不采用**其 backend-SPI 设计。 -- **MTMV（D-038）**：SPI 无 MTMV 面（E10/MTMV 缺），`PluginDrivenExternalTable` 不实现任何 MTMV 接口 → 翻闸前须落 fe-core `PaimonPluginDrivenExternalTable` 桥（否则静默回归）；paimon 是**首个真消费 E5(MVCC)/E6(vended)/E7(sys-table)** 的 adopter，MC 无先例。 -- **重复类 `PaimonPredicateConverter`**（fe-core `source/:43` vs 连接器 `:57`）翻闸时删 fe-core 版；连接器版有 session-TZ bug（固定 UTC `:284`）须修。 +- **MTMV（D-038）**：✅ 已合入 #64446——翻闸落通用 **`PluginDrivenMvccExternalTable`**（capability-selected，**源无关**，D-042，非 paimon 专类；可复用 iceberg/hudi）implements MTMVRelatedTableIf+MTMVBaseTableIf+MvccTable；paimon 是**首个真消费 E5(MVCC)/E6(vended)/E7(sys-table)** 的 adopter，MC 无先例。 +- **重复类 `PaimonPredicateConverter`**（fe-core `source/PaimonPredicateConverter` vs 连接器版）：连接器版 TZ 已 parity-correct（NTZ 保 UTC、LTZ 不下推，D4）；**fe-core 重复版 = P5-T29 删除目标**（P1-T02 推迟项）。 - BE 经 JNI（**及 C++ native** `paimon_cpp_reader`）调 paimon-reader；连接器经 `ConnectorScanPlanProvider.getSerializedTable` 序列化 `Table`。BE 冻结不动；序列化身份是契约（Base64 非 blocker，BE 有 STD fallback；须 pin paimon-core 版本三方对齐）。 -- **0 个测试** —— 须建测试模块（no-mockito seam）+ parity baseline。 +- **测试**：连接器测试模块已建（no-mockito recording seam，~300+ 测）+ FE→BE serde round-trip smoke + parity baseline（live-e2e CI-gated `enablePaimonTest`）。 - 详尽 code-grounded 分析见 [recon](../research/p5-paimon-migration-recon.md) + [P5 设计 doc](../tasks/P5-paimon-migration.md)。 --- @@ -77,6 +78,10 @@ ## 进度日志 +### 2026-06-20（阶段里程碑 · 迁移+翻闸合入 #64446） +- **B0–B7 全完成并 squash-合入 `branch-catalog-spi`**（PR **#64446 / `38e7140ce56`** + `e9c5b3e70ce` 修编译）：B5 MTMV 桥（通用 `PluginDrivenMvccExternalTable`，D-040/041/042）+ B5b 时间旅行全 parity（AS-OF/tag/branch/@incr，D-043/044）+ B6 procedure no-op + **B7 翻闸**（入 `SPI_READY_TYPES` + GSON 原子 compat + D-045/046/047 restore SHOW PARTITIONS/SHOW CREATE）+ P6 全路径 clean-room review 全部 deviation fix。 +- **下一 = P5-T29（B8 删 fe-core legacy + maven 依赖）**：见 [tasks/P5 §P5-T29 执行计划](../tasks/P5-paimon-migration.md)（DEAD `datasource/paimon/`(30)+`metacache/paimon/`(3)+`systable/PaimonSysTable`；硬前置=迁出 `PaimonExternalCatalog` 常量；STILL-CONSUMED `property/metastore/Paimon*`(7) 保留；maven 方案 A/B）。 + ### 2026-06-10（B0–B4 实现里程碑，未提交） - **B4（本 session，T16-T20）= sys-tables E7 + MVCC E5**：连接器 SPI `listSupportedSysTables`/`getSysTableHandle`（D-039 复用 live `SysTableResolver` 机制）；fe-core 通用 `PluginDrivenSysExternalTable`/`PluginDrivenSysTable`；forceJni(binlog/audit_log)；`buildTableDescriptor`→HIVE_TABLE（同修 B2 遗留 [DV-024]）；sys 表 fail-loud 拒 time-travel/scan-params；E5 三方法（inert until B5）+ caps。3-lens 复审 1 BLOCKER（scan-path 丢 forceJni）已修。连接器 124 绿 + fe-core 100 绿。 - B0–B3 此前已落（测基建 / flavor 装配 / normal-read / DDL metadata；见 tasks/P5 阶段日志）。 diff --git a/plan-doc/tasks/P5-paimon-migration.md b/plan-doc/tasks/P5-paimon-migration.md index d8390202792fe0..6615d134c6d075 100644 --- a/plan-doc/tasks/P5-paimon-migration.md +++ b/plan-doc/tasks/P5-paimon-migration.md @@ -7,11 +7,11 @@ ## 元信息 -- **状态**：🟢 进行中（**B4 已完成 2026-06-10**：T16-T20 sys-tables E7 + MVCC E5，连接器 124/0/0/1 绿 + fe-core PluginDriven*Test 98+ 绿、checkstyle 0、import-gate 0；D-039 签字（E7 复用 live SysTable 机制，非 RFC §10 [DV-023]）；3-lens final holistic review = PARITY/SCOPE READY + 1 ADVERSARIAL BLOCKER（scan-path 丢 forceJni）**已修**（`PluginDrivenScanNode.create` 改走 seam）。下一批 = **B5 MTMV 桥**（gated on B4 全完，现满足）。B0-B3 见阶段日志） +- **状态**：🟢 进行中（**B0–B7 全完成并合入 `branch-catalog-spi`** —— 测基建/flavor/normal-read/DDL/sys-tables+MVCC/MTMV桥/时间旅行/**翻闸** + P6 全路径 clean-room review 的全部 deviation fix，全部 squash 进 **PR #64446 / `38e7140ce56`**（随后 `e9c5b3e70ce` 修编译）。paimon 现已在 `SPI_READY_TYPES`，FE 走 SPI 路径。**仅剩 B8 = P5-T29 删 legacy（+ B9 = P5-T30 回归）**。下一批 = **P5-T29**，见下文 §P5-T29 执行计划 + §当前阻塞项。B0–B7 见任务表与阶段日志） - **启动日期**：2026-06-09（recon+设计） -- **目标完成**：TBD（估时 ~5-6 周，含 D2-A 的 MTMV/MVCC 桥） -- **阻塞**：无（D1=A / D2=A 已签字）；分批实现按 B0→B9 启动 -- **阻塞下游**：P5 是最后一个 lakehouse full-adopter 样板验证（E5/E6/E7/E10 首次落地）；其 SPI 新面（E7 sys-table hook、E10 MTMV 桥、E5 wiring）将被未来 iceberg/hudi 翻闸复用——设计错须二次迁移 +- **目标完成**：B8/B9 后即收官（P5 阶段最后一块主体工作） +- **阻塞**：无（B7 翻闸已合入 #64446；P5-T29 无硬阻塞，但 D 项 maven scope 须先与用户对齐方案 A/B，见 §P5-T29 执行计划） +- **阻塞下游**：P5 是首个 lakehouse full-adopter 样板（E5/E6/E7/E10 已首次落地并合入）；其 SPI 新面（E7 sys-table hook、E10 MTMV 桥、E5 wiring）将被未来 iceberg/hudi 翻闸复用 - **主 owner**：@morningman / TBD --- @@ -114,12 +114,50 @@ Master plan [§3.6](../00-connector-migration-master-plan.md)；策略 = full ad | P5-T34 | **branch time-travel（新 SPI）**：连接器经 Identifier branch 分量 / branch-table load（`branchManager().branchExists` 校验）；scan 读 branch 表 | B5b | C+T | ✅ 连接器(B5b-2c)+fe-core(B5b-3)；inert until B7 | D-040；branch 独立 schema/snapshot；详 HANDOFF | | P5-T35 | **incremental `@incr`（新 SPI）**：port ~180 行 `validateIncrementalReadParams` + paimon `incremental-between`/`-timestamp`/`-scan-mode` 键**入连接器**；fe-core 仅传 raw doris incr param map；scan 应用 copy opts | B5b | C+T | ✅ 连接器(B5b-2b)+fe-core(B5b-3)；inert until B7 | D-040；与 tableSnapshot 互斥 | | P5-T26 | **procedure DOC no-op**：连接器档 E2 改「NOTHING TO PORT」（非「后续」）；钉死两假阳性（Spark migrate_table / iceberg expire_snapshots）；记未来 seam 位置（`ExecuteActionFactory:59-62` + 可选 `ConnectorProcedureOps`/E2 P6）；可选负回归（CALL/EXECUTE 仍报错）| B6 | D | ✅ | 零 code。B6 firsthand 核实：legacy `datasource/paimon/`+连接器 **0** procedure/action 文件；闭式 reject **双路**——`ALTER…EXECUTE`→`ExecuteActionFactory:59-62`（paimon=`PluginDrivenMvccExternalTable extends ExternalTable`→`else if(instanceof ExternalTable)`→`DdlException`），`CALL paimon.x`→`CallFunc:42-43`（闭式 switch default→`AnalysisException`）。doc 早于设计期已闭环（recon §3.3、connectors/paimon.md E2 行）。**neg-regression 归 B7 live-e2e**（验收 :72；结构已 guard，离线 UT 冗余故不加）| -| P5-T27 | **翻闸**：paimon 入 `SPI_READY_TYPES:52` + 删 built-in case `:142` + `pluginCatalogTypeToEngine` 加 `paimon→ENGINE_PAIMON`（`:937-944`）+ 删 `PhysicalPlanTranslator` PAIMON 分支(`:781`)+import(`:71`)| B7 | C | ✅ IMPLEMENTED（uncommitted，HEAD d2a2c8d）| **⚠️ 翻闸面 > 此 4-site 文档**：2026-06-11 9-agent 分类 + firsthand 证实**文档外 2 必修**（① `UserAuthentication:57-63` 加 PluginDrivenSysExternalTable unwrap = sys-表 auth 回归；② `PluginDrivenExternalTable.getEngine()/getEngineTableTypeName()` 加 `case "paimon"` = engine-名回归）+ `CreateTableInfo:395` 硬编码 MaxCompute 消息须修。余 site 全 LEGACY_DEAD/GENERIC_OK。**全 edit-set + 分类见 HANDOFF + [[catalog-spi-p5-b7-cutover-scope]]**。真正完成门 = B7 live-e2e（用户跑） | -| P5-T28 | **翻闸 GSON 原子**：5 catalog 名 + db + table 全转 `registerCompatibleSubtype`→PluginDriven*（table→**通用** `PluginDrivenMvccExternalTable`，D-042，非 paimon 专类）；加 5 flavor tag replay 测 | B7 | C+T | ✅ IMPLEMENTED（uncommitted）| 漏 db→ClassCastException。`GsonUtils:391-397/451/472`+删 7 import `:169-175`。**+ 用户签 D-045 = restore SHOW PARTITIONS 5 列 / D-046 = restore SHOW CREATE TABLE LOCATION+PROPERTIES**（full parity，非 MC 缩减；签 D-047=Hybrid SPI）→ **见 T36/T37 ✅** | -| P5-T36 | **D-045 restore SHOW PARTITIONS 5 列**：SPI `ConnectorPartitionInfo` 加 typed `long fileCount`（7-arg ctor，3-arg 默认 UNKNOWN，equals/hashCode）；`ConnectorCapability.SUPPORTS_PARTITION_STATS`；`PaimonConnector` 声明 capability + `collectPartitions:891` 喂 `partition.fileCount()`；`ShowPartitionsCommand` capability-gated 5 列 handler + getMetaData（`hasPartitionStatsCapability` 同 gate 两站点；MaxCompute 保持 1 列）| B7 | C+T | ✅ IMPLEMENTED+verified（uncommitted）| D-047=Hybrid。列：Partition/PartitionKey(=表分区列名 comma-join，每行同)/RecordCount(=getRowCount)/FileSizeInBytes(=getSizeBytes)/FileCount(=getFileCount)。**NIT（保留）**：5 列路径应用 partition-name `filterMap`（WHERE Partition=...），legacy 5 列 handler 忽略——新行为更正确（同 1 列/HMS 路径），无 golden 用 WHERE。测：ConnectorPartitionInfoTest(3)+PaimonConnectorMetadataPartitionTest.listPartitionsCarriesFileCount+ShowPartitionsCommandPluginDrivenTest.testHandlerEmitsFiveColumns。4-lens 对抗 review clean | -| P5-T37 | **D-046 restore SHOW CREATE TABLE LOCATION+PROPERTIES**：连接器 `buildTableSchema:202` 把 `((DataTable)table).coreOptions().toMap()`（含 path）+ 注入 `primary-key` merge 入 schemaProps（+ plumb `table` 入参，2 call-site）；`PluginDrivenSchemaCacheValue` 加 `tableProperties`（4-arg 重载，3-arg 默认 emptyMap）；`PluginDrivenExternalTable.getTableProperties()`（剔 schema-control 键 partition_columns/primary_keys）；`Env.getDdlStmt` PLUGIN 分支(`:4927`)render LOCATION ''+PROPERTIES，unwrap `PluginDrivenSysExternalTable`→source，**空-props gate**（MaxCompute 空→保持 comment-only）| B7 | C+T | ✅ IMPLEMENTED+verified（uncommitted）| D-047=Hybrid。byte-parity legacy（golden `test_paimon_table_properties.out`）。**凭据**：firsthand+4-lens credential-leak 证伪——table coreOptions 仅 path/write-only/file.format，catalog 凭据在 catalog 级（B2 已 neuter getProperties），不泄漏。**连接器侧 coreOptions merge 无离线 UT**（FakePaimonTable 非 DataTable）→ code-review + B9 live-e2e 覆盖。测（fe-core 侧）：PluginDrivenExternalTablePartitionTest.testGetTableProperties×2。仅 fix 4927（SHOW CREATE 走 `getDdlStmt(Command,...)` 重载；4507 重载 legacy 无 PAIMON LOCATION 故不改）| -| P5-T29 | **删 legacy**：`datasource/paimon/`(28) + `metacache/paimon/`(3) + 反向引用；确认零引用；验 paimon-core FE classpath 恰一份（R-004/R-007 NoClassDefFound 守）| B8 | C | ⏳ | gated on 翻闸 live 验 | -| P5-T30 | post-cutover 回归：SHOW PARTITIONS + partitions TVF（预接 FE 分发现返行）/DROP·CREATE DB·TABLE/no-ENGINE CREATE/edit-log replay/MTMV 增量刷/sys-table/session-TZ 谓词不丢行 | B9 | T | ⏳ | | +| P5-T27 | **翻闸**：paimon 入 `SPI_READY_TYPES:52` + 删 built-in case `:142` + `pluginCatalogTypeToEngine` 加 `paimon→ENGINE_PAIMON`（`:937-944`）+ 删 `PhysicalPlanTranslator` PAIMON 分支(`:781`)+import(`:71`)| B7 | C | ✅ 已合入 #64446 | **⚠️ 翻闸面 > 此 4-site 文档**：2026-06-11 9-agent 分类 + firsthand 证实**文档外 2 必修**（① `UserAuthentication:57-63` 加 PluginDrivenSysExternalTable unwrap = sys-表 auth 回归；② `PluginDrivenExternalTable.getEngine()/getEngineTableTypeName()` 加 `case "paimon"` = engine-名回归）+ `CreateTableInfo:395` 硬编码 MaxCompute 消息须修。余 site 全 LEGACY_DEAD/GENERIC_OK。**全 edit-set + 分类见 HANDOFF + [[catalog-spi-p5-b7-cutover-scope]]**。真正完成门 = B7 live-e2e（用户跑） | +| P5-T28 | **翻闸 GSON 原子**：5 catalog 名 + db + table 全转 `registerCompatibleSubtype`→PluginDriven*（table→**通用** `PluginDrivenMvccExternalTable`，D-042，非 paimon 专类）；加 5 flavor tag replay 测 | B7 | C+T | ✅ 已合入 #64446 | 漏 db→ClassCastException。`GsonUtils:391-397/451/472`+删 7 import `:169-175`。**+ 用户签 D-045 = restore SHOW PARTITIONS 5 列 / D-046 = restore SHOW CREATE TABLE LOCATION+PROPERTIES**（full parity，非 MC 缩减；签 D-047=Hybrid SPI）→ **见 T36/T37 ✅** | +| P5-T36 | **D-045 restore SHOW PARTITIONS 5 列**：SPI `ConnectorPartitionInfo` 加 typed `long fileCount`（7-arg ctor，3-arg 默认 UNKNOWN，equals/hashCode）；`ConnectorCapability.SUPPORTS_PARTITION_STATS`；`PaimonConnector` 声明 capability + `collectPartitions:891` 喂 `partition.fileCount()`；`ShowPartitionsCommand` capability-gated 5 列 handler + getMetaData（`hasPartitionStatsCapability` 同 gate 两站点；MaxCompute 保持 1 列）| B7 | C+T | ✅ 已合入 #64446 | D-047=Hybrid。列：Partition/PartitionKey(=表分区列名 comma-join，每行同)/RecordCount(=getRowCount)/FileSizeInBytes(=getSizeBytes)/FileCount(=getFileCount)。**NIT（保留）**：5 列路径应用 partition-name `filterMap`（WHERE Partition=...），legacy 5 列 handler 忽略——新行为更正确（同 1 列/HMS 路径），无 golden 用 WHERE。测：ConnectorPartitionInfoTest(3)+PaimonConnectorMetadataPartitionTest.listPartitionsCarriesFileCount+ShowPartitionsCommandPluginDrivenTest.testHandlerEmitsFiveColumns。4-lens 对抗 review clean | +| P5-T37 | **D-046 restore SHOW CREATE TABLE LOCATION+PROPERTIES**：连接器 `buildTableSchema:202` 把 `((DataTable)table).coreOptions().toMap()`（含 path）+ 注入 `primary-key` merge 入 schemaProps（+ plumb `table` 入参，2 call-site）；`PluginDrivenSchemaCacheValue` 加 `tableProperties`（4-arg 重载，3-arg 默认 emptyMap）；`PluginDrivenExternalTable.getTableProperties()`（剔 schema-control 键 partition_columns/primary_keys）；`Env.getDdlStmt` PLUGIN 分支(`:4927`)render LOCATION ''+PROPERTIES，unwrap `PluginDrivenSysExternalTable`→source，**空-props gate**（MaxCompute 空→保持 comment-only）| B7 | C+T | ✅ 已合入 #64446 | D-047=Hybrid。byte-parity legacy（golden `test_paimon_table_properties.out`）。**凭据**：firsthand+4-lens credential-leak 证伪——table coreOptions 仅 path/write-only/file.format，catalog 凭据在 catalog 级（B2 已 neuter getProperties），不泄漏。**连接器侧 coreOptions merge 无离线 UT**（FakePaimonTable 非 DataTable）→ code-review + B9 live-e2e 覆盖。测（fe-core 侧）：PluginDrivenExternalTablePartitionTest.testGetTableProperties×2。仅 fix 4927（SHOW CREATE 走 `getDdlStmt(Command,...)` 重载；4507 重载 legacy 无 PAIMON LOCATION 故不改）| +| P5-T29 | **删 legacy + maven 依赖**（🎯 **下一个 session 的活**）：删 `datasource/paimon/`(**30**) + `metacache/paimon/`(3) + `systable/PaimonSysTable`(1) + 清 8 处反向引用死分支/import + 删 fe-core paimon maven 依赖；**硬前置**=迁出 `PaimonExternalCatalog` 常量(被 5 个 STILL-CONSUMED metastore-props 引)；**STILL-CONSUMED 不删**=`property/metastore/Paimon*`(7)；验 paimon-core FE classpath 恰一份（R-004/R-007 NoClassDefFound 守）| B8 | C | 🚧 **Batch1 ✅** | **Batch1(C1)=删 33 dead + 清 6 reverse-ref + 5 dead-test + inline `getPaimonCatalogType` 常量**，local-commit `7632a074e4b`（未 push，fe-core test-compile+checkstyle 绿、49 测过）。用户签 **Plan B**(fully paimon-free) + **D-PB1** strip-in-place(不物理搬 7 类) + **D-PB2** phased；**B1-strip 6 metastore-props + 迁 `PaimonVendedCredentialsProvider` + 删 5 maven dep 移到 Batch2**(docker-gated)。完整修订计划见 [design doc](./designs/P5-T29-paimon-legacy-removal-design.md) | +| P5-T30 | post-cutover 回归：SHOW PARTITIONS + partitions TVF（预接 FE 分发现返行）/DROP·CREATE DB·TABLE/no-ENGINE CREATE/edit-log replay/MTMV 增量刷/sys-table/session-TZ 谓词不丢行 | B9 | T | ⏳ | 翻闸前已跑过一轮（B7 硬门）；P5-T29 删 legacy 后**复跑一遍**确认无回归，可与 T29 §E 验证合并 | + +--- + +## P5-T29 执行计划（B8 删 legacy + maven 依赖）— scope ledger（2026-06-20 在 `branch-catalog-spi` firsthand 核实） + +> 🎯 这是 P5 阶段最后一块主体工作。对照基线 = [`reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md`](../reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md) §B8 deletion readiness ledger。样板 = **P4 #64300**（"make fe-core odps-free"，删文件+清反向引用+删 maven 依赖+`dependency:tree` 验证）。 +> **⚠️ 这不是一次 `rm -rf datasource/paimon/`**——有 STILL-CONSUMED 子树 + 常量耦合前置，naive 删除断编译。 + +### A. DEAD —— 可删 +- [ ] `datasource/paimon/`（**30** 文件，含 `source/`、`profile/`；catalog/factory/db/table、`PaimonExternalCatalog`、`PaimonExternalMetaCache`、`PaimonSysExternalTable`、legacy `source/PaimonScanNode`/`PaimonSplit`/`PaimonSource`、legacy 重复 `source/PaimonPredicateConverter`/`PaimonValueConverter`(P1-T02 推迟项现可收)）。 +- [ ] `datasource/metacache/paimon/`（**3**：`PaimonTableLoader`/`PaimonPartitionInfoLoader`/`PaimonLatestSnapshotProjectionLoader`）。 +- [ ] `datasource/systable/PaimonSysTable.java`（**1**）。 +- [ ] 消费方死分支/import 清理（文件保留，只删 paimon 分支）：`ExternalMetaCacheMgr`（`paimon()`/`ENGINE_PAIMON` 路由 + `PaimonExternalMetaCache`）、`metacache/ExternalMetaCacheRouteResolver`（`ENGINE_PAIMON` 注册）、`catalog/Env`、`nereids/.../UserAuthentication`、`nereids/.../ShowPartitionsCommand`、`credentials/VendedCredentialsFactory`、`ExternalCatalog`（`buildDbForInit` 死分支）。**逐个 grep 确认是死分支再删。** +- [ ] 死测试：`ExternalMetaCacheRouteResolverTest`、`planner/PaimonPredicateConverterTest`（测 legacy 重复转换器）、`StatementContextTest`（paimon 用法）等——按编译失败/语义死亡逐个判。 + +### B. 硬前置（删 `datasource/paimon/` **之前**必做） +- [ ] **迁出 `PaimonExternalCatalog` 常量** `PAIMON_FILESYSTEM`/`PAIMON_HMS`（及其它被引常量）——被 **5 个 STILL-CONSUMED** `property/metastore/Paimon*MetaStoreProperties` 类 import（已核实）。须先搬到存活的家（metastore-props 模块常量持有者 / `fe-kerberos` / 新常量类），再删 catalog 类。 +- [ ] scrub 悬空 javadoc `{@link PaimonSysTable}`（`PluginDrivenSysTable`/`NativeSysTable` 等），否则 strict checkstyle/javadoc 挂。 +- [ ] 保 load-bearing dispatch ordering（PluginDriven 分支先于任何 legacy 分支）。 +- [ ] **`ENGINE_PAIMON` 区分**：`metacache` 两处 DEAD（删）；**`nereids/.../info/CreateTableInfo.ENGINE_PAIMON`（`:123`，被 `:790/:937/:967/:1150` 用作翻闸后 engine 名 + distribution 校验）是 LIVE，保留。** + +### C. STILL-CONSUMED —— **不在 P5-T29 删除范围**（删了断 cutover Kerberos 装配） +- `property/metastore/Paimon*MetaStoreProperties`（HMS/DLF/Rest/Jdbc/FileSystem，5）+ `AbstractPaimonProperties` + `PaimonPropertiesFactory`（共 7）+ 其测试 `Paimon*MetaStorePropertiesTest`。cutover `initPreExecutionAuthenticator`→Kerberos 装配 **LIVE**（P6 review R1）；属 metastore-storage-refactor 子线（D-016），主线 B8 不碰。 + +### D. Maven 依赖（用户明确点名「相关 maven 依赖」）— ⚠️ **核心冲突，须先定决策** +`fe/fe-core/pom.xml:543-563` 含 `paimon-core`/`paimon-common`/`paimon-format`/`paimon-s3`/`paimon-jindo`（+ `:576` s3 aws-bundle 注释，与 iceberg-aws 共享）。 +- **关键事实**：C 项 STILL-CONSUMED `property/metastore/Paimon*` **直接 import `org.apache.paimon.*` SDK**（已核实 6 文件）→ **只要它们留 fe-core，fe-core 就不可能像 P4(odps-free) 完全 paimon-free。** +- 可能可删：`paimon-format`/`-s3`/`-jindo`（legacy reader/格式/IO 专用，随 `datasource/paimon/source` 删除而无消费方）；可能保留：`paimon-core`/`-common`（metastore-props 用）。真实可删集合由 `dependency:tree | grep paimon` + 编译敲定。 +- **🔱 开放决策（建议下一 session 先 AskUserQuestion）**：是否把 `property/metastore/Paimon*` 一并迁出 fe-core 使其完全 paimon-free？ + - **方案 A（推荐，对齐 master plan B8 / D-016 scope）**：保留 STILL-CONSUMED metastore-props，**只删 DEAD 子树 + 部分 maven 依赖**（fe-core 保 paimon-core/common）。surface 小、与已签 B8 scope 一致。 + - **方案 B（更大，越界子线）**：连带迁出 metastore-props，fe-core 完全 paimon-free（对齐 P4 终态）；碰 metastore-storage-refactor 子线，宜单独立项或与子线 P2-T05 合并。 + +### E. 守门 / 验证（mirror P4 #64300） +- [ ] fe-core 编译 BUILD SUCCESS + checkstyle 0 + import-gate 净（`tools/check-connector-imports.sh`）。 +- [ ] 连接器测试仍绿（删 legacy 不应触连接器）。 +- [ ] `dependency:tree` 验 paimon-core 在 FE classpath 恰一份（R-004/R-007 NoClassDefFound / SDK 单例守）。 +- [ ] regression-gated live-e2e（`enablePaimonTest=true`，用户跑）= 删后 5-flavor 读 + sys-table + MTMV + DDL 不回归（= B9/P5-T30，可合并）。 +- [ ] 逐子树删 + 每批跑编译（参 master plan §3.9 / §4 playbook 第 13 步）。 --- @@ -255,6 +293,12 @@ B6 (procedure doc no-op, 独立) │ ## 阶段日志（倒序） +### 2026-06-20（阶段里程碑 · B5–B7 翻闸 + P6 clean-room review + **全部合入 #64446**；本 session = 文档对账，0 产线代码） +- **B0–B7 全完成并 squash-合入 `branch-catalog-spi`** —— **PR #64446 / `38e7140ce56`**（"[refactor](catalog) P5 paimon: migrate to catalog SPI + cutover"），随后 `e9c5b3e70ce`（修编译/HANDOFF）。涵盖：B5 MTMV 桥（通用 `PluginDrivenMvccExternalTable`，D-040/041/042）、B5b 显式时间旅行全 parity（AS-OF/tag/branch/@incr，D-043/044；RD-1 partitioned time-travel 0 行丢失 BLOCKER 已修）、B6 procedure no-op（doc）、**B7 翻闸**（paimon 入 `SPI_READY_TYPES` + GSON 原子转 compat + `PhysicalPlanTranslator` 删 PAIMON 分支 + `UserAuthentication`/`getEngine` 补接 + **D-045/046/047 = restore SHOW PARTITIONS 5 列 / SHOW CREATE TABLE LOCATION+PROPERTIES**，T36/T37）。 +- **P6 全路径 clean-room 对抗 review（6 维度 + 7 缺口线，2 波）完成** → 报告 [`reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md`](../reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md)。结论：**2 BLOCKER 都是 B8 删除护栏（非运行时 bug）**（R1=legacy metastore-props + `PaimonExternalCatalog` 常量 LIVE；R2=`property/storage/*Properties` 跨连接器共享 → **B8 不能整包删，须分阶段**）；**2 MAJOR 真活读路回归**（C1 MinIO、C2 HDFS XML，均已修）；其余 parity。发现项各自 fix task，**全部完成并合入 #64446**：C1 MinIO / C2 HDFS XML / R3-residual / R1-table / C4+R2+R3-catalog / 5 个 deviation→fix（A3 self-split-weight / A2 predicates-from-paimon / B-MC2 schema-at-memo / A1 split-weight / B-R2-be schema-dict-memo）。 +- **当前 legacy 残留（待 P5-T29 删）**：`datasource/paimon/`(30) + `metacache/paimon/`(3) + `systable/PaimonSysTable`(1) + 8 处反向引用文件 + fe-core paimon maven 依赖(5)。STILL-CONSUMED `property/metastore/Paimon*`(7) 保留。详见 §P5-T29 执行计划。 +- **本 session 净产出**：对账 stale 跟踪文档（PROGRESS 停在 B4、本 doc 停在 B5b、HANDOFF 停在历史工作分支）→ 全部刷到「迁移+翻闸已合入、下一步 P5-T29」状态，使下一 session 可直接开工。0 产线代码。 + ### 2026-06-10（B4 实现：sys-tables E7 + MVCC E5，T16-T20；understand workflow 纠偏 → 用户签 D-039；subagent-driven 5 dispatch + 双审/fix-loop + 3-lens final holistic（1 BLOCKER 修）） - **understand workflow（6-agent read-only）纠偏 2 处 plan 前提**：① **RFC §10 stale**（其 `$`-后缀-via-`getTableHandle` E7 设计从未落地；live fe-core 用 `SysTableResolver`+`NativeSysTable`+`TableIf.getSupportedSysTables/findSysTable`）→ 用户签 **D-039**（复用 live 机制，[DV-023]）；② **T20 MVCC inert until B5**（E5 方法已存在 default-no-op，但 `PluginDrivenExternalTable` 非 `MvccTable`、零 fe-core 消费方、capability 零 reader；翻闸 gated on B5 故 inert capability 安全）→ 用户签「T20 留 B4 作连接器 groundwork」。另核出 **BE 描述符**：legacy paimon（普通+sys）发 `HIVE_TABLE`，而连接器无 `buildTableDescriptor` override → 普通表走 `SCHEMA_TABLE` fallback（[DV-024]，B2 遗留缺陷，B4/T19 一处修）。 @@ -320,6 +364,6 @@ B6 (procedure doc no-op, 独立) │ ## 当前阻塞项 -- 无硬阻塞（D1=A / D2=A / D4=A / D5=A / D6=A / D7=B / **D-039 (E7=live SysTable 机制)** 已签字；**B0 + B1 + B2 + B3 + B4 已完成**）。下一 session = **B5 MTMV 桥**（gated on B4 全完，现满足）：T21 GAP-LISTPART-AT-SNAPSHOT / T22 fe-core `PaimonPluginDrivenExternalTable` implements MTMVRelatedTableIf+MTMVBaseTableIf+MvccTable + `loadSnapshot` / T23 子类 MTMV 方法 / T24 rehome `PaimonMvccSnapshot` / T25 isPartitionInvalid parity。**B5 须把 B4 inert 的 E5 接活**：`PluginDrivenExternalTable`（或新 paimon 子类）implements MvccTable → 调 `connector.getMetadata().beginQuerySnapshot` 包成 `ConnectorMvccSnapshotAdapter`（现零构造方）；并把 scan-params/time-travel 接到 `PluginDrivenScanNode`（T19 sys-table fail-loud guard 现可能 dormant，B5 接活后即生效）。**B6**（procedure doc no-op，独立）可随时穿插。 -- 翻闸（B7）仍 gated on B2+B3+B4+B5 全完 + live e2e（用户真实 paimon 各 flavor 环境）。**翻闸/live-e2e 硬门**（见阶段日志 B1 条 + 「风险/开放问题」）：hms/dlf metastore-client 跨 loader、jdbc driver_url 安全 allow-list、hive-site.xml 文件加载、live createCatalog；**B3 门**：DDL 的 `executeAuthenticated`（D7=B）Kerberized 正确性；**B4 新增 live-e2e 门**：① `buildTableDescriptor`→HIVE_TABLE 在 BE 真实 paimon 普通表+sys 表 scan（[DV-024]，离线只到连接器边界）；② MVCC SDK-delegation（`CatalogBackedPaimonCatalogOps` 的 DataTable cast / `earlierOrEqualTimeMills` / `tryGetSnapshot`，离线仅 fake 覆盖）；③ binlog/audit_log 真走 JNI（forceJni 端到端）+ snapshots/schemas sys 表查询；④ sys 表 time-travel 真 fail-loud（须 B5 接活 scan-params/snapshot 后）。 -- 复用资产：`PaimonCatalogFactory`；`PaimonCatalogOps` seam（现含 5 read + 4 DDL + 3 snapshot 方法）；`PaimonTableResolver`（sys-aware reload，B5 复用）；`PaimonTypeMapping`（双向）；`PaimonSchemaBuilder`；fe-core `PluginDrivenSysExternalTable`/`PluginDrivenSysTable`（通用 sys 机制，未来 iceberg/hudi 复用）；`RecordingPaimonCatalogOps`/`RecordingConnectorContext`/`FakePaimonTable` 测基建（B5 复用）；parity doc 是后续批次 gap 清单 + 翻闸门基准。 +- **无硬阻塞**。B0–B7（迁移 + 翻闸）+ P6 clean-room review 全部 deviation fix **已合入 `branch-catalog-spi`（#64446 / `38e7140ce56`，+ `e9c5b3e70ce` 修编译）**。翻闸 live-e2e（5-flavor，B7 硬门）已随合入跑过。**下一 session = P5-T29（B8 删 legacy + maven 依赖）+ P5-T30（B9 回归）**，见上文 §P5-T29 执行计划。 +- **唯一须先对齐的事**：P5-T29 §D 的 **maven scope 方案 A vs B**（fe-core 是否完全 paimon-free；建议先 AskUserQuestion）。其余按 §A–E checklist 逐子树删 + 每批跑编译即可。 +- 复用资产（删除时勿误删）：fe-core 通用桥 `PluginDrivenMvccExternalTable`/`PluginDrivenSysExternalTable`/`PluginDrivenSysTable`/`NativeSysTable`（未来 iceberg/hudi 复用）；连接器侧 `PaimonCatalogFactory`/`PaimonCatalogOps`/`PaimonTableResolver`/`PaimonTypeMapping`/`PaimonSchemaBuilder`/测基建——这些都在 `fe-connector-paimon`，**不是**删除目标。STILL-CONSUMED `property/metastore/Paimon*`（fe-core）保留。 diff --git a/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md b/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md index 4cadbe1a6e6658..b3bbd5418c7fca 100644 --- a/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md +++ b/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md @@ -21,13 +21,23 @@ Two decisions were taken via AskUserQuestion after the feasibility dig: physically relocated (B2 was rejected — it forces a generic `MetastoreProperties`-registry rework + cross-loader re-basing for no marginal benefit toward dropping deps, and iceberg/hive keep their metastore-props in fe-core *with* their engine SDK, so B1 makes paimon the clean outlier — parity-OK). -- **D-PB2 — sequencing = phased.** - - **Batch 1 (this doc, safe core):** delete the 33 DEAD files + reverse-ref cleanups + dead tests + - B1-strip the 6 metastore-props. **paimon maven deps STAY** (still needed by the one genuinely-LIVE - SDK class, `PaimonVendedCredentialsProvider`). - - **Batch 2 (later, docker-e2e-gated, separate):** migrate `PaimonVendedCredentialsProvider` out of - fe-core + rework the generic `VendedCredentialsFactory` paimon seam (shared with iceberg) + **drop - all 5 paimon maven deps**. This is the genuine cross-cutting piece; isolated for review/risk. +- **D-PB2 — sequencing = phased.** *(Refined 2026-06-20 after firsthand discovery — see note below.)* + - **Batch 1 (this doc, safe core) — ✅ DONE (commit `7632a074e4b`):** delete the 33 DEAD files + + reverse-ref cleanups + dead tests + decouple the metastore-props from the deleted + `PaimonExternalCatalog` constants (inline `getPaimonCatalogType` literals). **paimon maven deps STAY.** + - **Batch 2 (later, docker-e2e-gated, separate):** **B1-strip the 6 metastore-props** (remove their + paimon-SDK catalog-building methods + imports + trim the 7 catalog-building test files) + migrate + `PaimonVendedCredentialsProvider` out of fe-core + rework the generic `VendedCredentialsFactory` + paimon seam (shared with iceberg) + **drop all 5 paimon maven deps**. All SDK-removal lands together. + + > **Refinement (user-signed 2026-06-20):** the B1 strip was originally slotted into Batch 1, but + > firsthand recon showed it is *not* "deletions only" — it reshapes 6 LIVE classes (the strip-target + > methods have zero live main callers, but the live `executionAuthenticator`/`initExecutionAuthenticator` + > wiring at `PluginDrivenExternalCatalog:137-138` must be preserved) and gut-trims **7** metastore-props + > test files that assert the dead catalog-building (`AbstractPaimonPropertiesTest`, `PaimonCatalogTest` + > [@Disabled manual → delete], `Paimon{HMS,FileSystem,Jdbc,Rest,AliyunDLF}MetaStorePropertiesTest`). + > It drops no dep by itself (it is a *prerequisite* for the Batch-2 dep-drop). So it was **moved to + > Batch 2**, leaving Batch 1 as the clean, complete "remove dead legacy" PR. End state after Batch 1+2 = fe-core fully paimon-SDK-free (zero `org.apache.paimon.*` imports), all 5 paimon maven deps gone. @@ -121,7 +131,7 @@ generic `VendedCredentialsFactory.getProviderType()` `case PAIMON` ← `CatalogP --- -## 4. Batch 1 — B1 strip the 6 metastore-props (paimon-SDK-free) +## 4. Batch 2 — B1 strip the 6 metastore-props (paimon-SDK-free) — *moved out of Batch 1* **Strip from `AbstractPaimonProperties` + 5 flavors:** the `org.apache.paimon.*` imports; abstract+impl `initializeCatalog(...)`; `buildCatalogOptions()`/`appendCatalogOptions()`/abstract @@ -146,26 +156,28 @@ calls into stripped methods (e.g. `buildCatalogOptions`/`getCatalogOptionsMap`) --- -## 5. Commit plan (each compiles independently; cycle-safe) +## 5. Commit plan The dead subtree and the metastore-props are **mutually dependent** (subtree calls `initializeCatalog`; props reference `PaimonExternalCatalog.PAIMON_*`). **Additionally** the dead subtree calls a *removed* reverse-ref symbol: `PaimonUtils:57` → `ExternalMetaCacheMgr.paimon()`. So — exactly as P4 #64300 found ("reverse-ref removal and file deletion must land as one compiling unit") — severing the reverse-refs and -deleting the dead files **cannot** be split. Batch 1 = **2 commits**: +deleting the dead files **cannot** be split. +**Batch 1 = 1 commit — ✅ DONE (`7632a074e4b`):** - **C1 (sever reverse-refs + delete dead, atomic):** §2 reverse-ref cleanups (6 files) + §2 javadoc - scrubs (3) + §4 decouple (inline `getPaimonCatalogType` literals in 5 flavors, drop their - `PaimonExternalCatalog` import) + §3 fixture-test trims (2) + 2 constant-test repoints + `git rm` the - 33 dead files (§1) + 5 dead test files (§3). After C1 the metastore-props keep their SDK catalog-building - methods (now caller-less) but still compile against the present paimon deps. - *Verify:* fe-core `test-compile` green; `datasource/paimon/` holds only `PaimonVendedCredentialsProvider`. + scrubs (3) + §4-decouple only (inline `getPaimonCatalogType` literals in 5 flavors, drop their + `PaimonExternalCatalog` import — NOT the SDK strip) + §3 fixture-test trims (2) + 2 constant-test repoints + + `git rm` the 33 dead files (§1) + 5 dead test files (§3). After C1 the metastore-props keep their SDK + catalog-building methods (now caller-less) and still compile against the present paimon deps. + *Verified:* fe-core `test-compile` BUILD SUCCESS + checkstyle 0; 49 affected tests pass; + `datasource/paimon/` holds only `PaimonVendedCredentialsProvider`. *(First attempt split this into prep-then-delete; the `PaimonUtils → paimon()` coupling broke the intermediate compile — merged per P4 precedent.)* -- **C2 (B1 strip):** §4 strip SDK methods + imports from the 6 metastore-props + trim their tests' - assertions on the stripped catalog-building methods (`buildCatalogOptions`/`getCatalogOptions`/ - `getMetastoreType`). *Verify:* fe-core compiles; checkstyle 0; import-gate net; paimon connector UT - green; `grep org.apache.paimon fe-core/src/main` = only `PaimonVendedCredentialsProvider` remains (Batch 2). + +**Batch 2 (later, docker-gated):** §4 strip SDK methods + imports from the 6 metastore-props + trim the 7 +catalog-building test files; migrate `PaimonVendedCredentialsProvider`; rework `VendedCredentialsFactory`; +**drop all 5 paimon deps**. *Target:* `grep org.apache.paimon fe-core/src/main` = ∅; `dependency:tree | grep paimon` = ∅. **Hard pre-commit (HANDOFF):** scrub `regression-test/conf/regression-conf.groovy` (plaintext key); clean scratch (`.audit-scratch/`/`conf.cmy/`/`META-INF/`/`*.bak`). **Path-whitelist `git add`; NEVER `git add -A`.** From 32de7d16702cc01c0fe73d82f97f8f487afe58d6 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 20 Jun 2026 23:50:17 +0800 Subject: [PATCH 3/4] =?UTF-8?q?[P5-T29]=20paimon=20B8=20(Batch=202):=20mak?= =?UTF-8?q?e=20fe-core=20paimon-SDK-free=20=E2=80=94=20strip=20metastore-p?= =?UTF-8?q?rops=20+=20delete=20vended=20provider=20(GAMMA)=20+=20drop=20de?= =?UTF-8?q?ps?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Batch 2 of paimon legacy removal (Batch 1 = 7632a074e4b). After this, fe-core has zero org.apache.paimon imports and the 5 paimon maven deps are gone. Why now: the 6 STILL-CONSUMED property/metastore/Paimon* classes kept paimon-SDK in dead catalog-building methods (initializeCatalog etc., 0 live main callers — the plugin/cutover path builds catalogs connector-side), which blocked the dep drop. What changed: - Strip the dead paimon-SDK catalog-building cluster from AbstractPaimonProperties + the 5 flavors (initializeCatalog / buildCatalogOptions / appendCatalogOptions / appendCustomCatalogOptions / getMetastoreType / getCatalogOptionsMap / normalizeS3Config / appendUserHadoopConfig + the Options catalogOptions field; Jdbc also getBackendPaimonOptions / registerJdbcDriver / DriverShim). Keep all LIVE SDK-free duties: warehouse @ConnectorProperty; the executionAuthenticator / initExecutionAuthenticator / initHdfsExecutionAuthenticator Kerberos doAs wiring (read by PluginDrivenExternalCatalog:137-138); initNormalizeAndCheckProps/validation; getPaimonCatalogType; Type.PAIMON. - Vended credentials (GAMMA, user-signed): recon (+ adversarial review) found PaimonVendedCredentialsProvider's paimon-SDK methods are dead — reachable only via the iceberg-only getStoragePropertiesMapWithVendedCredentials; the real paimon vended path is the connector's PaimonScanPlanProvider.extractVendedToken (moved off fe-core at cutover FIX-1). Delete the provider + its test + the VendedCredentialsFactory case PAIMON, and relocate its one LIVE duty (the REST "skip static storage map" gate) to a new SDK-free MetastoreProperties.isVendedCredentialsEnabled() (base=false, PaimonRestMetaStoreProperties=true). CatalogProperty's gate routes iceberg through its provider (byte-identical) and everything else through the metastore-props method. - pom: drop paimon-core/common/format/s3/jindo; correct the s3-transfer-manager comment (real consumer = hadoop-aws, kept). fe/pom.xml paimon.version untouched (R-007: fe-connector-paimon + BE still consume it). - Tests: delete AbstractPaimonPropertiesTest / PaimonCatalogTest / PaimonDlfRestCatalogTest + PaimonVendedCredentialsProviderTest; trim the 5 flavor tests + VendedCredentialsFactoryTest; add gate tests (REST=true, HMS=false). Verified: fe-core test-compile BUILD SUCCESS + checkstyle 0; 32 affected tests green; tools/check-connector-imports.sh OK; dependency:tree -Dincludes=org.apache.paimon on fe-core = empty; s3-transfer-manager retained. live-e2e (enablePaimonTest=true) is docker-gated -> user-run (B9/P5-T30), NOT run here. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_011mTrPcvMZtFjsxWJM5TRnG --- fe/fe-core/pom.xml | 31 +- .../doris/datasource/CatalogProperty.java | 12 +- .../credentials/VendedCredentialsFactory.java | 7 +- .../PaimonVendedCredentialsProvider.java | 77 ---- .../metastore/AbstractPaimonProperties.java | 166 +-------- .../metastore/MetastoreProperties.java | 15 + .../PaimonAliyunDLFMetaStoreProperties.java | 63 +--- .../PaimonFileSystemMetaStoreProperties.java | 45 +-- .../PaimonHMSMetaStoreProperties.java | 53 --- .../PaimonJdbcMetaStoreProperties.java | 186 +--------- .../PaimonRestMetaStoreProperties.java | 35 +- .../VendedCredentialsFactoryTest.java | 32 -- .../PaimonVendedCredentialsProviderTest.java | 349 ------------------ .../AbstractPaimonPropertiesTest.java | 89 ----- ...aimonAliyunDLFMetaStorePropertiesTest.java | 142 ------- .../property/metastore/PaimonCatalogTest.java | 94 ----- .../metastore/PaimonDlfRestCatalogTest.java | 243 ------------ ...imonFileSystemMetaStorePropertiesTest.java | 41 +- .../PaimonHMSMetaStorePropertiesTest.java | 24 +- .../PaimonJdbcMetaStorePropertiesTest.java | 139 +------ .../PaimonRestMetaStorePropertiesTest.java | 78 +--- 21 files changed, 73 insertions(+), 1848 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProvider.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProviderTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/AbstractPaimonPropertiesTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonCatalogTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonDlfRestCatalogTest.java diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index 933c0d546e1342..ab6f279af1c24d 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -539,29 +539,6 @@ under the License. iceberg-aws ${iceberg.version} - - org.apache.paimon - paimon-core - - - - org.apache.paimon - paimon-common - - - - org.apache.paimon - paimon-format - - - - org.apache.paimon - paimon-s3 - - - org.apache.paimon - paimon-jindo - software.amazon.awssdk @@ -573,10 +550,10 @@ under the License. - + software.amazon.awssdk s3-transfer-manager diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java index 540e23e16281de..532b2767269d42 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java @@ -178,10 +178,18 @@ private void initStorageProperties() { if (storagePropertiesMap == null) { try { boolean checkStorageProperties = true; + MetastoreProperties msp = getMetastoreProperties(); AbstractVendedCredentialsProvider provider = - VendedCredentialsFactory.getProviderType(getMetastoreProperties()); + VendedCredentialsFactory.getProviderType(msp); if (provider != null) { - checkStorageProperties = !provider.isVendedCredentialsEnabled(getMetastoreProperties()); + checkStorageProperties = !provider.isVendedCredentialsEnabled(msp); + } else if (msp != null) { + // Non-provider backends signal vended credentials via the metastore-props gate + // (e.g. a Paimon REST catalog skips the static storage map): SDK-free replacement + // of the former VendedCredentialsFactory PAIMON case. Iceberg still routes through + // its provider above, so its behavior is unchanged. The null guard preserves the + // pre-change "build the static map" behavior when there is no metastore. + checkStorageProperties = !msp.isVendedCredentialsEnabled(); } if (checkStorageProperties) { this.orderedStoragePropertiesList = StorageProperties.createAll(getProperties()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/credentials/VendedCredentialsFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/credentials/VendedCredentialsFactory.java index 6528fdb89294ac..1c62f6141b83ca 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/credentials/VendedCredentialsFactory.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/credentials/VendedCredentialsFactory.java @@ -18,7 +18,6 @@ package org.apache.doris.datasource.credentials; import org.apache.doris.datasource.iceberg.IcebergVendedCredentialsProvider; -import org.apache.doris.datasource.paimon.PaimonVendedCredentialsProvider; import org.apache.doris.datasource.property.metastore.MetastoreProperties; import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.datasource.property.storage.StorageProperties.Type; @@ -62,10 +61,10 @@ public static AbstractVendedCredentialsProvider getProviderType(MetastorePropert switch (type) { case ICEBERG: return IcebergVendedCredentialsProvider.getInstance(); - case PAIMON: - return PaimonVendedCredentialsProvider.getInstance(); default: - // Other types do not support vendor credentials + // Other types either do not support vended credentials, or signal them via the + // MetastoreProperties.isVendedCredentialsEnabled() gate (e.g. Paimon REST) instead + // of a dedicated provider. return null; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProvider.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProvider.java deleted file mode 100644 index 0ea91a375c0aa9..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProvider.java +++ /dev/null @@ -1,77 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.datasource.credentials.AbstractVendedCredentialsProvider; -import org.apache.doris.datasource.property.metastore.MetastoreProperties; -import org.apache.doris.datasource.property.metastore.PaimonRestMetaStoreProperties; - -import com.google.common.collect.Maps; -import org.apache.paimon.rest.RESTToken; -import org.apache.paimon.rest.RESTTokenFileIO; -import org.apache.paimon.table.Table; - -import java.util.Map; - -public class PaimonVendedCredentialsProvider extends AbstractVendedCredentialsProvider { - private static final PaimonVendedCredentialsProvider INSTANCE = new PaimonVendedCredentialsProvider(); - - private PaimonVendedCredentialsProvider() { - // Singleton pattern - } - - public static PaimonVendedCredentialsProvider getInstance() { - return INSTANCE; - } - - @Override - public boolean isVendedCredentialsEnabled(MetastoreProperties metastoreProperties) { - // Paimon REST catalog always supports vended credentials if it's REST type - return metastoreProperties instanceof PaimonRestMetaStoreProperties; - } - - @Override - protected Map extractRawVendedCredentials(T tableObject) { - if (!(tableObject instanceof Table)) { - return Maps.newHashMap(); - } - - Table table = (Table) tableObject; - if (table.fileIO() == null || !(table.fileIO() instanceof RESTTokenFileIO)) { - return Maps.newHashMap(); - } - - RESTTokenFileIO restTokenFileIO = (RESTTokenFileIO) table.fileIO(); - RESTToken restToken = restTokenFileIO.validToken(); - Map tokens = restToken.token(); - - // Convert the original token to OSS format properties, let StorageProperties.createAll() further convert - Map rawProperties = Maps.newHashMap(); - rawProperties.putAll(tokens); - - return rawProperties; - } - - @Override - protected String getTableName(T tableObject) { - if (tableObject instanceof Table) { - return ((Table) tableObject).name(); - } - return super.getTableName(tableObject); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java index 44bee7fc03dfd5..029cd0534a4450 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java @@ -23,18 +23,10 @@ import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.foundation.property.ConnectorProperty; -import com.google.common.collect.ImmutableList; import lombok.Getter; -import org.apache.commons.lang3.StringUtils; -import org.apache.hadoop.conf.Configuration; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.options.CatalogOptions; -import org.apache.paimon.options.Options; -import java.util.HashMap; import java.util.List; import java.util.Map; -import java.util.concurrent.atomic.AtomicReference; public abstract class AbstractPaimonProperties extends MetastoreProperties { @ConnectorProperty( @@ -47,28 +39,18 @@ public abstract class AbstractPaimonProperties extends MetastoreProperties { protected ExecutionAuthenticator executionAuthenticator = new ExecutionAuthenticator() { }; - @Getter - protected Options catalogOptions; - - private final AtomicReference> catalogOptionsMapRef = new AtomicReference<>(); - public abstract String getPaimonCatalogType(); - private static final String USER_PROPERTY_PREFIX = "paimon."; - protected AbstractPaimonProperties(Map props) { super(Type.PAIMON, props); } - public abstract Catalog initializeCatalog(String catalogName, List storagePropertiesList); - /** - * Builds the HDFS Kerberos {@link ExecutionAuthenticator} from the catalog's storage properties, - * mirroring what {@code initializeCatalog} does for the legacy path. Shared by the filesystem and - * jdbc flavors' {@link #initExecutionAuthenticator} override so the plugin/cutover path wires a - * real {@code doAs} authenticator (the legacy {@code initializeCatalog} that did this is dead on - * that path). No-op when there is no HDFS storage (e.g. an S3-backed warehouse) — leaving the - * base no-op authenticator, which is correct (no Kerberos UGI to apply). + * Builds the HDFS Kerberos {@link ExecutionAuthenticator} from the catalog's storage properties. + * Shared by the filesystem and jdbc flavors' {@link #initExecutionAuthenticator} override so the + * plugin/cutover path wires a real {@code doAs} authenticator over Kerberized HDFS. No-op when + * there is no HDFS storage (e.g. an S3-backed warehouse) — leaving the base no-op authenticator, + * which is correct (no Kerberos UGI to apply). */ protected void initHdfsExecutionAuthenticator(List storagePropertiesList) { if (storagePropertiesList == null) { @@ -82,142 +64,4 @@ protected void initHdfsExecutionAuthenticator(List storagePro } } } - - protected void appendCatalogOptions() { - if (StringUtils.isNotBlank(warehouse)) { - catalogOptions.set(CatalogOptions.WAREHOUSE.key(), warehouse); - } - catalogOptions.set(CatalogOptions.METASTORE.key(), getMetastoreType()); - - // FIXME(cmy): Rethink these custom properties - origProps.forEach((k, v) -> { - if (k.toLowerCase().startsWith(USER_PROPERTY_PREFIX)) { - String newKey = k.substring(USER_PROPERTY_PREFIX.length()); - if (StringUtils.isNotBlank(newKey)) { - boolean excluded = userStoragePrefixes.stream().anyMatch(k::startsWith); - if (!excluded) { - catalogOptions.set(newKey, v); - } - } - } - }); - } - - /** - * Build catalog options including common and subclass-specific ones. - */ - public void buildCatalogOptions() { - catalogOptions = new Options(); - appendCatalogOptions(); - appendCustomCatalogOptions(); - } - - protected void appendUserHadoopConfig(Configuration conf) { - normalizeS3Config().forEach(conf::set); - } - - public Map getCatalogOptionsMap() { - // Return the cached map if already initialized - Map existing = catalogOptionsMapRef.get(); - if (existing != null) { - return existing; - } - - // Check that the catalog options source is available - if (catalogOptions == null) { - throw new IllegalStateException("Catalog options have not been initialized. Call" - + " buildCatalogOptions first."); - } - - // Construct the map manually using the provided keys - Map computed = new HashMap<>(); - for (String key : catalogOptions.keySet()) { - computed.put(key, catalogOptions.get(key)); - } - - // Attempt to set the constructed map atomically; only one thread wins - if (catalogOptionsMapRef.compareAndSet(null, computed)) { - return computed; - } else { - // Another thread already initialized it; return the existing one - return catalogOptionsMapRef.get(); - } - } - - /** - * @See org.apache.paimon.s3.S3FileIO - * Possible S3 config key prefixes: - * 1. "s3." - Paimon legacy custom prefix - * 2. "s3a." - Paimon-supported shorthand - * 3. "fs.s3a." - Hadoop S3A official prefix - * - * All of them are normalized to the Hadoop-recognized prefix "fs.s3a." - */ - private final List userStoragePrefixes = ImmutableList.of( - "paimon.s3.", "paimon.s3a.", "paimon.fs.s3.", "paimon.fs.oss." - ); - - /** Hadoop S3A standard prefix */ - private static final String FS_S3A_PREFIX = "fs.s3a."; - - /** - * Normalizes user-provided S3 config keys to Hadoop S3A keys - */ - protected Map normalizeS3Config() { - Map result = new HashMap<>(); - origProps.forEach((key, value) -> { - for (String prefix : userStoragePrefixes) { - if (key.startsWith(prefix)) { - result.put(FS_S3A_PREFIX + key.substring(prefix.length()), value); - return; // stop after the first matching prefix - } - } - }); - return result; - } - - - /** - * Hook method for subclasses to append metastore-specific or custom catalog options. - * - *

This method is invoked after common catalog options (e.g., warehouse path, - * metastore type, user-defined keys, and S3 compatibility mappings) have been - * added to the {@link org.apache.paimon.options.Options} instance. - * - *

Subclasses should override this method to inject additional configuration - * required for their specific metastore or environment. For example: - * - *

    - *
  • DLF-based catalog may require a custom metastore client class.
    • - *
  • HMS-based catalog may include URI and client pool parameters.
    • - *
  • Other environments may inject authentication, endpoint, or caching options.
    • - *
- * - *

If the subclass does not require any special options beyond the common ones, - * it can safely leave this method empty. - */ - protected abstract void appendCustomCatalogOptions(); - - /** - * Returns the metastore type identifier used by the Paimon catalog factory. - * - *

This identifier must match one of the known metastore types supported by - * Apache Paimon. Internally, the value returned here is used to configure the - * `metastore` option in {@code Options}, which determines the specific - * {@link org.apache.paimon.catalog.CatalogFactory} implementation to be used - * when instantiating the catalog. - * - *

You can find valid identifiers by reviewing implementations of the - * {@link org.apache.paimon.catalog.CatalogFactory} interface. Each implementation - * declares its identifier via a static {@code IDENTIFIER} field or equivalent constant. - * - *

Examples: - *

    - *
  • {@code "filesystem"} - for {@link org.apache.paimon.catalog.FileSystemCatalogFactory}
    • - *
  • {@code "hive"} - for {@link org.apache.paimon.hive.HiveCatalogFactory}
    • - *
- * - * @return the metastore type identifier string - */ - protected abstract String getMetastoreType(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/MetastoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/MetastoreProperties.java index 28310dfaa555a6..bf22f62d0095e1 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/MetastoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/MetastoreProperties.java @@ -157,4 +157,19 @@ public ExecutionAuthenticator getExecutionAuthenticator() { public void initExecutionAuthenticator(java.util.List storagePropertiesList) { // no-op by default } + + /** + * Whether this metastore supplies storage credentials by vending them per-table at scan time + * rather than from a static catalog-level storage map. When {@code true}, the catalog skips + * building the static {@link StorageProperties} map (a vended catalog — e.g. a Paimon REST + * catalog — has no static storage credentials by design; they arrive with each table token). + * + *

The default is {@code false} (use the static storage map). The Paimon REST flavor overrides + * it to {@code true}. This is the SDK-free replacement of the former + * {@code VendedCredentialsFactory} PAIMON type-switch (the Iceberg path still routes through its + * provider). Read by {@code CatalogProperty.initStorageProperties}.

+ */ + public boolean isVendedCredentialsEnabled() { + return false; + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java index b205257b8a3232..8a27c2927b4761 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStoreProperties.java @@ -17,17 +17,6 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.datasource.property.storage.StorageProperties; - -import com.aliyun.datalake.metastore.common.DataLakeConfig; -import com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient; -import org.apache.hadoop.hive.conf.HiveConf; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.CatalogFactory; -import org.apache.paimon.hive.HiveCatalogOptions; - -import java.util.List; import java.util.Map; /** @@ -42,7 +31,7 @@ *

Key Characteristics: *

    *
  • Internally uses HiveCatalog with custom HiveConf configured for Aliyun DLF.
    • - *
  • Relies on {@link ProxyMetaStoreClient} to bridge DLF compatibility.
    • + *
  • Relies on a DLF proxy metastore client to bridge DLF compatibility.
    • *
  • Requires Aliyun OSS as the storage backend. Other storage types are not * currently verified for compatibility.
    • *
@@ -50,15 +39,9 @@ *

Note: This is an internal extension and not an officially supported Paimon * metastore type. Future compatibility should be validated when upgrading Paimon * or changing storage backends. - * - * @see org.apache.paimon.hive.HiveCatalog - * @see org.apache.paimon.catalog.CatalogFactory - * @see ProxyMetaStoreClient */ public class PaimonAliyunDLFMetaStoreProperties extends AbstractPaimonProperties { - private AliyunDLFBaseProperties baseProperties; - protected PaimonAliyunDLFMetaStoreProperties(Map props) { super(props); } @@ -66,46 +49,10 @@ protected PaimonAliyunDLFMetaStoreProperties(Map props) { @Override public void initNormalizeAndCheckProps() { super.initNormalizeAndCheckProps(); - baseProperties = AliyunDLFBaseProperties.of(origProps); - } - - private HiveConf buildHiveConf() { - HiveConf hiveConf = new HiveConf(); - hiveConf.set(DataLakeConfig.CATALOG_ACCESS_KEY_ID, baseProperties.dlfAccessKey); - hiveConf.set(DataLakeConfig.CATALOG_ACCESS_KEY_SECRET, baseProperties.dlfSecretKey); - hiveConf.set(DataLakeConfig.CATALOG_ENDPOINT, baseProperties.dlfEndpoint); - hiveConf.set(DataLakeConfig.CATALOG_REGION_ID, baseProperties.dlfRegion); - hiveConf.set(DataLakeConfig.CATALOG_SECURITY_TOKEN, baseProperties.dlfSessionToken); - hiveConf.set(DataLakeConfig.CATALOG_USER_ID, baseProperties.dlfUid); - hiveConf.set(DataLakeConfig.CATALOG_ID, baseProperties.dlfCatalogId); - hiveConf.set(DataLakeConfig.CATALOG_PROXY_MODE, baseProperties.dlfProxyMode); - return hiveConf; - } - - @Override - public Catalog initializeCatalog(String catalogName, List storagePropertiesList) { - HiveConf hiveConf = buildHiveConf(); - buildCatalogOptions(); - StorageProperties ossProps = storagePropertiesList.stream() - .filter(sp -> sp.getType() == StorageProperties.Type.OSS - || sp.getType() == StorageProperties.Type.OSS_HDFS) - .findFirst() - .orElseThrow(() -> new IllegalStateException("Paimon DLF metastore requires OSS storage properties.")); - ossProps.getHadoopStorageConfig().forEach(entry -> hiveConf.set(entry.getKey(), entry.getValue())); - appendUserHadoopConfig(hiveConf); - CatalogContext catalogContext = CatalogContext.create(catalogOptions, hiveConf); - return CatalogFactory.createCatalog(catalogContext); - } - - @Override - protected void appendCustomCatalogOptions() { - catalogOptions.set("metastore.client.class", ProxyMetaStoreClient.class.getName()); - catalogOptions.set("client-pool-cache.keys", "conf:" + DataLakeConfig.CATALOG_ID); - } - - @Override - protected String getMetastoreType() { - return HiveCatalogOptions.IDENTIFIER; + // Validate the DLF properties: AliyunDLFBaseProperties.of(...) runs checkAndInit() and throws on + // missing dlf.access_key/dlf.secret_key/dlf.endpoint. The bound object is not retained because the + // catalog is now built connector-side — only the validation side effect is needed here. + AliyunDLFBaseProperties.of(origProps); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java index 5762f97a9082b6..3b0a73e6f85ebf 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStoreProperties.java @@ -17,16 +17,8 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.property.storage.HdfsProperties; import org.apache.doris.datasource.property.storage.StorageProperties; -import org.apache.hadoop.conf.Configuration; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.CatalogFactory; -import org.apache.paimon.catalog.FileSystemCatalogFactory; - import java.util.List; import java.util.Map; @@ -35,47 +27,16 @@ protected PaimonFileSystemMetaStoreProperties(Map props) { super(props); } - @Override - public Catalog initializeCatalog(String catalogName, List storagePropertiesList) { - buildCatalogOptions(); - Configuration conf = new Configuration(); - storagePropertiesList.forEach(storageProperties -> { - conf.addResource(storageProperties.getHadoopStorageConfig()); - if (storageProperties.getType().equals(StorageProperties.Type.HDFS)) { - this.executionAuthenticator = new HadoopExecutionAuthenticator(((HdfsProperties) storageProperties) - .getHadoopAuthenticator()); - } - }); - appendUserHadoopConfig(conf); - CatalogContext catalogContext = CatalogContext.create(catalogOptions, conf); - try { - return this.executionAuthenticator.execute(() -> CatalogFactory.createCatalog(catalogContext)); - } catch (Exception e) { - throw new RuntimeException(e); - } - } - /** - * Wires the HDFS Kerberos authenticator on the plugin/cutover path (rereview2 M-8). Legacy set - * it inside {@link #initializeCatalog}, which is dead on that path, so the runtime authenticator - * stayed the base no-op and {@code doAs} was silently lost over Kerberized HDFS. Mirrors HMS, - * which sets its authenticator in {@code initNormalizeAndCheckProps}. + * Wires the HDFS Kerberos authenticator on the plugin/cutover path (rereview2 M-8): the runtime + * authenticator would otherwise stay the base no-op and {@code doAs} would be silently lost over + * Kerberized HDFS. Mirrors HMS, which sets its authenticator in {@code initNormalizeAndCheckProps}. */ @Override public void initExecutionAuthenticator(List storagePropertiesList) { initHdfsExecutionAuthenticator(storagePropertiesList); } - @Override - protected void appendCustomCatalogOptions() { - //nothing need to do - } - - @Override - protected String getMetastoreType() { - return FileSystemCatalogFactory.IDENTIFIER; - } - @Override public String getPaimonCatalogType() { return "filesystem"; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java index f8d6404bdea367..fd8165097fc841 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStoreProperties.java @@ -18,17 +18,8 @@ package org.apache.doris.datasource.property.metastore; import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.foundation.property.ConnectorProperty; -import org.apache.commons.lang3.exception.ExceptionUtils; -import org.apache.hadoop.conf.Configuration; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.CatalogFactory; -import org.apache.paimon.hive.HiveCatalogOptions; - -import java.util.List; import java.util.Map; import java.util.concurrent.TimeUnit; @@ -68,48 +59,4 @@ public void initNormalizeAndCheckProps() { hmsBaseProperties = HMSBaseProperties.of(origProps); this.executionAuthenticator = new HadoopExecutionAuthenticator(hmsBaseProperties.getHmsAuthenticator()); } - - - /** - * Builds the Hadoop Configuration by adding hive-site.xml and storage-specific configs. - */ - private Configuration buildHiveConfiguration(List storagePropertiesList) { - Configuration conf = hmsBaseProperties.getHiveConf(); - - for (StorageProperties sp : storagePropertiesList) { - if (sp.getHadoopStorageConfig() != null) { - conf.addResource(sp.getHadoopStorageConfig()); - } - } - return conf; - } - - @Override - public Catalog initializeCatalog(String catalogName, List storagePropertiesList) { - buildCatalogOptions(); - Configuration conf = buildHiveConfiguration(storagePropertiesList); - appendUserHadoopConfig(conf); - CatalogContext catalogContext = CatalogContext.create(catalogOptions, conf); - try { - return executionAuthenticator.execute(() -> CatalogFactory.createCatalog(catalogContext)); - } catch (Exception e) { - throw new RuntimeException("Failed to create Paimon catalog with HMS metastore, msg: " - + ExceptionUtils.getRootCause(e), e); - } - - } - - @Override - protected String getMetastoreType() { - //See org.apache.paimon.hive.HiveCatalogFactory - return HiveCatalogOptions.IDENTIFIER; - } - - @Override - protected void appendCustomCatalogOptions() { - catalogOptions.set(CLIENT_POOL_CACHE_EVICTION_INTERVAL_MS_KEY, - String.valueOf(clientPoolCacheEvictionIntervalMs)); - catalogOptions.set(LOCATION_IN_PROPERTIES_KEY, String.valueOf(locationInProperties)); - catalogOptions.set("uri", hmsBaseProperties.getHiveMetastoreUri()); - } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java index a122dc020293dc..6b44c3cf545b07 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStoreProperties.java @@ -17,39 +17,15 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.catalog.JdbcResource; -import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.property.storage.HdfsProperties; import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.foundation.property.ConnectorProperty; import org.apache.commons.lang3.StringUtils; -import org.apache.hadoop.conf.Configuration; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.CatalogFactory; -import org.apache.paimon.jdbc.JdbcCatalogFactory; -import org.apache.paimon.options.CatalogOptions; -import java.net.MalformedURLException; -import java.net.URL; -import java.net.URLClassLoader; -import java.util.Collections; -import java.util.HashMap; import java.util.List; import java.util.Map; -import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; public class PaimonJdbcMetaStoreProperties extends AbstractPaimonProperties { - private static final Logger LOG = LogManager.getLogger(PaimonJdbcMetaStoreProperties.class); - private static final String JDBC_PREFIX = "jdbc."; - private static final String JDBC_DRIVER_URL = JDBC_PREFIX + JdbcResource.DRIVER_URL; - private static final String JDBC_DRIVER_CLASS = JDBC_PREFIX + JdbcResource.DRIVER_CLASS; - private static final Map DRIVER_CLASS_LOADER_CACHE = new ConcurrentHashMap<>(); - private static final Set REGISTERED_DRIVER_KEYS = ConcurrentHashMap.newKeySet(); @ConnectorProperty( names = {"uri", "paimon.jdbc.uri"}, @@ -107,169 +83,13 @@ protected void checkRequiredProperties() { } } - @Override - public Catalog initializeCatalog(String catalogName, List storagePropertiesList) { - buildCatalogOptions(); - Configuration conf = new Configuration(); - for (StorageProperties storageProperties : storagePropertiesList) { - if (storageProperties.getHadoopStorageConfig() != null) { - conf.addResource(storageProperties.getHadoopStorageConfig()); - } - if (storageProperties.getType().equals(StorageProperties.Type.HDFS)) { - this.executionAuthenticator = new HadoopExecutionAuthenticator(((HdfsProperties) storageProperties) - .getHadoopAuthenticator()); - } - } - appendUserHadoopConfig(conf); - if (StringUtils.isNotBlank(driverUrl)) { - registerJdbcDriver(driverUrl, driverClass); - LOG.info("Using dynamic JDBC driver for Paimon JDBC catalog from: {}", driverUrl); - } - CatalogContext catalogContext = CatalogContext.create(catalogOptions, conf); - try { - return this.executionAuthenticator.execute(() -> CatalogFactory.createCatalog(catalogContext)); - } catch (Exception e) { - throw new RuntimeException("Failed to create Paimon catalog with JDBC metastore: " + e.getMessage(), e); - } - } - /** - * Wires the HDFS Kerberos authenticator on the plugin/cutover path (rereview2 M-8). Legacy set - * it inside {@link #initializeCatalog}, which is dead on that path, so the runtime authenticator - * stayed the base no-op and {@code doAs} was silently lost over Kerberized HDFS. Mirrors HMS, - * which sets its authenticator in {@code initNormalizeAndCheckProps}. + * Wires the HDFS Kerberos authenticator on the plugin/cutover path (rereview2 M-8): the runtime + * authenticator would otherwise stay the base no-op and {@code doAs} would be silently lost over + * Kerberized HDFS. Mirrors HMS, which sets its authenticator in {@code initNormalizeAndCheckProps}. */ @Override public void initExecutionAuthenticator(List storagePropertiesList) { initHdfsExecutionAuthenticator(storagePropertiesList); } - - @Override - protected void appendCustomCatalogOptions() { - catalogOptions.set(CatalogOptions.URI.key(), uri); - addIfNotBlank("jdbc.user", jdbcUser); - addIfNotBlank("jdbc.password", jdbcPassword); - appendRawJdbcCatalogOptions(); - } - - @Override - protected String getMetastoreType() { - return JdbcCatalogFactory.IDENTIFIER; - } - - private void addIfNotBlank(String key, String value) { - if (StringUtils.isNotBlank(value)) { - catalogOptions.set(key, value); - } - } - - private void appendRawJdbcCatalogOptions() { - origProps.forEach((key, value) -> { - if (key != null && key.startsWith(JDBC_PREFIX) && !catalogOptions.keySet().contains(key)) { - catalogOptions.set(key, value); - } - }); - } - - public Map getBackendPaimonOptions() { - if (StringUtils.isBlank(driverUrl)) { - return Collections.emptyMap(); - } - if (StringUtils.isBlank(driverClass)) { - throw new IllegalArgumentException("jdbc.driver_class or paimon.jdbc.driver_class is required when " - + "jdbc.driver_url or paimon.jdbc.driver_url is specified"); - } - Map backendPaimonOptions = new HashMap<>(); - backendPaimonOptions.put(JDBC_DRIVER_URL, JdbcResource.getFullDriverUrl(driverUrl)); - backendPaimonOptions.put(JDBC_DRIVER_CLASS, driverClass); - return backendPaimonOptions; - } - - /** - * Register JDBC driver with DriverManager. - * This is necessary because DriverManager.getConnection() doesn't use Thread.contextClassLoader. - */ - private void registerJdbcDriver(String driverUrl, String driverClassName) { - try { - if (StringUtils.isBlank(driverClassName)) { - throw new IllegalArgumentException( - "jdbc.driver_class or paimon.jdbc.driver_class is required when jdbc.driver_url " - + "or paimon.jdbc.driver_url is specified"); - } - - String fullDriverUrl = JdbcResource.getFullDriverUrl(driverUrl); - URL url = new URL(fullDriverUrl); - String driverKey = fullDriverUrl + "#" + driverClassName; - if (!REGISTERED_DRIVER_KEYS.add(driverKey)) { - LOG.info("JDBC driver already registered for Paimon catalog: {} from {}", - driverClassName, fullDriverUrl); - return; - } - try { - ClassLoader classLoader = DRIVER_CLASS_LOADER_CACHE.computeIfAbsent(url, u -> { - ClassLoader parent = getClass().getClassLoader(); - return URLClassLoader.newInstance(new URL[] {u}, parent); - }); - Class loadedDriverClass = Class.forName(driverClassName, true, classLoader); - java.sql.Driver driver = (java.sql.Driver) loadedDriverClass.getDeclaredConstructor().newInstance(); - java.sql.DriverManager.registerDriver(new DriverShim(driver)); - LOG.info("Successfully registered JDBC driver for Paimon catalog: {} from {}", - driverClassName, fullDriverUrl); - } catch (ClassNotFoundException e) { - REGISTERED_DRIVER_KEYS.remove(driverKey); - throw new IllegalArgumentException("Failed to load JDBC driver class: " + driverClassName, e); - } catch (Exception e) { - REGISTERED_DRIVER_KEYS.remove(driverKey); - throw new RuntimeException("Failed to register JDBC driver: " + driverClassName, e); - } - } catch (MalformedURLException e) { - throw new IllegalArgumentException("Invalid driver URL: " + driverUrl, e); - } catch (IllegalArgumentException e) { - throw e; - } - } - - private static class DriverShim implements java.sql.Driver { - private final java.sql.Driver delegate; - - DriverShim(java.sql.Driver delegate) { - this.delegate = delegate; - } - - @Override - public java.sql.Connection connect(String url, java.util.Properties info) throws java.sql.SQLException { - return delegate.connect(url, info); - } - - @Override - public boolean acceptsURL(String url) throws java.sql.SQLException { - return delegate.acceptsURL(url); - } - - @Override - public java.sql.DriverPropertyInfo[] getPropertyInfo(String url, java.util.Properties info) - throws java.sql.SQLException { - return delegate.getPropertyInfo(url, info); - } - - @Override - public int getMajorVersion() { - return delegate.getMajorVersion(); - } - - @Override - public int getMinorVersion() { - return delegate.getMinorVersion(); - } - - @Override - public boolean jdbcCompliant() { - return delegate.jdbcCompliant(); - } - - @Override - public java.util.logging.Logger getParentLogger() throws java.sql.SQLFeatureNotSupportedException { - return delegate.getParentLogger(); - } - } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java index 48f9246150e165..fbc6103829b9ff 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStoreProperties.java @@ -17,22 +17,15 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.foundation.property.ConnectorProperty; import org.apache.doris.foundation.property.ParamRules; import lombok.Getter; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.CatalogFactory; -import java.util.List; import java.util.Map; public class PaimonRestMetaStoreProperties extends AbstractPaimonProperties { - private static final String PAIMON_REST_PROPERTY_PREFIX = "paimon.rest."; - @ConnectorProperty(names = {"paimon.rest.uri", "uri"}, description = "The uri of the Paimon rest catalog service.") private String paimonRestUri = ""; @@ -74,27 +67,15 @@ public String getPaimonCatalogType() { return "rest"; } + /** + * A Paimon REST catalog vends storage credentials per-table at scan time (via the connector's + * REST token path) and has no static catalog-level storage map, so the catalog skips building it. + * SDK-free replacement of the former {@code PaimonVendedCredentialsProvider.isVendedCredentialsEnabled} + * gate. Read by {@code CatalogProperty.initStorageProperties}. + */ @Override - public Catalog initializeCatalog(String catalogName, List storagePropertiesList) { - buildCatalogOptions(); - CatalogContext catalogContext = CatalogContext.create(catalogOptions); - return CatalogFactory.createCatalog(catalogContext); - } - - @Override - protected void appendCustomCatalogOptions() { - catalogOptions.set("uri", paimonRestUri); - for (Map.Entry entry : origProps.entrySet()) { - if (entry.getKey().startsWith(PAIMON_REST_PROPERTY_PREFIX)) { - String key = entry.getKey().substring(PAIMON_REST_PROPERTY_PREFIX.length()); - catalogOptions.set(key, entry.getValue()); - } - } - } - - @Override - protected String getMetastoreType() { - return "rest"; + public boolean isVendedCredentialsEnabled() { + return true; } private ParamRules buildRules() { diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/VendedCredentialsFactoryTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/VendedCredentialsFactoryTest.java index edde53ac8ae63d..a09b53025a0c4a 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/VendedCredentialsFactoryTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/VendedCredentialsFactoryTest.java @@ -19,7 +19,6 @@ import org.apache.doris.datasource.property.metastore.IcebergRestProperties; import org.apache.doris.datasource.property.metastore.MetastoreProperties; -import org.apache.doris.datasource.property.metastore.PaimonRestMetaStoreProperties; import org.apache.doris.datasource.property.storage.StorageProperties; import org.apache.doris.datasource.property.storage.StorageProperties.Type; @@ -62,25 +61,6 @@ public void testGetStoragePropertiesMapWithVendedCredentialsForIceberg() { Assertions.assertNotNull(result); } - @Test - public void testGetStoragePropertiesMapWithVendedCredentialsForPaimon() { - // Mock Paimon REST properties - PaimonRestMetaStoreProperties paimonProperties = Mockito.mock(PaimonRestMetaStoreProperties.class); - Mockito.when(paimonProperties.getType()).thenReturn(MetastoreProperties.Type.PAIMON); - Mockito.when(paimonProperties.getTokenProvider()).thenReturn("dlf"); - - // Mock Paimon table - org.apache.paimon.table.Table paimonTable = Mockito.mock(org.apache.paimon.table.Table.class); - - Map baseStorageMap = new HashMap<>(); - - Map result = VendedCredentialsFactory - .getStoragePropertiesMapWithVendedCredentials(paimonProperties, baseStorageMap, paimonTable); - - // Should return the result from PaimonVendedCredentialsProvider or fall back to base map - Assertions.assertNotNull(result); - } - @Test public void testGetStoragePropertiesMapWithVendedCredentialsForUnsupportedType() { // Mock unsupported metastore type (e.g., HMS) @@ -197,17 +177,5 @@ public void testGetProviderTypeReturnsCorrectProvider() { // Should use IcebergVendedCredentialsProvider Assertions.assertNotNull(result1); - - // Test Paimon type - PaimonRestMetaStoreProperties paimonProperties = Mockito.mock(PaimonRestMetaStoreProperties.class); - Mockito.when(paimonProperties.getType()).thenReturn(MetastoreProperties.Type.PAIMON); - - org.apache.paimon.table.Table paimonTable = Mockito.mock(org.apache.paimon.table.Table.class); - - Map result2 = VendedCredentialsFactory - .getStoragePropertiesMapWithVendedCredentials(paimonProperties, new HashMap<>(), paimonTable); - - // Should use PaimonVendedCredentialsProvider - Assertions.assertNotNull(result2); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProviderTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProviderTest.java deleted file mode 100644 index d672d69045e401..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonVendedCredentialsProviderTest.java +++ /dev/null @@ -1,349 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.paimon; - -import org.apache.doris.datasource.credentials.CredentialUtils; -import org.apache.doris.datasource.credentials.VendedCredentialsFactory; -import org.apache.doris.datasource.property.metastore.MetastoreProperties; -import org.apache.doris.datasource.property.metastore.PaimonRestMetaStoreProperties; -import org.apache.doris.datasource.property.storage.StorageProperties; -import org.apache.doris.datasource.property.storage.StorageProperties.Type; - -import org.apache.paimon.rest.RESTToken; -import org.apache.paimon.rest.RESTTokenFileIO; -import org.apache.paimon.table.Table; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; -import org.mockito.Mockito; - -import java.util.HashMap; -import java.util.Map; - -public class PaimonVendedCredentialsProviderTest { - - @Test - public void testIsVendedCredentialsEnabled() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - // Test with PaimonRestMetaStore and DLF token provider - PaimonRestMetaStoreProperties restProperties = Mockito.mock(PaimonRestMetaStoreProperties.class); - Mockito.when(restProperties.getType()).thenReturn(MetastoreProperties.Type.PAIMON); - Mockito.when(restProperties.getTokenProvider()).thenReturn("dlf"); - - Assertions.assertTrue(provider.isVendedCredentialsEnabled(restProperties)); - - // Test with PaimonRestMetaStore but unsupported token provider - // Note: PaimonVendedCredentialsProvider enables vended credentials for all PaimonRestMetaStore - // regardless of token provider, actual provider check happens later - Mockito.when(restProperties.getTokenProvider()).thenReturn("unsupported"); - Assertions.assertTrue(provider.isVendedCredentialsEnabled(restProperties)); - - // Test with non-PaimonRest metastore - MetastoreProperties nonRestProperties = Mockito.mock(MetastoreProperties.class); - Mockito.when(nonRestProperties.getType()).thenReturn(MetastoreProperties.Type.HMS); - Assertions.assertFalse(provider.isVendedCredentialsEnabled(nonRestProperties)); - } - - @Test - public void testExtractRawVendedCredentials() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - // Mock table with OSS vended credentials - Table table = Mockito.mock(Table.class); - RESTTokenFileIO restTokenFileIO = Mockito.mock(RESTTokenFileIO.class); - RESTToken restToken = Mockito.mock(RESTToken.class); - - Map tokenMap = new HashMap<>(); - tokenMap.put("fs.oss.accessKeyId", "STS.testAccessKey123"); - tokenMap.put("fs.oss.accessKeySecret", "testSecretKey456"); - tokenMap.put("fs.oss.securityToken", "testSessionToken789"); - tokenMap.put("fs.oss.endpoint", "oss-cn-beijing.aliyuncs.com"); - - Mockito.when(table.fileIO()).thenReturn(restTokenFileIO); - Mockito.when(table.name()).thenReturn("test_table"); - Mockito.when(restTokenFileIO.validToken()).thenReturn(restToken); - Mockito.when(restToken.token()).thenReturn(tokenMap); - - Map rawCredentials = provider.extractRawVendedCredentials(table); - - Assertions.assertEquals("STS.testAccessKey123", rawCredentials.get("fs.oss.accessKeyId")); - Assertions.assertEquals("testSecretKey456", rawCredentials.get("fs.oss.accessKeySecret")); - Assertions.assertEquals("testSessionToken789", rawCredentials.get("fs.oss.securityToken")); - Assertions.assertEquals("oss-cn-beijing.aliyuncs.com", rawCredentials.get("fs.oss.endpoint")); - } - - @Test - public void testExtractRawVendedCredentialsWithNullTable() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - Map rawCredentials = provider.extractRawVendedCredentials(null); - Assertions.assertTrue(rawCredentials.isEmpty()); - } - - @Test - public void testExtractRawVendedCredentialsWithNullFileIO() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - Table table = Mockito.mock(Table.class); - Mockito.when(table.fileIO()).thenReturn(null); - - Map rawCredentials = provider.extractRawVendedCredentials(table); - Assertions.assertTrue(rawCredentials.isEmpty()); - } - - @Test - public void testExtractRawVendedCredentialsWithNonRESTTokenFileIO() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - Table table = Mockito.mock(Table.class); - // Mock a different FileIO type that's not RESTTokenFileIO - Mockito.when(table.fileIO()).thenReturn(Mockito.mock(org.apache.paimon.fs.FileIO.class)); - - Map rawCredentials = provider.extractRawVendedCredentials(table); - Assertions.assertTrue(rawCredentials.isEmpty()); - } - - @Test - public void testExtractRawVendedCredentialsWithEmptyToken() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - Table table = Mockito.mock(Table.class); - RESTTokenFileIO restTokenFileIO = Mockito.mock(RESTTokenFileIO.class); - RESTToken restToken = Mockito.mock(RESTToken.class); - - Mockito.when(table.fileIO()).thenReturn(restTokenFileIO); - Mockito.when(table.name()).thenReturn("test_table"); - Mockito.when(restTokenFileIO.validToken()).thenReturn(restToken); - Mockito.when(restToken.token()).thenReturn(new HashMap<>()); - - Map rawCredentials = provider.extractRawVendedCredentials(table); - Assertions.assertTrue(rawCredentials.isEmpty()); - } - - @Test - public void testExtractRawVendedCredentialsWithPartialOSSCredentials() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - Table table = Mockito.mock(Table.class); - RESTTokenFileIO restTokenFileIO = Mockito.mock(RESTTokenFileIO.class); - RESTToken restToken = Mockito.mock(RESTToken.class); - - Map tokenMap = new HashMap<>(); - tokenMap.put("fs.oss.accessKeyId", "testAccessKey"); - tokenMap.put("fs.oss.accessKeySecret", "testSecretKey"); - // Missing endpoint and session token - - Mockito.when(table.fileIO()).thenReturn(restTokenFileIO); - Mockito.when(table.name()).thenReturn("test_table"); - Mockito.when(restTokenFileIO.validToken()).thenReturn(restToken); - Mockito.when(restToken.token()).thenReturn(tokenMap); - - Map rawCredentials = provider.extractRawVendedCredentials(table); - - Assertions.assertEquals("testAccessKey", rawCredentials.get("fs.oss.accessKeyId")); - Assertions.assertEquals("testSecretKey", rawCredentials.get("fs.oss.accessKeySecret")); - Assertions.assertFalse(rawCredentials.containsKey("fs.oss.securityToken")); - Assertions.assertFalse(rawCredentials.containsKey("fs.oss.endpoint")); - } - - @Test - public void testFilterCloudStoragePropertiesWithOSS() { - Map rawCredentials = new HashMap<>(); - rawCredentials.put("oss.access-key-id", "testAccessKey"); - rawCredentials.put("oss.secret-access-key", "testSecretKey"); - rawCredentials.put("oss.endpoint", "oss-cn-beijing.aliyuncs.com"); - rawCredentials.put("paimon.table.name", "test_table"); - rawCredentials.put("other.property", "other_value"); - - Map filtered = CredentialUtils.filterCloudStorageProperties(rawCredentials); - - Assertions.assertEquals(3, filtered.size()); - Assertions.assertEquals("testAccessKey", filtered.get("oss.access-key-id")); - Assertions.assertEquals("testSecretKey", filtered.get("oss.secret-access-key")); - Assertions.assertEquals("oss-cn-beijing.aliyuncs.com", filtered.get("oss.endpoint")); - Assertions.assertFalse(filtered.containsKey("paimon.table.name")); - Assertions.assertFalse(filtered.containsKey("other.property")); - } - - @Test - public void testGetStoragePropertiesMapWithVendedCredentials() { - // Mock metastore properties with DLF token provider - PaimonRestMetaStoreProperties restProperties = Mockito.mock(PaimonRestMetaStoreProperties.class); - Mockito.when(restProperties.getType()).thenReturn(MetastoreProperties.Type.PAIMON); - Mockito.when(restProperties.getTokenProvider()).thenReturn("dlf"); - - // Mock table with vended credentials - Table table = Mockito.mock(Table.class); - RESTTokenFileIO restTokenFileIO = Mockito.mock(RESTTokenFileIO.class); - RESTToken restToken = Mockito.mock(RESTToken.class); - - Map tokenMap = new HashMap<>(); - tokenMap.put("fs.oss.accessKeyId", "STS.testAccessKey123"); - tokenMap.put("fs.oss.accessKeySecret", "testSecretKey456"); - tokenMap.put("fs.oss.securityToken", "testSessionToken789"); - tokenMap.put("fs.oss.endpoint", "oss-cn-beijing.aliyuncs.com"); - - Mockito.when(table.fileIO()).thenReturn(restTokenFileIO); - Mockito.when(table.name()).thenReturn("test_table"); - Mockito.when(restTokenFileIO.validToken()).thenReturn(restToken); - Mockito.when(restToken.token()).thenReturn(tokenMap); - - // Test using VendedCredentialsFactory - Map result = VendedCredentialsFactory - .getStoragePropertiesMapWithVendedCredentials(restProperties, new HashMap<>(), table); - - // Should not be null (assuming StorageProperties.createAll works correctly) - // Note: The actual result depends on whether StorageProperties.createAll() can properly map the credentials - // This test verifies the integration flow works without exceptions - Assertions.assertNotNull(result); - } - - @Test - public void testGetStoragePropertiesMapWithVendedCredentialsDisabled() { - // Mock metastore properties with unsupported token provider - PaimonRestMetaStoreProperties restProperties = Mockito.mock(PaimonRestMetaStoreProperties.class); - Mockito.when(restProperties.getType()).thenReturn(MetastoreProperties.Type.PAIMON); - Mockito.when(restProperties.getTokenProvider()).thenReturn("unsupported"); - - Table table = Mockito.mock(Table.class); - - Map result = VendedCredentialsFactory - .getStoragePropertiesMapWithVendedCredentials(restProperties, new HashMap<>(), table); - - // Should return the baseStoragePropertiesMap (empty HashMap) - Assertions.assertNotNull(result); - Assertions.assertTrue(result.isEmpty()); - } - - @Test - public void testGetStoragePropertiesMapWithNullTable() { - PaimonRestMetaStoreProperties restProperties = Mockito.mock(PaimonRestMetaStoreProperties.class); - Mockito.when(restProperties.getType()).thenReturn(MetastoreProperties.Type.PAIMON); - Mockito.when(restProperties.getTokenProvider()).thenReturn("dlf"); - - Map result = VendedCredentialsFactory - .getStoragePropertiesMapWithVendedCredentials(restProperties, new HashMap<>(), null); - - // Should return the baseStoragePropertiesMap (empty HashMap) - Assertions.assertNotNull(result); - Assertions.assertTrue(result.isEmpty()); - } - - @Test - public void testGetStoragePropertiesMapWithNonPaimonRest() { - // Test with non-PaimonRest metastore - MetastoreProperties nonRestProperties = Mockito.mock(MetastoreProperties.class); - Mockito.when(nonRestProperties.getType()).thenReturn(MetastoreProperties.Type.HMS); - Table table = Mockito.mock(Table.class); - - Map result = VendedCredentialsFactory - .getStoragePropertiesMapWithVendedCredentials(nonRestProperties, new HashMap<>(), table); - - // Should return the baseStoragePropertiesMap (empty HashMap) - Assertions.assertNotNull(result); - Assertions.assertTrue(result.isEmpty()); - } - - @Test - public void testGetBackendPropertiesFromStorageMapWithOSS() { - // Create mock storage properties - StorageProperties ossProperties = Mockito.mock(StorageProperties.class); - StorageProperties hdfsProperties = Mockito.mock(StorageProperties.class); - - Map ossBackendProps = new HashMap<>(); - ossBackendProps.put("AWS_ACCESS_KEY", "testOssAccessKey"); - ossBackendProps.put("AWS_SECRET_KEY", "testOssSecretKey"); - ossBackendProps.put("AWS_TOKEN", "testOssToken"); - ossBackendProps.put("AWS_ENDPOINT", "oss-cn-beijing.aliyuncs.com"); - - Map hdfsBackendProps = new HashMap<>(); - hdfsBackendProps.put("HDFS_PROPERTY", "hdfsValue"); - - Mockito.when(ossProperties.getBackendConfigProperties()).thenReturn(ossBackendProps); - Mockito.when(hdfsProperties.getBackendConfigProperties()).thenReturn(hdfsBackendProps); - - Map storagePropertiesMap = new HashMap<>(); - storagePropertiesMap.put(Type.OSS, ossProperties); - storagePropertiesMap.put(Type.HDFS, hdfsProperties); - - Map result = CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap); - - Assertions.assertEquals(5, result.size()); - Assertions.assertEquals("testOssAccessKey", result.get("AWS_ACCESS_KEY")); - Assertions.assertEquals("testOssSecretKey", result.get("AWS_SECRET_KEY")); - Assertions.assertEquals("testOssToken", result.get("AWS_TOKEN")); - Assertions.assertEquals("oss-cn-beijing.aliyuncs.com", result.get("AWS_ENDPOINT")); - Assertions.assertEquals("hdfsValue", result.get("HDFS_PROPERTY")); - } - - @Test - public void testGetBackendPropertiesFromStorageMapWithNullValues() { - StorageProperties ossProperties = Mockito.mock(StorageProperties.class); - - Map ossBackendProps = new HashMap<>(); - ossBackendProps.put("AWS_ACCESS_KEY", "testAccessKey"); - ossBackendProps.put("AWS_SECRET_KEY", null); // null value should be filtered out - ossBackendProps.put("AWS_TOKEN", "testToken"); - - Mockito.when(ossProperties.getBackendConfigProperties()).thenReturn(ossBackendProps); - - Map storagePropertiesMap = new HashMap<>(); - storagePropertiesMap.put(Type.OSS, ossProperties); - - Map result = CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap); - - Assertions.assertEquals(2, result.size()); - Assertions.assertEquals("testAccessKey", result.get("AWS_ACCESS_KEY")); - Assertions.assertEquals("testToken", result.get("AWS_TOKEN")); - Assertions.assertFalse(result.containsKey("AWS_SECRET_KEY")); - } - - @Test - public void testEndpointToRegionConversion() { - PaimonVendedCredentialsProvider provider = PaimonVendedCredentialsProvider.getInstance(); - - // Test different OSS endpoint patterns and their expected regions - String[] endpoints = { - "oss-cn-beijing.aliyuncs.com", - "oss-cn-shanghai.aliyuncs.com", - "oss-us-west-1.aliyuncs.com", - "oss-ap-southeast-1.aliyuncs.com" - }; - - for (int i = 0; i < endpoints.length; i++) { - Table table = Mockito.mock(Table.class); - RESTTokenFileIO restTokenFileIO = Mockito.mock(RESTTokenFileIO.class); - RESTToken restToken = Mockito.mock(RESTToken.class); - - Map tokenMap = new HashMap<>(); - tokenMap.put("fs.oss.accessKeyId", "testAccessKey"); - tokenMap.put("fs.oss.accessKeySecret", "testSecretKey"); - tokenMap.put("fs.oss.endpoint", endpoints[i]); - - Mockito.when(table.fileIO()).thenReturn(restTokenFileIO); - Mockito.when(table.name()).thenReturn("test_table"); - Mockito.when(restTokenFileIO.validToken()).thenReturn(restToken); - Mockito.when(restToken.token()).thenReturn(tokenMap); - - Map rawCredentials = provider.extractRawVendedCredentials(table); - - Assertions.assertEquals(endpoints[i], rawCredentials.get("fs.oss.endpoint")); - // Note: Current implementation doesn't convert endpoint to region, so region is not set - } - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/AbstractPaimonPropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/AbstractPaimonPropertiesTest.java deleted file mode 100644 index e5a775ba6e3ef2..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/AbstractPaimonPropertiesTest.java +++ /dev/null @@ -1,89 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.property.metastore; - -import org.apache.doris.datasource.property.storage.StorageProperties; - -import org.apache.paimon.catalog.Catalog; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; - -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -public class AbstractPaimonPropertiesTest { - - private static class TestPaimonProperties extends AbstractPaimonProperties { - - - protected TestPaimonProperties(Map props) { - super(props); - } - - @Override - public String getPaimonCatalogType() { - return "test"; - } - - @Override - public Catalog initializeCatalog(String catalogName, List storagePropertiesList) { - return null; - } - - @Override - protected void appendCustomCatalogOptions() { - - } - - @Override - protected String getMetastoreType() { - return "test"; - } - } - - TestPaimonProperties props; - - @BeforeEach - void setup() { - Map input = new HashMap<>(); - input.put("warehouse", "s3://tmp/warehouse"); - input.put("paimon.metastore", "filesystem"); - input.put("paimon.s3.access-key", "AK"); - input.put("paimon.s3.secret-key", "SK"); - input.put("paimon.custom.key", "value"); - props = new TestPaimonProperties(input); - } - - @Test - void testNormalizeS3Config() { - Map input = new HashMap<>(); - input.put("paimon.s3.list.version", "1"); - input.put("paimon.s3.paging.maximum", "100"); - input.put("paimon.fs.s3.read.ahead.buffer.size", "1"); - input.put("paimon.s3a.replication.factor", "3"); - TestPaimonProperties testProps = new TestPaimonProperties(input); - Map result = testProps.normalizeS3Config(); - Assertions.assertTrue("1".equals(result.get("fs.s3a.list.version"))); - Assertions.assertTrue("100".equals(result.get("fs.s3a.paging.maximum"))); - Assertions.assertTrue("1".equals(result.get("fs.s3a.read.ahead.buffer.size"))); - Assertions.assertTrue("3".equals(result.get("fs.s3a.replication.factor"))); - } - -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStorePropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStorePropertiesTest.java index 1e02de6a5a43a5..490cbf7a97f652 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStorePropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonAliyunDLFMetaStorePropertiesTest.java @@ -17,21 +17,10 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.property.storage.StorageProperties; - -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.CatalogFactory; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; -import org.mockito.MockedStatic; -import org.mockito.Mockito; -import java.util.ArrayList; -import java.util.Collections; import java.util.HashMap; -import java.util.List; import java.util.Map; public class PaimonAliyunDLFMetaStorePropertiesTest { @@ -62,136 +51,5 @@ void testInitNormalizeAndCheckProps() { dlfProps.getPaimonCatalogType(), "Catalog type should be PAIMON_DLF" ); - Assertions.assertEquals( - "hive", - dlfProps.getMetastoreType(), - "Metastore type should be hive" - ); - } - - @Test - void testInitializeCatalogWithValidOssProperties() throws UserException { - Map props = createValidProps(); - PaimonAliyunDLFMetaStoreProperties dlfProps = - new PaimonAliyunDLFMetaStoreProperties(props); - dlfProps.initNormalizeAndCheckProps(); - - // Prepare OSSProperties mock - Map ossProps = new HashMap<>(); - ossProps.put("oss.access_key", "ak"); - ossProps.put("oss.secret_key", "sk"); - ossProps.put("oss.endpoint", "oss-cn-hangzhou.aliyuncs.com"); - - - List storageProperties = StorageProperties.createAll(ossProps); - - Catalog mockCatalog = Mockito.mock(Catalog.class); - - try (MockedStatic mocked = Mockito.mockStatic(CatalogFactory.class)) { - mocked.when(() -> CatalogFactory.createCatalog(Mockito.any(CatalogContext.class))) - .thenReturn(mockCatalog); - - Catalog catalog = dlfProps.initializeCatalog("testCatalog", storageProperties); - - Assertions.assertNotNull(catalog, "Catalog should not be null"); - Assertions.assertEquals(mockCatalog, catalog, "Catalog should be the mocked one"); - - mocked.verify(() -> CatalogFactory.createCatalog(Mockito.any(CatalogContext.class))); - } - } - - - @Test - void testInitializeCatalogWithValidOssHdfsProperties() throws UserException { - Map props = createValidProps(); - PaimonAliyunDLFMetaStoreProperties dlfProps = - new PaimonAliyunDLFMetaStoreProperties(props); - dlfProps.initNormalizeAndCheckProps(); - - // Prepare OSSProperties mock - Map ossProps = new HashMap<>(); - ossProps.put("dlf.access_key", "ak"); - ossProps.put("dlf.secret_key", "sk"); - ossProps.put("dlf.endpoint", "dlf-vpc.cn-beijing.aliyuncs.com"); - ossProps.put("dlf.region", "cn-beijing"); - ossProps.put("oss.hdfs.enabled", "true"); - - - List storageProperties = StorageProperties.createAll(ossProps); - - Catalog mockCatalog = Mockito.mock(Catalog.class); - - try (MockedStatic mocked = Mockito.mockStatic(CatalogFactory.class)) { - mocked.when(() -> CatalogFactory.createCatalog(Mockito.any(CatalogContext.class))) - .thenReturn(mockCatalog); - - Catalog catalog = dlfProps.initializeCatalog("testCatalog", storageProperties); - - Assertions.assertNotNull(catalog, "Catalog should not be null"); - Assertions.assertEquals(mockCatalog, catalog, "Catalog should be the mocked one"); - - mocked.verify(() -> CatalogFactory.createCatalog(Mockito.any(CatalogContext.class))); - } - ossProps = new HashMap<>(); - ossProps.put("dlf.access_key", "ak"); - ossProps.put("dlf.secret_key", "sk"); - ossProps.put("dlf.endpoint", "dlf-vpc.cn-beijing.aliyuncs.com"); - ossProps.put("dlf.region", "cn-beijing"); - ossProps.put("oss.access_key", "ak"); - ossProps.put("oss.secret_key", "sk"); - ossProps.put("oss.endpoint", "oss-cn-beijing.oss-dls.aliyuncs.com"); - storageProperties = StorageProperties.createAll(ossProps); - - mockCatalog = Mockito.mock(Catalog.class); - - try (MockedStatic mocked = Mockito.mockStatic(CatalogFactory.class)) { - mocked.when(() -> CatalogFactory.createCatalog(Mockito.any(CatalogContext.class))) - .thenReturn(mockCatalog); - - Catalog catalog = dlfProps.initializeCatalog("testCatalog", storageProperties); - - Assertions.assertNotNull(catalog, "Catalog should not be null"); - Assertions.assertEquals(mockCatalog, catalog, "Catalog should be the mocked one"); - - mocked.verify(() -> CatalogFactory.createCatalog(Mockito.any(CatalogContext.class))); - } - - } - - @Test - void testInitializeCatalogWithoutOssPropertiesThrows() { - Map props = createValidProps(); - PaimonAliyunDLFMetaStoreProperties dlfProps = - new PaimonAliyunDLFMetaStoreProperties(props); - dlfProps.initNormalizeAndCheckProps(); - - List storageProperties = new ArrayList<>(); // No OSS properties - - IllegalStateException ex = Assertions.assertThrows( - IllegalStateException.class, - () -> dlfProps.initializeCatalog("testCatalog", storageProperties) - ); - - Assertions.assertTrue(ex.getMessage().contains("OSS storage properties")); - } - - @Test - void testInitializeCatalogWithNonOssTypeThrows() { - Map props = createValidProps(); - PaimonAliyunDLFMetaStoreProperties dlfProps = - new PaimonAliyunDLFMetaStoreProperties(props); - dlfProps.initNormalizeAndCheckProps(); - - StorageProperties nonOssProps = Mockito.mock(StorageProperties.class); - Mockito.when(nonOssProps.getType()).thenReturn(StorageProperties.Type.HDFS); - - List storageProperties = Collections.singletonList(nonOssProps); - - IllegalStateException ex = Assertions.assertThrows( - IllegalStateException.class, - () -> dlfProps.initializeCatalog("testCatalog", storageProperties) - ); - - Assertions.assertTrue(ex.getMessage().contains("Paimon DLF metastore requires OSS storage properties.")); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonCatalogTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonCatalogTest.java deleted file mode 100644 index 85633008a7145b..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonCatalogTest.java +++ /dev/null @@ -1,94 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.property.metastore; - -import org.apache.doris.datasource.property.storage.StorageProperties; - -import org.apache.paimon.catalog.Catalog; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Disabled; -import org.junit.jupiter.api.Test; - -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -@Disabled("only used for your local test") -public class PaimonCatalogTest { - @Test - public void testNameSpace() throws Exception { - Map pa = new HashMap<>(); - pa.put("type", "paimon"); - pa.put("paimon.catalog.type", "hms"); - pa.put("hive.metastore.uris", "thrift://172.20.48.119:9383"); - pa.put("warehouse", "s3a://doris/paimon_warehouse"); - pa.put("s3.region", "ap-east-1"); - - // User must provide real Access Key / Secret Key to enable initialization - pa.put("s3.access_key", ""); - pa.put("s3.secret_key", ""); - pa.put("s3.endpoint", "s3.ap-east-1.amazonaws.com"); - - Catalog catalog = initCatalog(pa); - if (catalog != null) { - catalog.listDatabases().forEach(System.out::println); - } - } - - /** - * Initializes a Paimon HMS Catalog. - *

- * Initialization is skipped by default. Users must provide valid S3 - * Access Key and Secret Key in the configuration map to enable it. - *

- * Steps: - * 1. Validate that credentials are provided. - * 2. Normalize and check metastore properties. - * 3. Create storage properties. - * 4. Initialize and return the Catalog instance. - * - * @param params A map containing the configuration parameters. - * @return Catalog instance if initialized, or {@code null} if skipped. - * @throws Exception If initialization fails. - */ - private Catalog initCatalog(Map params) throws Exception { - if (isDisabled(params)) { - System.out.println("Catalog initialization skipped: Missing valid S3 Access Key/Secret Key."); - return null; - } - AbstractPaimonProperties metaStoreProps = - (AbstractPaimonProperties) MetastoreProperties.create(params); - metaStoreProps.initNormalizeAndCheckProps(); - Assertions.assertNotNull(metaStoreProps.getExecutionAuthenticator()); - List storageProps = StorageProperties.createAll(params); - - return metaStoreProps.initializeCatalog("paimon_catalog", storageProps); - } - - /** - * Checks if initialization should be skipped due to missing credentials. - * - * @param params The configuration parameters. - * @return {@code true} if missing AK/SK, {@code false} otherwise. - */ - private boolean isDisabled(Map params) { - String ak = params.get("s3.access_key"); - String sk = params.get("s3.secret_key"); - return ak == null || ak.isEmpty() || sk == null || sk.isEmpty(); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonDlfRestCatalogTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonDlfRestCatalogTest.java deleted file mode 100644 index ce317382606b9a..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonDlfRestCatalogTest.java +++ /dev/null @@ -1,243 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.property.metastore; - -import org.apache.doris.common.UserException; -import org.apache.doris.common.util.S3URI; -import org.apache.doris.common.util.Util; - -import com.amazonaws.ClientConfiguration; -import com.amazonaws.auth.AWSStaticCredentialsProvider; -import com.amazonaws.auth.BasicSessionCredentials; -import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration; -import com.amazonaws.services.s3.AmazonS3; -import com.amazonaws.services.s3.AmazonS3ClientBuilder; -import com.amazonaws.services.s3.model.GetObjectRequest; -import com.amazonaws.services.s3.model.S3Object; -import org.apache.hadoop.hive.conf.HiveConf; -import org.apache.paimon.catalog.Catalog.DatabaseNotExistException; -import org.apache.paimon.catalog.Catalog.TableNotExistException; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.CatalogFactory; -import org.apache.paimon.catalog.Database; -import org.apache.paimon.fs.FileIO; -import org.apache.paimon.options.Options; -import org.apache.paimon.rest.RESTToken; -import org.apache.paimon.rest.RESTTokenFileIO; -import org.apache.paimon.table.source.DataSplit; -import org.apache.paimon.table.source.RawFile; -import org.apache.paimon.table.source.ReadBuilder; -import org.apache.paimon.table.source.Split; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Disabled; -import org.junit.jupiter.api.Test; -import software.amazon.awssdk.auth.credentials.AwsSessionCredentials; -import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; -import software.amazon.awssdk.core.ResponseInputStream; -import software.amazon.awssdk.regions.Region; -import software.amazon.awssdk.services.s3.S3Client; -import software.amazon.awssdk.services.s3.S3Configuration; - -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.net.URI; -import java.nio.charset.StandardCharsets; -import java.util.List; -import java.util.Map; -import java.util.Optional; - -@Disabled("set aliyun access key, secret key before running the test") -public class PaimonDlfRestCatalogTest { - - private String aliyunAk = ""; - private String aliyunSk = ""; - - @Test - public void testPaimonDlfRestCatalog() throws DatabaseNotExistException, TableNotExistException, UserException { - org.apache.paimon.catalog.Catalog catalog = initPaimonDlfRestCatalog(); - System.out.println(catalog); - List dbs = catalog.listDatabases(); - for (String dbName : dbs) { - System.out.println("test debug get db: " + dbName); - Database db = catalog.getDatabase(dbName); - System.out.println("test debug get db instance: " + db.name() + ", " + db.options() + ", " + db.comment()); - List tables = catalog.listTables(dbName); - for (String tblName : tables) { - System.out.println("test debug get table: " + tblName); - if (!tblName.equalsIgnoreCase("users_samples")) { - continue; - } - org.apache.paimon.table.Table table = catalog.getTable( - org.apache.paimon.catalog.Identifier.create(dbName, tblName)); - System.out.println("test debug get table instance: " + table.name() + ", " + table.options() + ", " - + table.comment()); - - FileIO fileIO = table.fileIO(); - if (fileIO instanceof RESTTokenFileIO) { - System.out.println("test debug get file io instance: " + fileIO.getClass().getName()); - RESTTokenFileIO restTokenFileIO = (RESTTokenFileIO) fileIO; - RESTToken restToken = restTokenFileIO.validToken(); - Map tokens = restToken.token(); - for (Map.Entry kv : tokens.entrySet()) { - System.out.println("test debug get token: " + kv.getKey() + ", " + kv.getValue()); - } - // String accType = tokens.get("fs.oss.token.access.type"); - String tmpAk = tokens.get("fs.oss.accessKeyId"); - String tmpSk = tokens.get("fs.oss.accessKeySecret"); - String stsToken = tokens.get("fs.oss.securityToken"); - String endpoint = tokens.get("fs.oss.endpoint"); - - ReadBuilder readBuilder = table.newReadBuilder(); - List paimonSplits = readBuilder.newScan().plan().splits(); - for (Split split : paimonSplits) { - System.out.println("test debug get split: " + split); - if (split instanceof DataSplit) { - DataSplit dataSplit = (DataSplit) split; - Optional> rawFiles = dataSplit.convertToRawFiles(); - if (rawFiles.isPresent()) { - for (RawFile rawFile : rawFiles.get()) { - System.out.println("test debug get raw file: " + rawFile.path()); - readByAwsSdkV1(rawFile.path(), tmpAk, tmpSk, stsToken, endpoint, "oss-cn-beijing"); - readByAwsSdkV2(rawFile.path(), tmpAk, tmpSk, stsToken, endpoint, "oss-cn-beijing"); - } - } else { - System.out.println("test debug no raw files in this data split"); - } - } - } - } else { - System.out.println( - "test debug fileIO is not RESTTokenFileIO, it is: " + fileIO.getClass().getName()); - } - } - } - } - - /** - * https://paimon.apache.org/docs/1.1/concepts/rest/dlf/ - * CREATE CATALOG `paimon-rest-catalog` - * WITH ( - * 'type' = 'paimon', - * 'uri' = '', - * 'metastore' = 'rest', - * 'warehouse' = 'my_instance_name', - * 'token.provider' = 'dlf', - * 'dlf.access-key-id'='', - * 'dlf.access-key-secret'='', - * ); - * - * @return - */ - private org.apache.paimon.catalog.Catalog initPaimonDlfRestCatalog() { - HiveConf hiveConf = new HiveConf(); - Options catalogOptions = new Options(); - catalogOptions.set("metastore", "rest"); - catalogOptions.set("warehouse", "new_dfl_paimon_catalog"); - catalogOptions.set("uri", "http://cn-beijing-vpc.dlf.aliyuncs.com"); - catalogOptions.set("token.provider", "dlf"); - catalogOptions.set("dlf.access-key-id", aliyunAk); - catalogOptions.set("dlf.access-key-secret", aliyunSk); - CatalogContext catalogContext = CatalogContext.create(catalogOptions, hiveConf); - return CatalogFactory.createCatalog(catalogContext); - } - - private void readByAwsSdkV1(String filePath, String accessKeyId, String secretAccessKey, - String sessionToken, String endpoint, String region) throws UserException { - BasicSessionCredentials sessionCredentials = new BasicSessionCredentials( - accessKeyId, - secretAccessKey, - sessionToken - ); - ClientConfiguration clientConfig = new ClientConfiguration(); - clientConfig.setSignerOverride("AWSS3V4SignerType"); - - AmazonS3 s3Client = AmazonS3ClientBuilder.standard() - .withCredentials(new AWSStaticCredentialsProvider(sessionCredentials)) - .withEndpointConfiguration(new EndpointConfiguration(endpoint, region)) - .withClientConfiguration(clientConfig) - .withPathStyleAccessEnabled(false) - .build(); - - S3URI s3URI = S3URI.create(filePath); - System.out.println("test debug s3uri: " + s3URI); - try { - String content = downloadAndReadFileWithSdkV1(s3Client, s3URI.getBucket(), s3URI.getKey()); - System.out.println("Content: " + content); - } catch (Exception e) { - e.printStackTrace(); - Assertions.fail(Util.getRootCauseMessage(e)); - } - } - - private String downloadAndReadFileWithSdkV1(AmazonS3 s3Client, String bucketName, String objectKey) - throws IOException { - S3Object s3Object = s3Client.getObject(new GetObjectRequest(bucketName, objectKey)); - try (InputStream inputStream = s3Object.getObjectContent(); - InputStreamReader reader = new InputStreamReader(inputStream, StandardCharsets.UTF_8)) { - StringBuilder content = new StringBuilder(); - char[] buffer = new char[1024]; - int bytesRead; - while ((bytesRead = reader.read(buffer)) != -1) { - content.append(buffer, 0, bytesRead); - } - return content.toString(); - } - } - - private void readByAwsSdkV2(String path, String tmpAk, String tmpSk, String stsToken, String endpoint, - String region) { - S3Client s3Client = S3Client.builder() - .credentialsProvider(StaticCredentialsProvider.create(AwsSessionCredentials.create(tmpAk, tmpSk, - stsToken))) - .region(Region.of(region)) - .endpointOverride(URI.create("https://" + endpoint)) - .serviceConfiguration(S3Configuration.builder() - .chunkedEncodingEnabled(false) - .pathStyleAccessEnabled(false) - .build()) - .build(); - try { - S3URI s3URI = S3URI.create(path); - System.out.println("test debug s3uri: " + s3URI); - downloadAndReadFileWithSdkV2(s3Client, s3URI.getBucket(), s3URI.getKey()); - } catch (Exception e) { - Assertions.fail(Util.getRootCauseMessage(e)); - } finally { - s3Client.close(); - } - } - - private void downloadAndReadFileWithSdkV2(S3Client s3Client, String bucketName, String objectKey) - throws IOException { - software.amazon.awssdk.services.s3.model.GetObjectRequest request - = software.amazon.awssdk.services.s3.model.GetObjectRequest.builder() - .bucket(bucketName) - .key(objectKey) - .build(); - - try (ResponseInputStream inputStream = s3Client.getObject(request); - BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream))) { - String line; - while ((line = reader.readLine()) != null) { - System.out.println(line); - } - } - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStorePropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStorePropertiesTest.java index abc28709b0184e..acc14381bf8476 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStorePropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonFileSystemMetaStorePropertiesTest.java @@ -18,10 +18,8 @@ package org.apache.doris.datasource.property.metastore; import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; -import org.apache.doris.datasource.property.storage.HdfsProperties; import org.apache.doris.datasource.property.storage.StorageProperties; -import org.apache.paimon.catalog.FileSystemCatalogFactory; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -30,23 +28,6 @@ public class PaimonFileSystemMetaStorePropertiesTest { - @Test - public void testKerberosCatalog() throws Exception { - Map props = new HashMap<>(); - props.put(HdfsProperties.FS_HDFS_SUPPORT, "true"); - props.put("fs.defaultFS", "hdfs://mycluster_test"); - props.put("hadoop.security.authentication", "kerberos"); - props.put("hadoop.kerberos.principal", "myprincipal"); - props.put("hadoop.kerberos.keytab", "mykeytab"); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "filesystem"); - props.put("warehouse", "hdfs://mycluster_test/paimon"); - PaimonFileSystemMetaStoreProperties paimonProps = (PaimonFileSystemMetaStoreProperties) MetastoreProperties.create(props); - //We expect a Kerberos-related exception, but because the messages vary by environment, we’re only doing a simple check. - Assertions.assertThrows(RuntimeException.class, () -> paimonProps.initializeCatalog("paimon", StorageProperties.createAll(props)) - ); - } - @Test public void testInitExecutionAuthenticatorWiresHdfsAuthenticatorWithoutInitializeCatalog() throws Exception { Map props = new HashMap<>(); @@ -56,30 +37,16 @@ public void testInitExecutionAuthenticatorWiresHdfsAuthenticatorWithoutInitializ props.put("warehouse", "file:///tmp"); PaimonFileSystemMetaStoreProperties paimonProps = (PaimonFileSystemMetaStoreProperties) MetastoreProperties.create(props); - // M-8: before wiring, the runtime authenticator is the base no-op — filesystem only set the - // real authenticator inside initializeCatalog(), which is DEAD on the plugin/cutover path, so - // doAs was silently lost over Kerberized HDFS. This assertion pins the bug. + // M-8: before wiring, the runtime authenticator is the base no-op — the filesystem flavor only + // set the real authenticator inside the legacy initializeCatalog() (removed with the legacy + // catalog-build path), so doAs was silently lost over Kerberized HDFS. This assertion pins the bug. Assertions.assertNotEquals(HadoopExecutionAuthenticator.class, paimonProps.getExecutionAuthenticator().getClass()); // The fix: initExecutionAuthenticator builds the HDFS authenticator from the storage props at - // catalog-init time (the path PluginDrivenExternalCatalog now invokes), WITHOUT initializeCatalog. + // catalog-init time (the path PluginDrivenExternalCatalog now invokes). // MUTATION: removing the filesystem initExecutionAuthenticator override leaves the no-op -> red. paimonProps.initExecutionAuthenticator(StorageProperties.createAll(props)); Assertions.assertEquals(HadoopExecutionAuthenticator.class, paimonProps.getExecutionAuthenticator().getClass()); } - - @Test - public void testNonKerberosCatalog() throws Exception { - Map props = new HashMap<>(); - props.put("fs.defaultFS", "file:///tmp"); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "filesystem"); - props.put("warehouse", "file:///tmp"); - PaimonFileSystemMetaStoreProperties paimonProps = (PaimonFileSystemMetaStoreProperties) MetastoreProperties.create(props); - Assertions.assertEquals(FileSystemCatalogFactory.IDENTIFIER, paimonProps.getMetastoreType()); - Assertions.assertEquals("filesystem", paimonProps.getPaimonCatalogType()); - Assertions.assertDoesNotThrow(() -> paimonProps.initializeCatalog("paimon", StorageProperties.createAll(props))); - Assertions.assertEquals(HadoopExecutionAuthenticator.class, paimonProps.getExecutionAuthenticator().getClass()); - } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStorePropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStorePropertiesTest.java index ef382c2c517f3b..e9f6c0f288a4a7 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStorePropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonHMSMetaStorePropertiesTest.java @@ -18,35 +18,14 @@ package org.apache.doris.datasource.property.metastore; import org.apache.doris.datasource.property.storage.HdfsProperties; -import org.apache.doris.datasource.property.storage.StorageProperties; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; -import java.util.Collections; import java.util.HashMap; -import java.util.List; import java.util.Map; public class PaimonHMSMetaStorePropertiesTest { - @Test - public void testKerberosCatalog() throws Exception { - Map props = new HashMap<>(); - props.put(HdfsProperties.FS_HDFS_SUPPORT, "true"); - props.put("fs.defaultFS", "hdfs://mycluster_test"); - props.put("hadoop.security.authentication", "kerberos"); - props.put("hadoop.kerberos.principal", "myprincipal"); - props.put("hadoop.kerberos.keytab", "mykeytab"); - props.put("type", "paimon"); - props.put("hive.metastore.uris", "thrift://localhost:12345"); - props.put("paimon.catalog.type", "hms"); - props.put("warehouse", "hdfs://mycluster/paimon"); - PaimonHMSMetaStoreProperties paimonProps = (PaimonHMSMetaStoreProperties) MetastoreProperties.create(props); - List storagePropertiesList = Collections.singletonList(StorageProperties.createPrimary(props)); - //We expect a Kerberos-related exception, but because the messages vary by environment, we’re only doing a simple check. - Assertions.assertThrows(RuntimeException.class, - () -> paimonProps.initializeCatalog("paimon", storagePropertiesList)); - } @Test public void testNonKerberosCatalog() throws Exception { @@ -59,6 +38,9 @@ public void testNonKerberosCatalog() throws Exception { props.put("warehouse", "file:///tmp"); PaimonHMSMetaStoreProperties paimonProps = (PaimonHMSMetaStoreProperties) MetastoreProperties.create(props); Assertions.assertEquals("hms", paimonProps.getPaimonCatalogType()); + // Parity: only the REST flavor vends credentials; non-REST paimon flavors keep building the + // static storage map (the former provider gated on instanceof PaimonRestMetaStoreProperties). + Assertions.assertFalse(paimonProps.isVendedCredentialsEnabled()); //should mock connection to hms } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java index f7517259fcc5ac..10ef572d851fbd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonJdbcMetaStorePropertiesTest.java @@ -17,42 +17,17 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.doris.catalog.JdbcResource; import org.apache.doris.common.security.authentication.HadoopExecutionAuthenticator; import org.apache.doris.datasource.property.storage.StorageProperties; -import org.apache.paimon.options.CatalogOptions; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; -import java.util.Collections; import java.util.HashMap; import java.util.Map; public class PaimonJdbcMetaStorePropertiesTest { - @Test - public void testBasicJdbcProperties() throws Exception { - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:mysql://localhost:3306/paimon"); - props.put("warehouse", "s3://warehouse/path"); - props.put("paimon.jdbc.user", "paimon"); - props.put("paimon.jdbc.password", "secret"); - - PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - jdbcProps.initNormalizeAndCheckProps(); - jdbcProps.buildCatalogOptions(); - - Assertions.assertEquals("jdbc", jdbcProps.getPaimonCatalogType()); - Assertions.assertEquals("jdbc", jdbcProps.getCatalogOptions().get(CatalogOptions.METASTORE.key())); - Assertions.assertEquals("jdbc:mysql://localhost:3306/paimon", - jdbcProps.getCatalogOptions().get(CatalogOptions.URI.key())); - Assertions.assertEquals("paimon", jdbcProps.getCatalogOptions().get("jdbc.user")); - Assertions.assertEquals("secret", jdbcProps.getCatalogOptions().get("jdbc.password")); - } - @Test public void testInitExecutionAuthenticatorWiresHdfsAuthenticatorWithoutInitializeCatalog() throws Exception { Map props = new HashMap<>(); @@ -63,58 +38,18 @@ public void testInitExecutionAuthenticatorWiresHdfsAuthenticatorWithoutInitializ props.put("fs.defaultFS", "file:///tmp"); PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - // M-8: like filesystem, the jdbc flavor only set the real authenticator inside the now-dead - // initializeCatalog(), so the cutover path kept the base no-op and lost doAs over Kerberized - // HDFS. This assertion pins the bug. + // M-8: like filesystem, the jdbc flavor only set the real authenticator inside the legacy + // initializeCatalog() (removed with the legacy catalog-build path), so the cutover path kept the + // base no-op and lost doAs over Kerberized HDFS. This assertion pins the bug. Assertions.assertNotEquals(HadoopExecutionAuthenticator.class, jdbcProps.getExecutionAuthenticator().getClass()); - // The fix wires the HDFS authenticator from the storage props WITHOUT initializeCatalog. + // The fix wires the HDFS authenticator from the storage props at catalog-init time. // MUTATION: removing the jdbc initExecutionAuthenticator override leaves the no-op -> red. jdbcProps.initExecutionAuthenticator(StorageProperties.createAll(props)); Assertions.assertEquals(HadoopExecutionAuthenticator.class, jdbcProps.getExecutionAuthenticator().getClass()); } - @Test - public void testJdbcPrefixPassthrough() throws Exception { - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:mysql://localhost:3306/paimon"); - props.put("warehouse", "s3://warehouse/path"); - props.put("paimon.jdbc.useSSL", "true"); - props.put("paimon.jdbc.verifyServerCertificate", "true"); - - PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - jdbcProps.initNormalizeAndCheckProps(); - jdbcProps.buildCatalogOptions(); - - Assertions.assertEquals("true", jdbcProps.getCatalogOptions().get("jdbc.useSSL")); - Assertions.assertEquals("true", jdbcProps.getCatalogOptions().get("jdbc.verifyServerCertificate")); - } - - @Test - public void testRawJdbcPrefixPassthrough() throws Exception { - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:mysql://localhost:3306/paimon"); - props.put("warehouse", "s3://warehouse/path"); - props.put("jdbc.user", "raw_user"); - props.put("jdbc.password", "raw_password"); - props.put("jdbc.useSSL", "true"); - props.put("jdbc.verifyServerCertificate", "true"); - - PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - jdbcProps.initNormalizeAndCheckProps(); - jdbcProps.buildCatalogOptions(); - - Assertions.assertEquals("raw_user", jdbcProps.getCatalogOptions().get("jdbc.user")); - Assertions.assertEquals("raw_password", jdbcProps.getCatalogOptions().get("jdbc.password")); - Assertions.assertEquals("true", jdbcProps.getCatalogOptions().get("jdbc.useSSL")); - Assertions.assertEquals("true", jdbcProps.getCatalogOptions().get("jdbc.verifyServerCertificate")); - } - @Test public void testFactoryCreateJdbcType() throws Exception { Map props = new HashMap<>(); @@ -146,70 +81,4 @@ public void testMissingUri() throws Exception { Assertions.assertThrows(IllegalArgumentException.class, () -> MetastoreProperties.create(props)); } - - @Test - public void testDriverClassRequiredWhenDriverUrlIsSet() throws Exception { - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:mysql://localhost:3306/paimon"); - props.put("warehouse", "s3://warehouse/path"); - props.put("paimon.jdbc.driver_url", "https://example.com/mysql-connector-java.jar"); - - PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - jdbcProps.initNormalizeAndCheckProps(); - Assertions.assertThrows(IllegalArgumentException.class, - () -> jdbcProps.initializeCatalog("paimon_catalog", Collections.emptyList())); - } - - @Test - public void testRawDriverClassRequiredWhenDriverUrlIsSet() throws Exception { - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:mysql://localhost:3306/paimon"); - props.put("warehouse", "s3://warehouse/path"); - props.put("jdbc.driver_url", "https://example.com/mysql-connector-java.jar"); - - PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - jdbcProps.initNormalizeAndCheckProps(); - Assertions.assertThrows(IllegalArgumentException.class, - () -> jdbcProps.initializeCatalog("paimon_catalog", Collections.emptyList())); - } - - @Test - public void testGetBackendPaimonOptions() throws Exception { - String driverUrl = "file:///tmp/postgresql-42.5.0.jar"; - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:postgresql://127.0.0.1:5442/postgres"); - props.put("warehouse", "s3://warehouse/path"); - props.put("paimon.jdbc.driver_url", driverUrl); - props.put("paimon.jdbc.driver_class", "org.postgresql.Driver"); - - PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - Map backendOptions = jdbcProps.getBackendPaimonOptions(); - - Assertions.assertEquals( - JdbcResource.getFullDriverUrl(driverUrl), - backendOptions.get("jdbc.driver_url")); - Assertions.assertEquals("org.postgresql.Driver", backendOptions.get("jdbc.driver_class")); - Assertions.assertEquals(2, backendOptions.size()); - } - - @Test - public void testGetBackendPaimonOptionsRequiresDriverClass() throws Exception { - Map props = new HashMap<>(); - props.put("type", "paimon"); - props.put("paimon.catalog.type", "jdbc"); - props.put("uri", "jdbc:postgresql://127.0.0.1:5442/postgres"); - props.put("warehouse", "s3://warehouse/path"); - props.put("paimon.jdbc.driver_url", "file:///tmp/postgresql-42.5.0.jar"); - - PaimonJdbcMetaStoreProperties jdbcProps = (PaimonJdbcMetaStoreProperties) MetastoreProperties.create(props); - IllegalArgumentException exception = Assertions.assertThrows(IllegalArgumentException.class, - jdbcProps::getBackendPaimonOptions); - Assertions.assertTrue(exception.getMessage().contains("driver_class")); - } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java index 0fefe365dfa9de..ab36f870afb8fb 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/PaimonRestMetaStorePropertiesTest.java @@ -17,7 +17,6 @@ package org.apache.doris.datasource.property.metastore; -import org.apache.paimon.options.Options; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -37,60 +36,22 @@ public void testBasicRestProperties() { restProps.initNormalizeAndCheckProps(); Assertions.assertEquals("rest", restProps.getPaimonCatalogType()); - Assertions.assertEquals("rest", restProps.getMetastoreType()); } @Test - public void testUriAliases() { - // Test different URI property names - Map props1 = new HashMap<>(); - props1.put("uri", "http://localhost:8080"); - props1.put("paimon.rest.token.provider", "none"); - props1.put("warehouse", "catalog_name"); - PaimonRestMetaStoreProperties restProps1 = new PaimonRestMetaStoreProperties(props1); - restProps1.initNormalizeAndCheckProps(); - - Map props2 = new HashMap<>(); - props2.put("paimon.rest.uri", "http://localhost:8080"); - props2.put("paimon.rest.token.provider", "none"); - props2.put("warehouse", "catalog_name"); - PaimonRestMetaStoreProperties restProps2 = new PaimonRestMetaStoreProperties(props2); - restProps2.initNormalizeAndCheckProps(); - - // Both should work and set the same URI in catalog options - restProps1.buildCatalogOptions(); - restProps2.buildCatalogOptions(); - - Options options1 = restProps1.getCatalogOptions(); - Options options2 = restProps2.getCatalogOptions(); - - Assertions.assertEquals("http://localhost:8080", options1.get("uri")); - Assertions.assertEquals("http://localhost:8080", options2.get("uri")); - } - - @Test - public void testPaimonRestPropertiesPassthrough() { + public void testIsVendedCredentialsEnabled() { + // A Paimon REST catalog vends credentials per-table and has no static storage map; the SDK-free + // gate (replacing the former PaimonVendedCredentialsProvider) must report true so + // CatalogProperty.initStorageProperties skips building the static StorageProperties map. Map props = new HashMap<>(); props.put("paimon.rest.uri", "http://localhost:8080"); - props.put("paimon.rest.custom.property", "custom-value"); - props.put("paimon.rest.timeout", "30000"); - props.put("paimon.rest.retry.count", "3"); - props.put("paimon.rest.token.provider", "none"); props.put("warehouse", "catalog_name"); + props.put("paimon.rest.token.provider", "none"); PaimonRestMetaStoreProperties restProps = new PaimonRestMetaStoreProperties(props); restProps.initNormalizeAndCheckProps(); - restProps.buildCatalogOptions(); - Options catalogOptions = restProps.getCatalogOptions(); - - // Basic URI should be set - Assertions.assertEquals("http://localhost:8080", catalogOptions.get("uri")); - - // Custom paimon.rest.* properties should be passed through without prefix - Assertions.assertEquals("custom-value", catalogOptions.get("custom.property")); - Assertions.assertEquals("30000", catalogOptions.get("timeout")); - Assertions.assertEquals("3", catalogOptions.get("retry.count")); + Assertions.assertTrue(restProps.isVendedCredentialsEnabled()); } @Test @@ -335,33 +296,6 @@ public void testDlfTokenProviderNegativeValidation() { Assertions.assertTrue(errorMessage3.contains("DLF token provider requires")); } - @Test - public void testPaimonRestPropertiesWithMultipleCustomProperties() { - Map props = new HashMap<>(); - props.put("paimon.rest.uri", "http://localhost:8080"); - props.put("paimon.rest.custom.auth.token", "token123"); - props.put("paimon.rest.custom.header.x-api-key", "api-key-456"); - props.put("paimon.rest.custom.ssl.verify", "false"); - props.put("non.paimon.property", "should-not-be-included"); - props.put("paimon.rest.token.provider", "none"); - props.put("warehouse", "catalog_name"); - - PaimonRestMetaStoreProperties restProps = new PaimonRestMetaStoreProperties(props); - restProps.initNormalizeAndCheckProps(); - - restProps.buildCatalogOptions(); - Options catalogOptions = restProps.getCatalogOptions(); - - // paimon.rest.* properties should be passed through without prefix - Assertions.assertEquals("token123", catalogOptions.get("custom.auth.token")); - Assertions.assertEquals("api-key-456", catalogOptions.get("custom.header.x-api-key")); - Assertions.assertEquals("false", catalogOptions.get("custom.ssl.verify")); - - // Non-paimon.rest properties should not be included - Assertions.assertNull(catalogOptions.get("non.paimon.property")); - Assertions.assertNull(catalogOptions.get("should-not-be-included")); - } - @Test public void testMissingTokenProviderThrowsException() { Map props = new HashMap<>(); From 895e214b2bd54a430c15900479ec1397e2b98334 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 20 Jun 2026 23:50:44 +0800 Subject: [PATCH 4/4] =?UTF-8?q?[P5-T29]=20docs:=20Batch=202=20DONE=20?= =?UTF-8?q?=E2=80=94=20fe-core=20paimon-SDK-free;=20vended=20provider=20GA?= =?UTF-8?q?MMA;=20next=20=3D=20B9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update HANDOFF + design doc to record P5-T29 (B8) completion: - Batch 2 implemented: 6 metastore-props stripped to SDK-free, PaimonVendedCredentialsProvider deleted, gate relocated to MetastoreProperties.isVendedCredentialsEnabled(), 5 paimon deps dropped. - Records the GAMMA deviation (user-resigned 2026-06-20) from the design's original "migrate out + cross-loader seam" sketch — recon + adversarial review proved the provider's SDK methods were already dead, so the migration was unnecessary. - Next session task set to B9 (P5-T30) post-cutover live-e2e regression (docker-gated, user-run) + accepted-deviation sign-off. P5 main work is complete. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_011mTrPcvMZtFjsxWJM5TRnG --- plan-doc/HANDOFF.md | 32 +++++++++++---- .../P5-T29-paimon-legacy-removal-design.md | 41 +++++++++++++++---- 2 files changed, 55 insertions(+), 18 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 443b7eca8d96be..f84babddd2afd2 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -6,18 +6,32 @@ --- -# 🎯 下一个 session 的任务 — **P5-T29（批 B8）Batch 2：strip metastore-props SDK + 迁 `PaimonVendedCredentialsProvider` + 删 paimon maven 依赖（docker-gated）** - -> 📍 **完整修订计划（authoritative，含 firsthand 核实 + 用户签字 D-PB1/D-PB2 + 逐文件 edit 清单）见 -> [`tasks/designs/P5-T29-paimon-legacy-removal-design.md`](./tasks/designs/P5-T29-paimon-legacy-removal-design.md)**（本 session 2026-06-20 新建）。 -> 下文 §「P5-T29 scope ledger」是旧框架，部分被该 design doc **取代**（尤其 §B 常量「搬家」实际改为内联字面量、§D maven 决策见下「关键 scope 修正」）。 -> 样板 = **P4 #64300**（`73832991962`）；对照基线 = `reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md` §B8 ledger。 +# 🎯 下一个 session 的任务 — **P5-T30（B9）post-cutover 回归（live-e2e，用户跑）+ accepted-deviation 签字** + +> **✅ P5-T29（B8）Batch 1 + Batch 2 全部完成 — fe-core 现已完全 paimon-SDK-free（`grep org.apache.paimon +> fe/fe-core/src/{main,test}` = ∅，5 个 paimon maven 依赖已删）。** 仅剩 **B9（P5-T30）post-cutover live-e2e +> 回归**（docker-gated，`enablePaimonTest=true`，用户跑）+ deviations 签字。**P5 阶段主体工作至此收尾。** +> +> **Batch 2 已完成（local-commit 见 git log，未 push）**：详见 +> [`tasks/designs/P5-T29-paimon-legacy-removal-design.md`](./tasks/designs/P5-T29-paimon-legacy-removal-design.md) §5「Batch 2 DONE」+ §6 gates。 +> **关键偏离（用户 2026-06-20 改签 GAMMA）**：recon（`wf_12d67943-eeb`）+ 对抗 review（`wf_ef1fd738-3b9`）证实 +> `PaimonVendedCredentialsProvider` 的 SDK 方法**早已死**（extract* 仅 iceberg 路调用；真 paimon vended 路在连接器 +> `PaimonScanPlanProvider.extractVendedToken`，cutover FIX-1 已搬走），唯一 LIVE 是 SDK-free 的 +> `isVendedCredentialsEnabled` gate。故**不做** design 原设想的「迁出 + cross-loader seam」，改 **GAMMA**： +> **整删 provider**（+ 其 test）+ 删 `VendedCredentialsFactory` `case PAIMON` + gate 下放到新 SDK-free +> `MetastoreProperties.isVendedCredentialsEnabled()`（base=false，`PaimonRestMetaStoreProperties`→true）。 +> iceberg gate 路径 byte-identical（review 6-path 真值表核实），LIVE Kerberos auth 装配未动。 +> 顺手：删 Jdbc `getBackendPaimonOptions`（SDK-free 但 0 live caller）、删 `PaimonDlfRestCatalogTest`（§3 漏列的 SDK importer）、 +> 修 `s3-transfer-manager` 注释（真消费者 = hadoop-aws 非 paimon-s3）。**fe/pom.xml `paimon.version` 保留（R-007）。** +> +> 下文 §「P5-T29 scope ledger」是旧框架（Batch 1 视角），已被 design doc §5/§6 **取代**，仅作历史参考。 +> 样板 = **P4 #64300**（`73832991962`）。 **✅ Batch 1（C1）已完成 + local-commit `7632a074e4b`（未 push）**：删 **33 dead 文件**（`datasource/paimon/*` 除 LIVE `PaimonVendedCredentialsProvider`、`metacache/paimon/*`、`systable/PaimonSysTable`）+ 清 **6 处 live reverse-ref**（`ExternalCatalog`/`ExternalMetaCacheMgr`/`ExternalMetaCacheRouteResolver`/`Env`[保 LIVE D-046 PLUGIN 分支]/`UserAuthentication`/`ShowPartitionsCommand`[保 `hasPartitionStatsCapability`+live `PAIMON_EXTERNAL_TABLE` 枚举]）+ **3 javadoc scrub** + **5 dead test 删** + 2 generic fixture test 修（`StatementContextTest` mock→`PluginDrivenMvccExternalTable`、`ExternalMetaCacheRouteResolverTest`）+ metastore-props `getPaimonCatalogType` 内联字面量（脱钩已删的 `PaimonExternalCatalog`，免「常量搬家」前置）。**fe-core test-compile BUILD SUCCESS + checkstyle 0 + 49 改动测试绿**；`datasource/paimon/` 现仅剩 `PaimonVendedCredentialsProvider`。`reverse-ref + 删文件须同一 commit`（P4 precedent：`PaimonUtils:57`→已删的 `ExternalMetaCacheMgr.paimon()`）。 **🔱 关键 scope 修正（本 session firsthand，推翻旧 §D 框架）**：方案 A/B 都**只碰 7 个 metastore-props**，都**不能单独删** 5 个 paimon maven 依赖——31 个 fe-core 文件 import `org.apache.paimon.*`，其中 ~23 是 Batch 1 已删的 dead 子树；剩 **6 metastore-props**（SDK 100% 在 dead catalog-building 方法→可 strip）+ **`PaimonVendedCredentialsProvider`**（genuinely LIVE，runtime 用 paimon REST SDK，挂在 generic `VendedCredentialsFactory.getProviderType` 的 `case PAIMON`，经 `CatalogProperty:182`）。**用户签 = Plan B（fe-core fully paimon-free）+ D-PB1 strip-in-place（不物理搬 7 类，与 iceberg/hive parity）+ D-PB2 phased**。strip 因「reshape 6 live 类 + trim 7 test 文件 + 单独不删 dep」从 Batch 1 **移到 Batch 2**（用户 2026-06-20 签）。 -**Batch 2（下一步，docker-gated，design doc §4 有逐文件清单）**： +**Batch 2（✅ 已完成；下为「原计划」历史记录，第 3 项 vended 实际改用 GAMMA — 见上 banner + design doc §5）**： 1. **B1-strip 6 metastore-props**（`AbstractPaimonProperties`+5 flavor）：删 `initializeCatalog`/`buildCatalogOptions`/`appendCatalogOptions`/`appendCustomCatalogOptions`/`getCatalogOptionsMap`/`getCatalogOptions`(catalogOptions 字段)/`getMetastoreType`/`appendUserHadoopConfig`/`normalizeS3Config`/Jdbc `getBackendPaimonOptions`+`registerJdbcDriver`+`DriverShim` + 全 `org.apache.paimon.*` import。**保 LIVE**：`warehouse` @ConnectorProperty、`executionAuthenticator`+`getExecutionAuthenticator`、`initExecutionAuthenticator`/`initHdfsExecutionAuthenticator`（`PluginDrivenExternalCatalog:137-138` 读，Kerberos doAs）、`initNormalizeAndCheckProps`/validation、`getPaimonCatalogType`（已内联）。**这些 strip 方法 0 live main caller**（只 test）。 2. **trim 7 test**：`PaimonCatalogTest`（@Disabled 手测→直接删）、`AbstractPaimonPropertiesTest`（test-local subclass override 被删的抽象方法→修）、`Paimon{HMS,FileSystem,Jdbc,Rest,AliyunDLF}MetaStorePropertiesTest`（去 catalog-building 断言，保 validation/binding/type/auth）。 3. **迁 `PaimonVendedCredentialsProvider` 出 fe-core** + 改 generic `VendedCredentialsFactory`（switch on `MetastoreProperties.Type.PAIMON`，与 iceberg 共享；需新 fe-core seam 让 plugin-loader 侧 provider 喂回，cross-loader）。**这是真正的 cross-cutting 件**，碰 generic/shared fe-core（iceberg 也在同 factory）。 @@ -77,8 +91,8 @@ # 📦 仓库 / 进度状态 - **当前分支 = `branch-catalog-spi`**（开发主分支）。HEAD 近端：`38e7140ce56`（#64446 P5 迁移+翻闸）← `e9c5b3e70ce`（修编译）。 P0–P5(迁移+翻闸) + P3 hybrid + P4 全部已合入本分支。 -- **P5 状态**：B0–B7 全完成并合入 #64446；**仅剩 B8（P5-T29 删 legacy）+ B9（P5-T30 回归）**。 -- **legacy 仍在 fe-core**（待 P5-T29 删）：`datasource/paimon/`(30) + `metacache/paimon/`(3) + `systable/PaimonSysTable`(1) + 8 处反向引用文件 + paimon maven 依赖(5)。STILL-CONSUMED `property/metastore/Paimon*`(7) **保留**。 +- **P5 状态**：B0–B7 合入 #64446；**B8（P5-T29 删 legacy）= Batch 1 + Batch 2 全完成**（local-commit，未 push）；**仅剩 B9（P5-T30 live-e2e 回归，用户跑）**。 +- **fe-core 现已完全 paimon-SDK-free**：`datasource/paimon/` 整目录已删；6 个 `property/metastore/Paimon*` 已 strip 成 SDK-free 描述符（保 LIVE auth/validation/@ConnectorProperty/type）；5 个 paimon maven 依赖已删。`grep org.apache.paimon fe/fe-core/src/{main,test}` = ∅。**fe/pom.xml `paimon.version` 保留**（R-007：fe-connector-paimon + BE 仍用）。 - ⚠️ `regression-test/conf/regression-conf.groovy` 若仍 modified 且含**明文 Aliyun key** → commit 前继续 path-whitelist，**严禁 `git add -A`**；`regression-conf.groovy.bak` 同理排除。 - 未跟踪 scratch：`.audit-scratch/` `conf.cmy/` `META-INF/` `*.bak` 等——commit 前清，勿 add。 - `reviews/P6-paimon-fullpath-cleanroom-2026-06-18.md`（B8 readiness ledger 来源）若仍未跟踪，下次方便时 vet + commit 或保留本地。 diff --git a/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md b/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md index b3bbd5418c7fca..a6ebeae913e2b9 100644 --- a/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md +++ b/plan-doc/tasks/designs/P5-T29-paimon-legacy-removal-design.md @@ -175,9 +175,31 @@ deleting the dead files **cannot** be split. *(First attempt split this into prep-then-delete; the `PaimonUtils → paimon()` coupling broke the intermediate compile — merged per P4 precedent.)* -**Batch 2 (later, docker-gated):** §4 strip SDK methods + imports from the 6 metastore-props + trim the 7 -catalog-building test files; migrate `PaimonVendedCredentialsProvider`; rework `VendedCredentialsFactory`; -**drop all 5 paimon deps**. *Target:* `grep org.apache.paimon fe-core/src/main` = ∅; `dependency:tree | grep paimon` = ∅. +**Batch 2 = 1 code commit — ✅ DONE:** §4 strip SDK methods + imports from the 6 metastore-props + trim +the 8 catalog-building test files; **drop all 5 paimon deps**. *Target met:* `grep org.apache.paimon +fe-core/src/{main,test}` = ∅; `dependency:tree -Dincludes=org.apache.paimon` on fe-core = ∅. + +> **🔱 Vended-provider deviation (firsthand recon `wf_12d67943-eeb` + user re-signed 2026-06-20 → GAMMA):** +> the design above (§0.2/§4) assumed `PaimonVendedCredentialsProvider` was LIVE end-to-end and had to be +> *migrated out* of fe-core with a cross-loader `VendedCredentialsFactory` seam. **Recon refuted that +> premise** (adversarially confirmed): the provider's paimon-SDK methods (`extractRawVendedCredentials`/ +> `getTableName`) are **dead** — reachable only via `getStoragePropertiesMapWithVendedCredentials`, whose +> only callers are iceberg; the real paimon runtime vended path moved to the connector +> (`PaimonScanPlanProvider.extractVendedToken`) at cutover FIX-1. The provider's only LIVE duty was the +> SDK-free `isVendedCredentialsEnabled` gate (`instanceof PaimonRestMetaStoreProperties`) read by +> `CatalogProperty.initStorageProperties`. So the cross-loader migration was unnecessary. The user chose +> **GAMMA**: **delete `PaimonVendedCredentialsProvider` entirely** (+ its test), remove the +> `VendedCredentialsFactory` `case PAIMON` (+ import), and **relocate the gate** to a new SDK-free +> `MetastoreProperties.isVendedCredentialsEnabled()` (base = `false`, `PaimonRestMetaStoreProperties` → +> `true`). `CatalogProperty`'s gate now routes the provider path for iceberg (byte-identical) and the +> metastore-props path for everything else. A 3-agent adversarial review (`wf_ef1fd738-3b9`) verified the +> `checkStorageProperties` truth table is byte-identical to HEAD on all 6 paths and no LIVE Kerberos +> auth-wiring was severed. Also: `getBackendPaimonOptions` (Jdbc) was dropped (SDK-free but 0 live callers — +> connector has its own); `PaimonDlfRestCatalogTest` (paimon-SDK importer not in §3's named list) was deleted. +> *Verified:* fe-core `test-compile` BUILD SUCCESS + checkstyle 0; 32 affected tests green; import-gate OK; +> `s3-transfer-manager` retained (real consumer = hadoop-aws, comment corrected). **fe/pom.xml +> dependencyManagement `paimon.version` kept (R-007: fe-connector-paimon + BE still consume it).** +> live-e2e `enablePaimonTest=true` is **docker-gated → user-run (B9/P5-T30)**, NOT run here. **Hard pre-commit (HANDOFF):** scrub `regression-test/conf/regression-conf.groovy` (plaintext key); clean scratch (`.audit-scratch/`/`conf.cmy/`/`META-INF/`/`*.bak`). **Path-whitelist `git add`; NEVER `git add -A`.** @@ -187,9 +209,10 @@ Each commit: `[P5-T29] ` + root cause + fix + tests + `Co-Authored-By: Cla ## 6. Verification gates (mirror P4 #64300) -- [ ] fe-core `compile` BUILD SUCCESS + `testCompile` + checkstyle 0 (`validate` phase) per commit. -- [ ] `tools/check-connector-imports.sh` exit 0. -- [ ] paimon connector module UT green (`-pl :fe-connector-paimon -am package -Dassembly.skipAssembly=true`). -- [ ] After C3: `grep -rl "import org.apache.paimon\." fe/fe-core/src/main` = ONLY `PaimonVendedCredentialsProvider`. -- [ ] Batch 2 (separate): `dependency:tree | grep paimon` = removed set absent; live-e2e `enablePaimonTest=true`. -- [ ] regression-gated live-e2e (B9/P5-T30, user-run) after Batch 2 — 5-flavor read + sys-table + MTMV + DDL no regression. +- [x] **Batch 1+2:** fe-core `test-compile` BUILD SUCCESS + checkstyle 0 (`validate` phase). +- [x] **Batch 1+2:** `tools/check-connector-imports.sh` exit 0. +- [x] **Batch 2:** `grep -rl "import org.apache.paimon\." fe/fe-core/src/{main,test}` = ∅ (was: only `PaimonVendedCredentialsProvider`). +- [x] **Batch 2:** `dependency:tree -Dincludes=org.apache.paimon` on fe-core = ∅; `s3-transfer-manager` retained. +- [x] **Batch 2:** 32 affected tests green (5 trimmed flavor tests + `VendedCredentialsFactoryTest`). +- [ ] paimon connector module UT green (`-pl :fe-connector-paimon -am package -Dassembly.skipAssembly=true`) — connector untouched; spot-check optional. +- [ ] regression-gated live-e2e (B9/P5-T30, **docker-gated, user-run**) after Batch 2 — 5-flavor read + sys-table + MTMV + DDL no regression.