[feature](paimon) Support Paimon table option passthrough by zhangstar333 · Pull Request #65955 · apache/doris

zhangstar333 · 2026-07-23T08:42:52Z

What problem does this PR solve?

Paimon provides many configurable table options, now introduces the paimon.table-option.* namespace.
Doris removes the prefix, validates the option through Paimon's
SupportedTableOptions, and applies it to the Paimon Table before serialization.
Explicit options, such as paimon.table-option.read.batch-size, are preserved by the JNI scanner.
Invalid or unsupported options fail fast during Catalog initialization.
eg:

  CREATE CATALOG paimon_catalog PROPERTIES (
      "type" = "paimon",
      "paimon.catalog.type" = "filesystem",
      "warehouse" = "hdfs://127.0.0.1:8020/user/paimon/warehouse",

      "paimon.table-option.read.batch-size" = "4096",

      "paimon.jni.enable_jni_io_manager" = "true",
      "paimon.jni.io_manager.tmp_dir" = "/data/doris/paimon_jni_tmp"
  );

let format v2 could work with jni format and lance format.
some jni params all named with paimon.jni.xxxx

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-07-23T08:42:56Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhangstar333 · 2026-07-23T08:51:49Z

/review

github-actions · 2026-07-23T09:35:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-07-23T09:35:34Z

PR approved by anyone and no changes requested.

github-actions · 2026-07-23T09:36:33Z

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 28th, 2026 5:08 PM.
Workflow run: https://github.com/apache/doris/actions/runs/29992804595

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Gabriel39 · 2026-07-23T14:05:17Z

run buildall

Gabriel39 · 2026-07-23T14:05:23Z

/review

github-actions

I found eight blocking correctness and compatibility defects, grouped into six inline comments.

Checkpoint conclusions:

Goal and scope: the option-passthrough path is patch-scoped, but reflecting every CoreOptions key admits dynamic-copy-illegal and context-sensitive options.
Correctness, errors, and lifecycle: unsupported scans lose their V1 path; admitted JNI scans can evaluate predicates twice; zero batch size can spin; failed ALTER validation does not roll back; branch and time-travel contexts can be corrupted or collide.
Compatibility and protocol: the IO-manager key rename breaks persisted catalogs and both directions of rolling FE/BE compatibility.
Concurrency and resources: no separate concurrency defect was substantiated; the material resource risk is the zero-progress batch loop.
Tests: changed tests cover parsing and isolated option construction, but not selector capability, non-deterministic JNI predicates, mixed-version keys, dynamic-copy rejection, zero-progress reads, failed-ALTER state, divergent branch schemas, or cross-kind time-travel selectors.
Performance and observability: deterministic JNI predicates are also redundantly evaluated; no separate observability defect was substantiated.
Transactions and security: no additional applicable finding.
User focus: no extra focus was specified.

Convergence: the configured three-round maximum was reached. The final BE and risk passes found two additional issues; both were independently validated and included here, but the review did not reach all-lanes NO_NEW_VALUABLE_FINDINGS convergence and is therefore incomplete under the review contract.

Validation was static only, as required by the runner contract; no builds or tests were run.

github-actions · 2026-07-23T15:04:19Z

-           scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP &&
-           scan_params.format_type != TFileFormatType::FORMAT_LANCE &&
-           scan_params.format_type != TFileFormatType::FORMAT_JNI && !is_transactional_hive;
+           scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP && !is_transactional_hive;


[P1] Keep JNI and Lance on V1 until V2 is semantically ready

This scan-level switch activates V2 for every eligible JNI/Lance scan, but both capability and residual semantics are incomplete. V2 has no Lance case, rejects LakeSoul, Paimon PAIMON_CPP, and legacy Paimon without reader_type, then returns NotSupported with no V1 fallback; unchanged selector tests at file_scanner_v2_test.cpp:420-461 still require those shapes to stay on V1. Even admitted JNI readers clone every scanner conjunct and filter it in finalize_jni_block(), after which Scanner::_filter_output_block() evaluates the original again, so a non-deterministic predicate such as rand() is applied twice (unlike V1). Please keep JNI/Lance on V1 until selection is capability-gated and the JNI residual path executes each predicate once, with tests for unsupported shapes and non-deterministic predicates.

github-actions · 2026-07-23T15:04:20Z

-    private static final String DORIS_ENABLE_JNI_IO_MANAGER = "doris.enable_jni_io_manager";
-    private static final String DORIS_JNI_IO_MANAGER_TMP_DIR = "doris.jni_io_manager.tmp_dir";
-    private static final String DORIS_JNI_IO_MANAGER_IMPL_CLASS = "doris.jni_io_manager.impl_class";
+    private static final String DORIS_ENABLE_JNI_IO_MANAGER = "jni.enable_jni_io_manager";


[P1] Preserve the legacy IO-manager keys across upgrades

These names are persisted catalog properties and cross-version protocol inputs, not internal-only constants. Existing catalogs store paimon.doris.*, but the new FE now searches only paimon.jni.*, so the option disappears and the Java scanner silently defaults the IO manager to disabled. Old-FE/new-BE and new-FE/old-BE pairs have the same inverse spelling mismatch. Please accept both namespaces for a compatibility window, define new-key precedence, normalize at FE/BE/Java boundaries, and retain old-catalog plus mixed-version tests.

github-actions · 2026-07-23T15:04:20Z

+        private static SupportedTableOptions build() {
+            Map<String, ConfigOption<?>> exactOptions = new HashMap<>();
+            Map<String, ConfigOption<?>> prefixOptions = new HashMap<>();
+            for (ConfigOption<?> option : CoreOptions.getOptions()) {


[P1] Only admit options legal for Paimon dynamic copies

CoreOptions.getOptions() includes bucket and other immutable table options, so for example paimon.table-option.bucket=4 passes this validator. On first table load, PaimonExternalCatalog calls table.copy(tableOptions); bundled Paimon 1.3.1 rejects changed bucket in dynamic options and rejects its immutable option set. Catalog creation can therefore succeed while a table with a different stored value fails metadata/query access. Please build the allow-list from options Paimon permits in Table.copy (canonicalizing fallbacks), or reject immutable/bucket options during CREATE/ALTER, and add an end-to-end negative test.

github-actions · 2026-07-23T15:04:20Z

+        }
+
+        try {
+            new Options(Collections.singletonMap(key, value)).get(option);


[P1] Reject a zero Paimon read batch before scanning

Options.get() checks only that this value is an integer, so paimon.table-option.read.batch-size=0 is accepted and the new putIfAbsent path preserves it. In bundled Paimon 1.3.1, the Parquet reader then computes a zero-row batch, returns success without advancing rowsReturned, and repeats that result. Doris consumes the empty iterator and immediately calls readBatch() again, so a nonempty Parquet scan spins indefinitely. Please require read.batch-size > 0 during CREATE/ALTER validation and cover zero/non-positive values with a reader-progress regression.

github-actions · 2026-07-23T15:04:20Z

+            return executionAuthenticator.execute(() -> {
+                Table table = catalog.getTable(identifier);
+                Map<String, String> tableOptions = paimonProperties.getTableOptionsMap();
+                return tableOptions.isEmpty() ? table : table.copy(tableOptions);


[P1] Exclude context selectors from catalog table options

Reflecting every CoreOptions admits options that cannot be safely persisted and applied after load. branch=dev reaches this copy after main's TableSchema is loaded; Paimon retains that schema while branch managers/files switch to dev, so independent branch schemas can mismatch. Likewise a catalog scan.tag-name remains when explicit FOR VERSION AS OF later adds scan.snapshot-id; copy() merges both and Paimon rejects the mutually exclusive selectors (incremental overlays leave the same stale keys). Please restrict this namespace to context-free read tuning options; keep branch selection on the Identifier path and time-travel/incremental selectors query-scoped, with cross-branch-schema and cross-kind selector tests.

github-actions · 2026-07-23T15:04:20Z

+        try {
+            new Options(Collections.singletonMap(key, value)).get(option);
+        } catch (IllegalArgumentException e) {
+            throw new IllegalArgumentException("Invalid value for Paimon table option '" + key + "': "


[P1] Roll back catalog state for validation exceptions

This new IllegalArgumentException escapes the existing ALTER rollback contract. CatalogMgr.replayAlterCatalogProps() first calls tryModifyCatalogProps() (which mutates the map and resets its caches), but restores oldProperties only from catch (DdlException). A rejected unknown or malformed table option is therefore not edit-logged yet remains in the running FE; an unknown key cannot be removed by another SET and can keep property reconstruction failing until restart/recreation. Please translate these failures to DdlException or roll back on every validation exception, and add an ALTER-level test that verifies the prior catalog state remains usable after rejection.

hello-stephen · 2026-07-23T15:25:15Z

FE UT Coverage Report

Increment line coverage 85.25% (52/61) 🎉
Increment coverage report
Complete coverage report

support paimon table options

9120bc9

zhangstar333 requested review from Gabriel39 and yiguolei as code owners July 23, 2026 08:42

Gabriel39 added the dev/4.2.x label Jul 23, 2026

Gabriel39 approved these changes Jul 23, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2026

github-actions Bot added the reviewed label Jul 23, 2026

github-actions Bot requested changes Jul 23, 2026

View reviewed changes

Uh oh!

Conversation

zhangstar333 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 23, 2026

Uh oh!

zhangstar333 commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

Gabriel39 commented Jul 23, 2026

Uh oh!

Gabriel39 commented Jul 23, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Jul 23, 2026

FE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhangstar333 commented Jul 23, 2026 •

edited

Loading