[Feature](partition) Support new partition recycle mechanism - #57013
Merged
Conversation
zclllyybb
requested review from
CalvinKirs,
dataroaring,
gavinchou,
morningman and
w41ter
as code owners
October 15, 2025 13:52
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 190623 ms |
ClickBench: Total hot run time: 30.64 s |
zclllyybb
force-pushed
the
new_partition
branch
from
October 15, 2025 15:11
3261f06 to
4a6ce98
Compare
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 189945 ms |
ClickBench: Total hot run time: 30.08 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
dataroaring
reviewed
Oct 16, 2025
Contributor
|
what about using retention_count, e.g. "partition.retention_count" = "3"? |
dataroaring
reviewed
Oct 16, 2025
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 190908 ms |
ClickBench: Total hot run time: 30.38 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
8 tasks
dataroaring
reviewed
Oct 21, 2025
deardeng
reviewed
Nov 11, 2025
yujun777
reviewed
Nov 12, 2025
github-actions Bot
pushed a commit
that referenced
this pull request
Nov 12, 2025
now support:
```sql
create table auto_recycle(
k0 datetime(6) not null
)
auto partition by range (date_trunc(k0, 'day')) ()
DISTRIBUTED BY HASH(`k0`) BUCKETS 1
properties(
"replication_num" = "1",
"partition.retention_count" = "3"
);
```
which means only keep the latest 3 partition in history partitions.
ccr will be tested in selectdb/ccr-syncer#646
### Release note
Resolve the problem of different name format when need to recycle auto partition
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Nov 13, 2025
…57013) now support: ```sql create table auto_recycle( k0 datetime(6) not null ) auto partition by range (date_trunc(k0, 'day')) () DISTRIBUTED BY HASH(`k0`) BUCKETS 1 properties( "replication_num" = "1", "partition.retention_count" = "3" ); ``` which means only keep the latest 3 partition in history partitions. ccr will be tested in selectdb/ccr-syncer#646 ### Release note Resolve the problem of different name format when need to recycle auto partition
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Nov 18, 2025
…57013) now support: ```sql create table auto_recycle( k0 datetime(6) not null ) auto partition by range (date_trunc(k0, 'day')) () DISTRIBUTED BY HASH(`k0`) BUCKETS 1 properties( "replication_num" = "1", "partition.retention_count" = "3" ); ``` which means only keep the latest 3 partition in history partitions. ccr will be tested in selectdb/ccr-syncer#646 ### Release note Resolve the problem of different name format when need to recycle auto partition
16 tasks
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 26, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 26, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud regression test test_nereids_showpartitionid passed against the persistent Cloud runtime; Docker suites were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 26, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud regression test test_nereids_showpartitionid passed against the persistent Cloud runtime; Docker suites were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 30, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. ### Release note Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. ### Check List (For Author) - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
now support:
which means only keep the latest 3 partition in history partitions.
ccr will be tested in selectdb/ccr-syncer#646
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)