[feat](fe) Support rolling upgrades of partition inverted index formats - #66017
Closed
hoshinojyunn wants to merge 1 commit into
Closed
[feat](fe) Support rolling upgrades of partition inverted index formats#66017hoshinojyunn wants to merge 1 commit into
hoshinojyunn wants to merge 1 commit into
Conversation
hoshinojyunn
requested review from
924060929,
CalvinKirs,
dataroaring,
deardeng,
englefly,
gavinchou,
luwei16,
morningman,
morrySnow,
mymeiyi and
starocean999
as code owners
July 24, 2026 12:02
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hoshinojyunn
force-pushed
the
partition_upgrade
branch
2 times, most recently
from
July 25, 2026 06:02
b28406e to
21fda6a
Compare
hoshinojyunn
added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#66017 Problem Summary: Cloud tablet schemas are reused by index ID and schema version. When truncating an older partition after its inverted-index format changes, the current schema version could resolve to a newer V3 schema and recreate a V2 partition with V3 tablet metadata. Persist each partition's schema versions and reuse them during tablet recreation. Apply inverted-index storage format only to the base index; rollup indexes retain their schema versions but do not receive an inverted-index format. ### Release note Cloud partition truncate, recycle-bin recovery, and table recovery preserve each base partition's inverted-index storage format. ### Check List (For Author) - Test: Regression test / Manual test - Regression test: test_partition_inverted_index_storage_format_recycle_truncate - Manual test: Cloud recycle-partition, recover-table, and truncate lifecycle with a rollup index - Build: ./build.sh --fe - Behavior changed: Yes (cloud tablet recreation preserves partition-specific schemas) - Does this need documentation: No
hoshinojyunn
force-pushed
the
partition_upgrade
branch
3 times, most recently
from
July 26, 2026 04:21
9b34ab8 to
bbfe84b
Compare
Contributor
Author
|
run buildall |
hoshinojyunn
force-pushed
the
partition_upgrade
branch
from
July 26, 2026 06:08
bbfe84b to
4590c76
Compare
Contributor
TPC-H: Total hot run time: 29829 ms |
Contributor
TPC-DS: Total hot run time: 177980 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
ClickBench: Total hot run time: 25.08 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29161 ms |
Contributor
TPC-DS: Total hot run time: 177755 ms |
Contributor
ClickBench: Total hot run time: 25 s |
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud regression test test_nereids_showpartitionid passed against the persistent Cloud runtime; Docker suites were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn
force-pushed
the
partition_upgrade
branch
from
July 26, 2026 15:19
4590c76 to
2c493ad
Compare
hoshinojyunn
requested review from
Gabriel39,
airborne12,
csun5285,
eldenmoon and
liaoxin01
as code owners
July 26, 2026 15:19
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29906 ms |
Contributor
TPC-DS: Total hot run time: 177033 ms |
Contributor
ClickBench: Total hot run time: 24.94 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
airborne12
reviewed
Jul 27, 2026
| Map<Long, Integer> indexIdToSchemaVersion = new HashMap<>(); | ||
| if (needSchemaVersionUpdate) { | ||
| for (MaterializedIndexMeta indexMeta : olapTable.getIndexIdToMeta().values()) { | ||
| indexIdToSchemaVersion.put(indexMeta.getIndexId(), indexMeta.getSchemaVersion() + 1); |
Member
There was a problem hiding this comment.
[blocking] This bumps the table-level schema versions without first materializing each existing partition's pre-bump version. A legacy partition loaded from metadata without indexIdToSchemaVersion falls back through Partition.getSchemaVersion(indexId, defaultSchemaVersion) to the mutable table-level version. After this update changes version N to N+1, that old V2 partition will also resolve to N+1, while newly created V3 partitions record N+1 explicitly. Because Cloud Meta Service identifies a schema by (index_id, schema_version), rebuilding/truncating/repairing the old partition can then reuse the V3 schema under the same key.
Please snapshot version N into every existing partition that lacks an entry before changing the table-level versions, and persist/replay that partition backfill atomically with this property update. A compatibility test should deserialize a legacy partition with no isv, perform the V2→V3 rollout, and verify both the live catalog and edit-log replay keep the old partition on N while new partitions use N+1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #57013
Problem Summary:
An inverted-index storage-format rollout must not rewrite historical tablets. A table can therefore contain V2 partitions created before the rollout and V3 partitions created afterwards. Before this change, the format selected for a future partition was not persisted as partition metadata, so lifecycle paths could not consistently preserve the format that the partition requires.
This PR introduces
partition.inverted_index_storage_formatfor future logical partitions. It resolves and persists V2 or V3 inPartitionInfo, and passes that value to tablet and replica creation. Changing the property does not modify existing partitions.Implementation changes:
SHOW PARTITIONS,SHOW PARTITION <partition_id>,partitions(...), andSHOW CREATE TABLE.Example SQL:
Observability result:
SHOW PARTITIONS/partitions(...))GET <MetaUrl>)p_oldInvertedIndexStorageFormat = V2schema.inverted_index_storage_format = "V2"p_newInvertedIndexStorageFormat = V3schema.inverted_index_storage_format = "V3"SHOW TABLETS FROM db.t PARTITION(p_new)provides each tablet MetaUrl. Requesting the URL verifies the format stored in the actual BE tablet metadata, not only the FE partition metadata.Cloud mode support and validation
Cloud mode handles this property change in
CloudSchemaChangeHandler; the genericSchemaChangeHandlerremains unchanged for this purpose. Whenpartition.inverted_index_storage_formatchanges, everyMaterializedIndexMetaadvances to its next schema version, and the exact per-index target versions are persisted inModifyTablePropertyOperationLog.Cloud schema records are keyed by
(index_id, schema_version). This retains the old V2 schema record for existing tablet metadata while newly created partitions use the new V3 (or rolled-back V2) version, so the formats coexist without rewriting historical tablets.Cloud regression coverage adds
cloud_p0/test_partition_cloud_inverted_index_storage_format_rollout. It exercises RANGE, LIST, AUTO LIST, and DYNAMIC RANGE partitions across V2 -> V3 -> V2 rollouts; mixed-format inverted-index queries;INSERT OVERWRITE;SHOW PARTITION; and the schema format from every tabletMetaUrl.Release note
Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions.
Check List (For Author)
Partition-type regression coverage:
p_oldwith table format V2; setpartition.inverted_index_storage_formatto V3;ADD PARTITION p_new.p_oldremains V2 andp_newis V3 in bothSHOW PARTITIONand every tablet MetaUrl. AnINSERT OVERWRITEofp_oldalso remains V2 after the table property is V3.inverted_index_p0/test_partition_inverted_index_storage_format_rollout(passed)p_oldwith table format V2; setpartition.inverted_index_storage_formatto V3;ADD PARTITION p_new.p_oldremains V2 andp_newis V3 in bothSHOW PARTITIONand every tablet MetaUrl.inverted_index_p0/test_partition_inverted_index_storage_format_rollout(passed)CREATE TABLE ... AUTO PARTITION BY LIST(k) ()with V2; insertold; set the partition property to V3; insertnew.oldpartition remains V2; the newly auto-created partition is V3 in FE metadata and BE tablet headers.inverted_index_p0/test_partition_inverted_index_storage_format_rollout(passed)dynamic_partition.end = 1; set the partition property to V3; advancedynamic_partition.endto 3.inverted_index_p0/test_partition_inverted_index_storage_format_rollout(passed)All partition-type rows use the same assertion helper: it looks up the partition with
SHOW PARTITIONS, checksSHOW PARTITION <partition_id>, then requests the MetaUrl from every row ofSHOW TABLETS FROM <table> PARTITION(<partition>). This verifies that the format held by FE and the format stored by BE remain consistent.Lifecycle and recovery coverage:
p_v2, set the property to V3, addp_v3, then runADMIN SET REPLICA STATUS ... status = "drop"for a V2 tablet and wait for scheduling.p_v2is V2;p_v3remains V3. Both values are verified through FE metadata and BE tablet headers.inverted_index_p2/test_partition_inverted_index_storage_format_replica_recovery(passed)p1–p7, set the property to V3, addp8; runBACKUP SNAPSHOT, drop the table, then runRESTORE SNAPSHOT.p1–p7remain V2 andp8remains V3; all tablet headers are checked and inverted-index queries succeed.backup_restore/test_backup_restore_inverted_idx(passed with local MinIO)PartitionPersistInfowith V3 and deserialize it; deserialize a legacy record with no format. The replay path consumes this persisted field and restores it toPartitionInfo.PartitionPersistInfoTest(passed, 1 test)resetPartitionIdForRestore.RangePartitionInfoTest(passed, 17 tests)CreateReplicaTaskand materialize its agent request.TCreateTabletReqcarries V3 to BE.AgentTaskTest(passed, 4 tests)inverted_index_p0/test_partition_inverted_index_storage_format_recycle_truncate(passed);cloud_p0/test_partition_cloud_inverted_index_storage_format_recycle_truncate(passed).Test summary:
Passed
./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest.Passed
./run-fe-ut.sh --run org.apache.doris.catalog.RangePartitionInfoTest,org.apache.doris.persist.PartitionPersistInfoTest,org.apache.doris.task.AgentTaskTest(22 tests, no errors or failures).Passed
DISABLE_BUILD_UI=ON ./build.sh --fe, including FE Checkstyle and packaging.Passed
./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout.Passed
./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery.Passed
./run-regression-test.sh --run -d backup_restore -s test_backup_restore_inverted_idxwith local MinIO.The local SHOW regression case includes the additional coverage described above but was not rerun in this validation.
Passed
./run-regression-test.sh --run -d cloud_p0 -s test_partition_cloud_inverted_index_storage_format_rollout.Passed
./run-regression-test.sh --run -d cloud_p0 -s test_partition_cloud_inverted_index_format.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)
Additional Cloud lifecycle support
Cloud Meta Service stores schemas by
(index_id, schema_version). The partition format mapping preserves the logical V2/V3 selection, but Cloud also needs the schema version that each materialized index used when the partition was created. Without it, recreating an old V2 partition after a V3 rollout can resolve the current V3 schema duringTRUNCATE.This change persists the per-index creation-time schema versions for each partition. Cloud recycle-bin recovery and truncate reuse the corresponding source-partition versions; normal new partitions continue to use the current versions.
CloudRollupJobV2also records a rollup index version when the rollup becomes visible. The inverted-index storage format is supplied only to the base index: rollup indexes retain their schema version for Cloud lookup, but do not receive an inverted-index format because they have no secondary inverted index.Additional lifecycle coverage:
inverted_index_p0/test_partition_inverted_index_storage_format_recycle_truncatecoversDROP PARTITION/RECOVER PARTITION,DROP TABLE/RECOVER TABLE, andTRUNCATE TABLEfor mixed V2/V3 partitions. It verifiesSHOW PARTITION, every base tablet header, replacement tablet IDs, and inverted-index queries.cloud_p0/test_partition_cloud_inverted_index_storage_format_recycle_truncatemirrors those lifecycle paths with a rollup index. It verifies V2/V3 only on base tablets and confirms the rollup schema has noINVERTEDindex.Additional validation:
./build.sh --fe, including FE Checkstyle../run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_recycle_truncate.Additional Cloud schema-version handling
This round adds the following Cloud-specific behavior.
CloudSchemaChangeHandlercontinues to reject combined table-property modifications withproperties.size() != 1. Thereforepartition.inverted_index_storage_formatreaches its schema-version update branch only as the sole property; a combined request fails rather than silently skipping the update.partitionId -> baseSchemaVersion. It assigns a base shadow schema version for each format represented by existing partitions and the current table format.CloudSchemaChangeJobV2persists that snapshot in its job JSON (psv), uses it when creating the base shadow tablet metadata for each partition, and restores the selected version on the partition when the shadow index becomes visible. Rollup shadow indexes continue to use their own job schema version.AlterJobV2can be replayed after an FE restart while it is running. Replay must recreate the base shadow index with the same partition-specific schema version that was selected before the restart.Env.replayModifyTablePropertynow accepts legacyModifyTablePropertyOperationLogrecords that do not containindexIdToSchemaVersion, preventing an NPE during edit-log replay.Additional test coverage:
CloudSchemaChangeJobV2TestverifiespartitionIdToBaseSchemaVersionserialization/deserialization and edit-log replay: a base shadow index for a partition restores the recorded version rather than the default shadow version.EnvTestreplays a legacy JSONModifyTablePropertyOperationLogwithoutindexIdToSchemaVersionand verifies that replay completes.cloud_p0/test_partition_cloud_inverted_index_storage_format_rolloutadds a mixed V2/V3 table with a rollup. It blocks a column schema change in RUNNING, verifies that both V2 and V3 partitions have base and rollup shadow indexes, then completes the job and verifies that the base tablet formats remain V2 and V3 respectively.Validation status for this round:
CloudSchemaChangeJobV2Testedit-log replay coverage, theEnvTestlegacyModifyTablePropertyOperationLogreplay coverage, and the Cloud mixed V2/V3 schema-change coverage for both base and rollup shadow indexes.Follow-up: VTabletWriterV2 partition-scoped tablet-schema lookup
Problem:
The VTabletWriterV2 load path caches tablet schemas by index_id only. The same base index ID is shared by every partition, so after a V2 -> V3 -> V2 partition-format rollout, opening or asynchronously fetching two partitions can overwrite the cached schema for the other partition. A write routed to the V3 partition can consequently build its DeltaWriterV2 schema from the V2 tablet schema, producing an inverted-index format mismatch despite CreateReplicaTask having created the tablet with the correct format.
Fix:
Test coverage: