[fix](binlog) Align row binlog after schema with before values - #65438
Merged
Merged
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Row binlog MIN_DELTA outputs update_before and update_after rows through one unified output schema. Before this change, FE generated the after value column by copying the base value column definition directly. When the base column was NOT NULL and complex-typed, the update_before row still came from the nullable `__BEFORE__` mirror column but had to be written into the non-nullable after slot. That schema mismatch could crash BE during MIN_DELTA row materialization with a bad cast from `ColumnNullable` to the target complex column type. This change makes generated after value columns follow the same nullable/default-cleared contract as before value columns, adds FE unit coverage for row binlog schema generation, and adds a regression case that reproduces the scenario with a normal row binlog MIN_DELTA query on a `BITMAP NOT NULL` MoW table.
### Release note
None
### Check List (For Author)
- Test: FE UT + Regression test
- FE Unit Test: `./run-fe-ut.sh --run org.apache.doris.catalog.OlapTableRowBinlogSchemaTest`
- Regression test: `./run-regression-test.sh --run -d row_binlog_p0 -s test_binlog_changes_syntax -forceGenOut`
- Behavior changed: Yes (row binlog generated after value columns are now nullable and have cleared defaults to match MIN_DELTA unified output schema)
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yujun777
changed the title
3 tasks
Contributor
Author
|
run buildall |
morrySnow
approved these changes
Jul 10, 2026
Contributor
|
/review |
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
Bot
added
approved
Contributor
|
PR approved by anyone and no changes requested. |
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 10, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65438 Problem Summary: Row binlog MIN_DELTA materializes update-before rows and update-after rows through one unified output schema. Before this change, FE generated the after value column by copying the user column definition directly. When the user column was a complex NOT NULL type such as BITMAP, the update-before row still came from the nullable __DORIS_BEFORE__ mirror column but had to be written into the non-nullable after slot. That schema mismatch could crash BE during MIN_DELTA row materialization with a bad cast from ColumnNullable to the target complex column type. This change makes generated after value columns follow the same nullable and default-cleared contract as before value columns so the unified MIN_DELTA output schema can safely carry both before rows and after rows. ### Release note None ### Check List (For Author) - Test: Manual test - Reproduced the BE crash by temporarily reverting this change and rerunning the row binlog MIN_DELTA bitmap regression in doris-test - Behavior changed: Yes (row binlog generated after value columns now match the unified MIN_DELTA output schema used by both before rows and after rows) - Does this need documentation: No
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review conclusion: request changes.
Critical checkpoints:
- Goal/test: the fresh-schema path is covered by the new FE unit assertions and bitmap regression case, but the fix does not cover existing persisted row-binlog metadata after upgrade.
- Scope/focus: the code change is small and focused; the missing compatibility path is the blocking gap.
- Concurrency/lifecycle: no new concurrency or locking path is introduced. The relevant lifecycle issue is FE metadata load/replay of persisted row-binlog index schemas.
- Configuration/compatibility: no new config is added. Compatibility is affected because old row-binlog after-value columns can remain non-null/default-carrying in MaterializedIndexMeta.
- Parallel paths: new row-binlog schema creation and add-column paths use the helper; metadata-load compatibility does not.
- Tests: existing tests exercise fresh metadata and a new bitmap MIN_DELTA case, but not legacy persisted metadata normalization. I did not run FE/regression tests locally because this checkout is missing thirdparty/installed/bin/protoc.
- Observability/performance/transactions: no new logging, hot-path performance, transaction, or persistence-write concerns beyond the metadata compatibility issue.
User focus: no additional user-provided review focus was supplied.
Subagent conclusions: optimizer-rewrite found no valuable optimizer/rewrite issue. tests-session-config proposed TSC-1, which was accepted as M-1 and submitted as the inline comment. No candidates were dismissed as duplicates; existing inline threads were empty. Convergence round C1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same one-comment final set.
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
TPC-H: Total hot run time: 30134 ms |
Contributor
TPC-DS: Total hot run time: 179837 ms |
Contributor
ClickBench: Total hot run time: 24.99 s |
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 10, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65438 Problem Summary: Row binlog MIN_DELTA materializes update-before rows and update-after rows through one unified output schema. Before this change, FE generated the after value column by copying the user column definition directly. When the user column was a complex NOT NULL type such as BITMAP, the update-before row still came from the nullable __DORIS_BEFORE__ mirror column but had to be written into the non-nullable after slot. That schema mismatch could crash BE during MIN_DELTA row materialization with a bad cast from ColumnNullable to the target complex column type. This change makes generated after value columns follow the same nullable and default-cleared contract as before value columns so the unified MIN_DELTA output schema can safely carry both before rows and after rows. ### Release note None ### Check List (For Author) - Test: Manual test - Reproduced the BE crash by temporarily reverting this change and rerunning the row binlog MIN_DELTA bitmap regression in doris-test - Behavior changed: Yes (row binlog generated after value columns now match the unified MIN_DELTA output schema used by both before rows and after rows) - Does this need documentation: No
Contributor
FE Regression Coverage ReportIncrement line coverage |
69 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: #65418
Related PR: None
Problem Summary:
Row binlog
MIN_DELTAdoes not output separate schemas for update-before rows and update-after rows. Instead, it materializes both row kinds through one unified output schema, which is the generated after/output value schema.That means an update-before row also has to be written into the after value slots. For MoW row binlog queries with historical values enabled, the before values come from the nullable
__DORIS_BEFORE_*mirror columns. Before this change, FE generated after value columns by copying the user column definition directly. When the user column was a complexNOT NULLtype such asBITMAP, the after/output slot stayed non-nullable, butMIN_DELTAstill had to write the nullable before value into that same slot.This schema mismatch could crash BE during
MIN_DELTArow materialization with a bad cast fromColumnNullabletoColumnComplexType<BITMAP>. The fix is to make generated after value columns follow the same nullable and default-cleared contract as before value columns, so the unifiedMIN_DELTAoutput schema can safely carry both before rows and after rows.This PR also adds FE unit coverage for row binlog schema generation and a normal row binlog regression case that reproduces the crash with a
BITMAP NOT NULLcolumn onMIN_DELTA.Release note
None
Check List (For Author)
./run-fe-ut.sh --run org.apache.doris.catalog.OlapTableRowBinlogSchemaTest./run-regression-test.sh --run -d row_binlog_p0 -s test_binlog_changes_syntax -forceGenOutColumn.javachange, rebuilt FE, reran./run-regression-test.sh --run -d row_binlog_p0 -s test_binlog_changes_syntax, and reproducedF20260710 11:13:48.523105 ... Bad cast from type:doris::ColumnNullable to doris::ColumnComplexType<(doris::PrimitiveType)22>inbe.outMIN_DELTAoutput schema used by both before rows and after rows)