[improve](streaming-job) Support MySQL ADD DROP schema changes#65325
Merged
JNSimba merged 12 commits intoJul 14, 2026
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
/review |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the fs_brokers/cdc_client streaming-job CDC pipeline to handle MySQL schema-change events by enabling Debezium schema change capture, parsing MySQL ALTER TABLE DDL to detect supported ADD/DROP COLUMN operations, emitting corresponding Doris DDLs, and adding regression + unit tests to validate behavior (including FE restart scenarios).
Changes:
- Enable MySQL schema-change events in the source reader and propagate them through the record iterator/offset tracking.
- Implement MySQL DDL parsing and schema-change deserialization to generate Doris
ALTER TABLE ... ADD/DROP COLUMNoperations. - Add regression suites and Java unit tests covering MySQL ADD/DROP, mixed/unsupported DDL, and FE restart resilience.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc.groovy | End-to-end regression test for MySQL streaming job handling ADD/DROP column with follow-up DML. |
| regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc_restart_fe.groovy | Regression test verifying schema-change handling persists/recovers across FE restart. |
| regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc_latest.groovy | Regression test validating behavior when starting from offset=latest and applying schema changes afterward. |
| regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc_advanced.groovy | Regression test covering advanced/mixed DDL (FIRST/AFTER, mixed ADD+DROP, and unsupported MODIFY/CHANGE). |
| regression-test/data/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc.out | Expected output for test_streaming_mysql_job_sc. |
| regression-test/data/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc_restart_fe.out | Expected output for test_streaming_mysql_job_sc_restart_fe. |
| regression-test/data/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc_latest.out | Expected output for test_streaming_mysql_job_sc_latest. |
| regression-test/data/job_p0/streaming_job/cdc/test_streaming_mysql_job_sc_advanced.out | Expected output for test_streaming_mysql_job_sc_advanced. |
| fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/utils/SchemaChangeHelperTest.java | Adds unit tests for MySQL Debezium Column → Doris type mapping. |
| fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlSourceReaderTest.java | Verifies generated MySQL source config enables schema change events. |
| fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/source/parse/mysql/CustomMySqlAntlrDdlParserTest.java | Unit tests for parsing MySQL ADD/DROP and marking unsupported column changes. |
| fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/SchemaChangeHelper.java | Adds MySQL Column→Doris type conversion used for emitted ADD COLUMN DDL. |
| fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlSourceReader.java | Enables includeSchemaChanges(true) and forwards schema-change events via the iterator/offset updates. |
| fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/parse/mysql/MySqlSchemaChange.java | New model type representing parsed MySQL schema-change operations (ADD/DROP/UNSUPPORTED). |
| fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/parse/mysql/CustomMySqlAntlrDdlParserListener.java | Custom parse-tree listener wiring Debezium listeners plus Doris-specific ALTER TABLE listener. |
| fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/parse/mysql/CustomMySqlAntlrDdlParser.java | Custom MySQL DDL parser exposing parsed column changes for Doris handling. |
| fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/parse/mysql/CustomAlterTableParserListener.java | Extracts column ADD/DROP (and flags unsupported changes) during MySQL ALTER TABLE parsing. |
| fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/deserialize/MySqlDebeziumJsonDeserializer.java | Implements MySQL schema-change deserialization: baseline checks, DDL parsing, and Doris DDL emission. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+141
to
+146
| Preconditions.checkNotNull(column.typeName()); | ||
| String mysqlTypeName = column.typeName().toUpperCase(); | ||
| String[] typeFields = mysqlTypeName.split(" "); | ||
| String mysqlType = typeFields[0]; | ||
| boolean unsigned = typeFields.length > 1 && "UNSIGNED".equals(typeFields[1]); | ||
| int length = column.length(); |
JNSimba
changed the title
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review completed. I found one blocking CI/style issue: the CDC client Spotless validation fails on changed Java sources, so this should be fixed before relying on CI or test results.
Critical checkpoint conclusions:
- Goal/test proof: the PR targets MySQL ADD/DROP column schema changes for streaming jobs and adds unit/regression coverage for main, latest-offset, mixed-DDL, and FE-restart flows. Local validation is blocked by formatting.
- Scope: the implementation is focused on CDC client schema-change parsing/deserialization and related regression tests.
- Concurrency/lifecycle: no new independent concurrent state was added beyond the existing reader/coordinator path; schema DDL execution still happens before in-memory schema update in the write path.
- Compatibility/persistence: FE schema baseline serialization/update paths were checked; restart coverage was added, but could not be run here.
- Parallel paths: from-to write path executes schema changes; TVF/debug fetch remains DML-only, consistent with the existing PostgreSQL note.
- Tests/results: new expected outputs are ordered. I could not run functional tests in this shallow runner because
fe-common/ generatedorg.apache.doris.job.cdcclasses are unavailable after skipping Spotless; the PR body also says tests were not run. - User focus: no additional user-provided focus points were supplied.
- Existing review context: I did not duplicate the existing Copilot threads for ADD COLUMN null lookup or ZEROFILL unsigned handling.
Subagent conclusions:
tests-session-configcandidate TSC-1 was accepted as inline comment M-1.optimizer-rewritecandidate OPT-1 was dismissed/merged as duplicate/unsupported by evidence:SERIALis normalized by Debezium toBIGINT UNSIGNED, and the remaining ADD-column null failure overlaps the existing Copilot threads.- Final convergence round 1 ended with both live subagents replying
NO_NEW_VALUABLE_FINDINGSfor the same ledger/comment set.
| schemaChanges.add( | ||
| SchemaChangeOperation.dropColumn( | ||
| targetTable, | ||
| columnName, |
Contributor
There was a problem hiding this comment.
mvn -f fs_brokers/cdc_client/pom.xml -DskipTests -Dspotless.check.skip=false validate currently fails in spotless:check before the module compiles. Spotless reports five changed Java files need formatting, including this file around the buildDropColumnSql wrapping here and the return below, plus SchemaChangeHelper.java, CustomMySqlAntlrDdlParser.java, CustomAlterTableParserListener.java, and CustomMySqlAntlrDdlParserListener.java. Please run the configured formatter for this module and commit the result; otherwise the CDC client CI/style gate will fail before tests run.
Member
Author
|
/review |
Member
Author
|
run buildall |
Contributor
There was a problem hiding this comment.
Reviewed the full CDC schema-change path for this PR and checked the existing review threads before submitting.
Findings: three generated .out files contain added blank EOF lines that fail git diff --check.
Validation:
mvn -f fs_brokers/cdc_client/pom.xml -DskipTests -Dspotless.check.skip=false validatepassed locally.git diff --check 4f3cd539b6fd3c6fd55b4148e7ab0ac255f9dee2..HEADfailed on the three commented.outfiles.- Targeted Maven tests for the CDC client did not reach execution in this standalone checkout because generated
org.apache.doris.job.cdc.*classes were missing.
I did not duplicate the existing review threads for ZEROFILL unsigned handling, ADD-column null resolution, or the earlier Spotless report.
JNSimba
force-pushed
the
mysql-schemachange-add-drop-master
branch
from
July 7, 2026 14:51
c561598 to
d1db168
Compare
Member
Author
|
run buildall |
Member
Author
|
/review |
Contributor
There was a problem hiding this comment.
I found three additional correctness issues in the new MySQL ADD COLUMN type mapping. They are distinct from the existing live threads for Spotless, .out EOF blanks, ZEROFILL/UNSIGNED, add-column null resolution, and parser-test metadata coverage.
Critical checkpoint conclusions:
- Goal/test proof: the PR implements MySQL ADD/DROP schema-change handling and adds unit/regression coverage, but the added type mapper does not yet preserve several source types consistently with initial table creation.
- Scope: the implementation is focused on CDC client schema-change parsing/deserialization and related tests.
- Concurrency/lifecycle: no new independent shared-state concurrency issue was found; the coordinator still flushes before DDL, executes schema changes, applies updated schemas, and persists the schema baseline with offset commit.
- Compatibility/persistence: restart/baseline flow was checked statically and is covered by a new restart suite, but unsupported DDL remains intentionally skipped while advancing baseline.
- Parallel paths: no optimizer/Nereids changes apply. PostgreSQL-style unsupported-change behavior appears intentional for this MySQL path.
- Tests/results: I could not run the focused Maven tests to execution in this runner; compilation stops on missing org.apache.doris.job.cdc.* FE classes. thirdparty/installed is also absent, so validation here is static plus GitHub metadata/patch inspection.
- User focus: no additional user-provided focus points were supplied.
Subagent conclusions:
- optimizer-rewrite proposed OR-1, accepted as inline comment M-1.
- tests-session-config reported NO_NEW_VALUABLE_FINDINGS; its duplicate checks for existing formatting/.out/test-coverage threads were dismissed as already covered.
- Main review added M-2 and M-3 for additional MySQL type-mapping mismatches.
- Convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same final ledger/comment set.
Member
Author
|
run buildall |
Member
Author
|
/review |
Member
Author
|
run buildall |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 10th, 2026 6:44 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Member
Author
|
run buildall |
3 similar comments
Member
Author
|
run buildall |
Member
Author
|
run buildall |
Member
Author
|
run buildall |
Member
Author
|
/review |
Contributor
There was a problem hiding this comment.
Automated follow-up review for PR 65325 at be25955.
I did not find a new distinct inline issue to add on this pass. I am leaving this as a comment-only review, not an approval, because the PR already has unresolved CHANGES_REQUESTED review state from existing threads.
User focus: no additional user-provided review focus was supplied.
Critical checkpoint conclusions:
- Goal and scope: the change is scoped to the CDC client MySQL ADD/DROP schema-change path plus regression coverage. I reviewed the parser/deserializer path, schema-change execution, schema baseline commit, retry behavior, and added CDC/regression tests.
- Correctness: DDL-before-DML ordering is preserved by flushing before schema DDL execution and writing schema-event records only after
executeChangesandapplySchemaChange. Schema baselines are serialized into the committed offset when DDL is executed. - Compatibility/configuration: MySQL Debezium schema events are enabled with
includeSchemaChanges(true), and the source reload path restores table schemas from committed request state. I did not find a new cross-version or config compatibility issue in the reviewed path. - Concurrency/lifecycle: I did not find new unprotected shared state or lifecycle ordering issues in the changed path. Retry/idempotency behavior is handled through the schema-change manager and is covered by the new retry test.
- Parallel paths: I compared the MySQL event-driven path against the existing PostgreSQL schema-change flow and the from-to / TVF test coverage. No new uncovered split-path issue was found.
- Tests/results: the added result-producing regression queries use deterministic
ORDER BYor single aggregate rows. Existing comments already cover earlier formatting/output issues, and those appear stale for the current head. - Existing review threads: the remaining MySQL unsigned/ZEROFILL type-mapping problem is still covered by existing threads, so I did not duplicate it. Prior decimal, string length, datetime precision, parser-failure, parser-null, Spotless, and
.outEOF concerns are either already covered or resolved/stale in this head. - Observability/performance: the schema-change path logs the relevant DDL handling, and I did not find a hot-path performance regression; these events are not on the normal per-row path.
Validation:
- Passed:
mvn -f fs_brokers/cdc_client/pom.xml -DskipTests -Dspotless.check.skip=false spotless:check - Not completed:
mvn -f fs_brokers/cdc_client/pom.xml -DskipTests test-compilefailed before tests because the standalonecdc_clientmodule compile could not resolve existingorg.apache.doris.job.cdc.*classes in this checkout/classpath. - Not run: full Doris build and regression suites, because this review checkout lacks
.worktree_initializedandthirdparty/installed.
Subagent conclusions:
- optimizer-rewrite: returned
NO_NEW_VALUABLE_FINDINGSafter re-reading the final ledger/comment set. - tests-session-config: returned
NO_NEW_VALUABLE_FINDINGSafter re-reading the final ledger/comment set.
JNSimba
force-pushed
the
mysql-schemachange-add-drop-master
branch
from
July 9, 2026 08:08
4446b13 to
b0adfb8
Compare
Member
Author
|
run buildall |
Contributor
There was a problem hiding this comment.
Requesting changes based on the current live head (b0adfb81f88278a72837378a086a8d9e687f5d5b). I found twelve additional non-duplicate issues after deduplicating against existing review threads.
Critical checkpoints covered:
- Goal/scope: the PR adds CDC schema-change support for MySQL/Postgres ADD/DROP paths, but ADD COLUMN still diverges from initial table creation for several source types, unsupported type handling, and default/nullability semantics, and the new disable flag is not reachable through normal job source validation.
- Compatibility/persistence: baseline advancement and existing mixed/unsupported ALTER concerns were checked against current threads; the remaining new compatibility issues are type-equivalence drift, unsupported-type acceptance, skipped PostgreSQL Relation baseline advancement, default/backfill/nullability drift, and FE validation of the new source key.
- Tests: the live head added generated expected-output files for the TVF schema-change suites, so I withdrew those two test-output findings. A targeted cdc_client Maven test was attempted earlier but failed during compilation because this checkout cannot resolve Doris job/cdc classes.
- Validation limits: static review only. This worktree is missing
.worktree_initialized,thirdparty/installed, andthirdparty/installed/bin/protoc, so I did not run a full Doris build or regression suite.
No additional user-provided review focus was supplied.
Contributor
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29335 ms |
Contributor
TPC-DS: Total hot run time: 180106 ms |
Contributor
ClickBench: Total hot run time: 24.85 s |
Member
Author
|
/review |
Contributor
There was a problem hiding this comment.
Automated review completed.
I found one distinct issue not already covered by the existing review threads: the PostgreSQL direct fetch/debug endpoints still allow schema-change records by default but do not provide the Doris target database or handle SCHEMA_CHANGE results, so an ADD/DROP Relation event can fail while constructing Doris DDL.
Critical checkpoint conclusions:
- Goal/test coverage: the PR adds MySQL ADD/DROP schema-change handling and TVF schema-change disabling with new unit/regression coverage, but the direct PostgreSQL fetch path remains uncovered.
- Scope/focus: no additional user-provided review focus was supplied.
- Parallel paths/compatibility: MySQL fetch-path missing target DB, FE validation for schema_change_enabled, mapper parity, unsupported baseline advancement, positional ADD, mixed ALTER atomicity, default/nullability behavior, and during-snapshot determinism are already covered by existing inline threads, so I did not duplicate them. The new inline comment is the separate PostgreSQL fetch/debug instance.
- Concurrency/lifecycle: reviewed reader reuse, offset/schema persistence, retry/idempotency, and FE restart paths; no additional concurrency issue was substantiated beyond existing threads.
- Config/session propagation: reader-level propagation of schema_change_enabled is present; the from-to FE validation gap is already threaded.
- Validation: static review only.
git diff --check a35099df1f5634296dfd3b67f9a13b84ad91d719..HEADpassed, but FE/CDC builds and regression tests were not run because this checkout is not worktree-initialized and lacksthirdparty/installed/thirdparty/installed/bin/protoc.
Subagent conclusions:
- optimizer-rewrite proposed optimizer-rewrite-001; I verified and accepted it as the single inline comment.
- tests-session-config reported no new valuable finding; its noted concerns were duplicates of existing threads or dismissed by current evidence.
- Final convergence round 1 ended with both live subagents replying
NO_NEW_VALUABLE_FINDINGSfor the same ledger and one-comment set.
| col.name(), | ||
| SchemaChangeHelper.buildAddColumnSql( | ||
| db, targetTable, col.name(), colType, null, null))); | ||
| db, targetTable, col.name(), colType))); |
Contributor
There was a problem hiding this comment.
The PostgreSQL fetch/debug paths still have the same write-path-only target DB assumption. PostgresSourceReader defaults schema_change_enabled to true, and both /api/fetchRecords and /api/fetchRecordStream pass the raw FetchRecordRequest config into sourceReader.deserialize(...); unlike writeRecords(), they never add Constants.DORIS_TARGET_DB and they do not handle SCHEMA_CHANGE results. If a direct Postgres fetch sees a Relation ADD/DROP, this call builds Doris DDL with db == null, and SchemaChangeHelper.quoteTableIdentifier() dereferences it while the fetch path cannot execute that DDL anyway. The normal FE TVF now forces schema_change_enabled=false, but the public/debug fetch endpoints remain exposed to this by default. Please keep schema-change records disabled for all fetch paths, or carry target context and handle schema-change results there instead of building DDL with a null target DB.
Member
Author
There was a problem hiding this comment.
FE has already been configured; this scenario does not exist.
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
12 tasks
github-actions Bot
pushed a commit
that referenced
this pull request
Jul 16, 2026
### What problem does this PR solve? Issue Number: None Problem Summary: Support MySQL ADD/DROP column schema changes in streaming jobs. MySQL schema change events now update the FE schema baseline, execute Doris ADD/DROP column DDLs for supported changes, and keep unsupported changes aligned with the existing PostgreSQL schema-change behavior. This also keeps `cdc_stream` TVF behavior stable when source-side schema changes happen during incremental consumption. The TVF path disables schema-change handling internally, skips schema change records in the CDC client, and continues to consume DML that matches the query projection for both MySQL and PostgreSQL sources..
JNSimba
added a commit
that referenced
this pull request
Jul 23, 2026
### What problem does this PR solve? Related PR: #65325 Streaming jobs currently support MySQL and PostgreSQL CDC sources but cannot use OceanBase as a source. This change adds OceanBase MySQL compatibility mode as a streaming job data source. It introduces OceanBase source type handling in FE and the CDC client, reuses the MySQL-compatible CDC processing path, validates source configuration and compatibility mode before creating target tables, and supports initial, snapshot, latest, earliest, and specific startup offsets. The OceanBase third-party environment exposes separate JDBC and CDC ports. Test coverage includes basic synchronization, data types, startup offsets, schema changes, and FE restart recovery.
github-actions Bot
pushed a commit
that referenced
this pull request
Jul 23, 2026
### What problem does this PR solve? Related PR: #65325 Streaming jobs currently support MySQL and PostgreSQL CDC sources but cannot use OceanBase as a source. This change adds OceanBase MySQL compatibility mode as a streaming job data source. It introduces OceanBase source type handling in FE and the CDC client, reuses the MySQL-compatible CDC processing path, validates source configuration and compatibility mode before creating target tables, and supports initial, snapshot, latest, earliest, and specific startup offsets. The OceanBase third-party environment exposes separate JDBC and CDC ports. Test coverage includes basic synchronization, data types, startup offsets, schema changes, and FE restart recovery.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Problem Summary: Support MySQL ADD/DROP column schema changes in streaming jobs. MySQL schema change events now update the FE schema baseline, execute Doris ADD/DROP column DDLs for supported changes, and keep unsupported changes aligned with the existing PostgreSQL schema-change behavior.
This also keeps
cdc_streamTVF behavior stable when source-side schema changes happen during incremental consumption. The TVF path disables schema-change handling internally, skips schema change records in the CDC client, and continues to consume DML that matches the query projection for both MySQL and PostgreSQL sources.Release note
Support MySQL ADD/DROP column schema changes for streaming jobs.
cdc_streamTVF continues compatible incremental consumption when source-side schema changes are observed.