[FLINK-40170][table-planner] Infer update-producing changelog mode for early-fire interval join by weiqingy · Pull Request #28877 · apache/flink

weiqingy · 2026-08-02T22:46:13Z

Part of the FLIP-497 implementation stack under umbrella FLINK-36953. Landing order:

Step	Sub-task	Scope
PR-1a	FLINK-40167	EARLY_FIRE hint surface + option validation (#28353, merged)
PR-1b	FLINK-40168	Thread the hint into the interval join (#28796, merged)
PR-2	FLINK-40169	`target` option (#28827, merged)
PR-3 (this PR)	FLINK-40170	Update-producing changelog mode + insert-only guard
PR-4	FLINK-40171	Runtime early-fire emit + retraction
PR-5	FLINK-40172	Processing-time early fire on an event-time join
PR-6	FLINK-40173	State restore coverage
PR-7	FLINK-40174	User-facing documentation

What is the purpose of the change

With the EARLY_FIRE hint, an outer interval join emits a speculative null-padded row after the delay and corrects it when a match later arrives, so its result is no longer insert-only. This makes the planner infer that update-producing changelog mode, and reject an insert-only downstream with an error that names the hint instead of the generic "doesn't support consuming update changes" message. Inner joins and negative-window joins only ever emit inserts, so the hint stays inert for them and they remain append-only.

Runtime behavior is unchanged. The operator still ignores the hint; emission lands in PR-4.

Brief change log

FlinkChangelogModeInferenceProgram gets its own ModifyKindSet arm for StreamPhysicalIntervalJoin: children still consume insert-only, but the node provides INSERT and, when the hint makes it update-producing, UPDATE. It keeps its place in the UpdateKind and DeleteKind arms.
StreamPhysicalIntervalJoin.produceEarlyFireUpdates gates that on three conditions: the hint is set, the join is outer, and the window span is non-negative.
A tailored TableException when such a join feeds an insert-only downstream.
The compiled-plan round-trip test's sink now declares sink-insert-only=false. Its early-fire outer join produces updates under the new inference, so the previously insert-only sink no longer accepts that pipeline. This is the one existing behavior this PR changes.

Verifying this change

This change added tests and can be verified as follows:

EarlyFireJoinHintTest: an early-fire outer join is inferred as changelogMode=[I,UA]; an insert-only downstream fails with the tailored error; a negative-window join and an inner join both stay [I] while still carrying earlyFireDelay on the join node, so the hint is attached but inert rather than absent.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no (planner inference for the FLIP-497 hint)
If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Code (Anthropic)

flinkbot · 2026-08-02T22:50:16Z

CI report:

5e425a9 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

…fire interval join With the EARLY_FIRE hint, an outer interval join speculatively emits a padded unmatched row after the delay and corrects it when a match later arrives, so it no longer produces insert-only changes. Teach FlinkChangelogModeInferenceProgram to reflect this. Split StreamPhysicalIntervalJoin into its own ModifyKindSet arm: its children still consume insert-only, but the node provides INSERT and, when the hint makes it update-producing, UPDATE. A new produceEarlyFireUpdates accessor gates that on the hint being set, the join being outer, and a non-negative window span, so the hint stays inert for inner joins and negative-window joins (which only ever emit inserts). The interval join keeps its place in the UpdateKind and DeleteKind arms. When such a join feeds an insert-only downstream, planning fails with a tailored error that names the hint, rather than the generic "doesn't support consuming update changes" message. Runtime behavior is unchanged; the operator still ignores the hint. The compiled-plan round-trip test's sink now declares sink-insert-only=false: its early-fire outer join produces updates under the new inference, so the previously insert-only sink no longer accepts the pipeline.

weiqingy · 2026-08-08T04:47:08Z

Hi @RocMarshal, #28827 is merged, so this one is rebased onto master and out of draft. The diff is standalone now: 5 files, adding update-producing changelog inference for an early-fire outer interval join.

One thing worth flagging: this changes an existing test's sink. MySink in EarlyFireJoinHintTest was insert-only, and an early-fire outer join now produces updates, so testEarlyFireJsonPlanRoundTrip no longer plans against it. I declared sink-insert-only=false on that sink rather than weakening the test. That is the one existing behavior this PR changes.

PTAL when you have a moment. Thanks!

Copilot

Pull request overview

This PR updates the Flink Table planner’s changelog-mode inference for StreamPhysicalIntervalJoin to account for EARLY_FIRE: outer interval joins with a non-negative window become update-producing, and insert-only downstream consumers are rejected with a hint-specific error. It also updates planner tests and compiled-plan fixtures to reflect the new inferred changelog mode.

Changes:

Infer UPDATE (in addition to INSERT) as produced changelog for early-fire outer interval joins with non-negative windows, while still requiring insert-only inputs.
Add a tailored TableException when an early-fire update-producing interval join feeds an insert-only downstream.
Extend/update test coverage and compiled-plan round-trip fixtures to validate changelog inference and sink compatibility.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala	Adds dedicated `StreamPhysicalIntervalJoin` handling to infer update-producing modify kinds under `EARLY_FIRE` and throw a tailored insert-only error.
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalIntervalJoin.scala	Introduces `produceEarlyFireUpdates` to detect when `EARLY_FIRE` makes the join update-producing (outer join + non-negative window).
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/hints/stream/EarlyFireJoinHintTest.java	Adds tests asserting inferred changelog modes and the insert-only downstream failure behavior.
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/hints/stream/EarlyFireJoinHintTest.xml	Adds expected optimized-plan outputs for new test cases, including changelog modes for inner/negative-window/outer joins.
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/hints/stream/EarlyFireJoinHintTest_jsonplan/testEarlyFireJsonPlanRoundTrip.out	Updates JSON plan fixture to reflect non-insert-only sink configuration and updated inferred input changelog mode.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

weiqingy · 2026-08-08T17:49:09Z

+        if (intervalJoin.produceEarlyFireUpdates && !providedTrait.satisfies(requiredTrait)) {
+          throw new TableException(
+            s"$requester is insert-only, but the EARLY_FIRE hint makes this outer interval join " +
+              "produce update changes (a padded row is emitted speculatively and later corrected " +
+              "on a match). Remove the EARLY_FIRE hint, or write into a downstream/sink that " +
+              "accepts update changes.")
+        }


Reworded. Went with "doesn't support consuming update changes" to match the existing phrasing in this file (line 609) rather than "requires insert-only", so it reads like the other messages while still naming the hint.

weiqingy · 2026-08-08T17:49:10Z

+  def produceEarlyFireUpdates: Boolean =
+    earlyFireDelay != null && getJoinType.isOuterJoin &&
+      (windowBounds.getLeftUpperBound - windowBounds.getLeftLowerBound) >= 0


Changed to upper >= lower.

… window bounds directly The insert-only error named the requester as being insert-only, but the requester is the consumer that requires insert-only changes. Use the "doesn't support consuming" phrasing already used elsewhere in this file. Compare the window bounds directly instead of testing the sign of their difference, which cannot overflow and states the non-negative-window condition more plainly.

weiqingy · 2026-08-08T18:33:24Z

Copilot's two comments are resolved in 5e425a9: the insert-only error now uses the same "doesn't support consuming ... changes" wording as the rest of the file, and the window-span check compares the bounds directly instead of subtracting them.

cc @RocMarshal, this one is ready for review when you have time. Thanks!

RocMarshal

LGTM +1

Given the lack of further responses over the past days,
we will proceed with merging this PR to keep things moving. If there are additional suggestions, please feel free to submit a hotfix or follow-up PR.

Thanks!

RocMarshal · 2026-08-10T13:36:13Z

Thanks @weiqingy for the hard work.

weiqingy force-pushed the FLINK-36953-pr3-changelog branch from 8f7448f to 4f6ef24 Compare August 8, 2026 04:27

weiqingy marked this pull request as ready for review August 8, 2026 04:46

RocMarshal requested a lite review from Copilot August 8, 2026 09:26

Copilot started reviewing on behalf of RocMarshal August 8, 2026 09:27 View session

Copilot AI reviewed Aug 8, 2026

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Aug 8, 2026

RocMarshal approved these changes Aug 10, 2026

View reviewed changes

RocMarshal merged commit 21567fb into apache:master Aug 10, 2026

This was referenced Aug 10, 2026

[FLINK-40171][table-runtime] Emit and retract early-fire results in the interval join operator #28952

Merged

[FLINK-40172][table-runtime] Support processing-time early fire on a row-time interval join #28953

Open

weiqingy mentioned this pull request Aug 30, 2026

[FLINK-40173][table-planner] Add restore coverage for early-fire interval join #29045

Draft

1 task

Navigation Menu

Uh oh!

[FLINK-40170][table-planner] Infer update-producing changelog mode for early-fire interval join - #28877

[FLINK-40170][table-planner] Infer update-producing changelog mode for early-fire interval join#28877
RocMarshal merged 2 commits into
apache:masterfrom
weiqingy:FLINK-36953-pr3-changelog

weiqingy commented Aug 2, 2026 •

edited

Loading

Uh oh!

flinkbot commented Aug 2, 2026 •

edited

Loading

Uh oh!

weiqingy commented Aug 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

weiqingy Aug 8, 2026

Uh oh!

weiqingy Aug 8, 2026

Uh oh!

weiqingy commented Aug 8, 2026

Uh oh!

RocMarshal left a comment

Uh oh!

RocMarshal commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

weiqingy commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

weiqingy commented Aug 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

weiqingy Aug 8, 2026

Choose a reason for hiding this comment

Uh oh!

weiqingy Aug 8, 2026

Choose a reason for hiding this comment

Uh oh!

weiqingy commented Aug 8, 2026

Uh oh!

RocMarshal left a comment

Choose a reason for hiding this comment

Uh oh!

RocMarshal commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

weiqingy commented Aug 2, 2026 •

edited

Loading

flinkbot commented Aug 2, 2026 •

edited

Loading