Skip to content

[FLINK-40466][table] Refactor defining table arguments in PTF Test Harness for more complex table arguments - #29018

Merged
fhueske merged 1 commit into
apache:masterfrom
autophagy:FLINK-40466-harness-table-argument-builder
Sep 3, 2026
Merged

[FLINK-40466][table] Refactor defining table arguments in PTF Test Harness for more complex table arguments#29018
fhueske merged 1 commit into
apache:masterfrom
autophagy:FLINK-40466-harness-table-argument-builder

Conversation

@autophagy

Copy link
Copy Markdown
Contributor

What is the purpose of the change

While working on extending ProcessTableFunctionTestHarness to better support changelog processing modes and order by/late data handling, I noticed the current way of building table arguments started suffer from duplicated effort. For example, it started resulting in an API surface shape that looked like:

    ProcessTableFunctionTestHarness.ofClass(UpsertConsumerPTF.class)
    .withTableArgument("input", DataTypes.of("ROW<key STRING, value INT>"))
    .withPartitionBy("input", "key")
    .withTableArgumentUpsertKey("input", "key")
    .withTableArgumentChangelogMode("input", ChangelogMode.upsert(true))
    .build())

Defining these features of the table argument, reusing the same string table argument id over and over again, seemed fragile and repetitious. This PR replaces this way of defining table arguments with a builder, so the above example might look like this instead:

ProcessTableFunctionTestHarness.ofClass(UpsertConsumerPTF.class)
.withTableArgument(
        TableArgument.forArgument("input")
            .type(DataTypes.of("ROW<key STRING, value INT>"))
            .partitionBy("key")
            .withUpsertKey(key")
            .withChangelogMode(ChangelogMode.upsert(true))
            .build())

This change removes the previous API surface for defining a table argument, plus removes the old partition by method. I figured since this is still public evolving and unreleased, mutating the api surface here is still okay. Happy to be corrected though!

Brief change log

  • Added a builder based configuration option for defining table arguments in the PTF test harness.

Verifying this change

This change is already covered by existing tests, such as the ProcessTableFunctionTestHarnessTest class.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (yes(ish))
  • If yes, how is the feature documented? (docs / JavaDocs)

@flinkbot

flinkbot commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@autophagy
autophagy force-pushed the FLINK-40466-harness-table-argument-builder branch 2 times, most recently from d7160ff to ccebaea Compare August 25, 2026 21:03

@fhueske fhueske left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the refactoring.
I agree, it's much nicer this way. Backwards compatibility is not an issue since this feature has not been released yet.

I left one bikeshedding comment, feel free to ignore if you disagree.
Just let me know if you'd like to adjust or merge as is.

Cheers, Fabian

ProcessTableFunctionTestHarness.ofClass(DoublePTF.class)
.withTableArgument("input", DataTypes.of("ROW<value INT>"))
.withTableArgument(
TableArgument.forArgument("input")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit
TableArgument.forArgument() reads a bit repetitive. What do you think about this:

Suggested change
TableArgument.forArgument("input")
TableArgument.name("input")

or forName()?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think forName is great (and yes, less repetitive 😅 )

@autophagy
autophagy force-pushed the FLINK-40466-harness-table-argument-builder branch from ccebaea to 65b320c Compare September 3, 2026 11:42
@fhueske
fhueske merged commit e0fed5b into apache:master Sep 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants