Skip to content

changefeedccl: support CREATE DATABASE CHANGEFEED syntax - #147663

Merged
craig[bot] merged 1 commit into
cockroachdb:masterfrom
KeithCh:db-level-changefeed-parser
Jul 2, 2025
Merged

craig[bot] merged 1 commit into
cockroachdb:masterfrom
KeithCh:db-level-changefeed-parser

Conversation

@KeithCh

@KeithCh KeithCh commented Jun 2, 2025

Copy link
Copy Markdown

Add basic syntax support for database-level
changefeeds.
CREATE DATABASE CHANGEFEED for foo
This PR only allows the aforementioned statement
to be parsed; no changefeed is created.

Epic: CRDB-1421
Resolves: #147369

Release note: None

@blathers-crl

blathers-crl Bot commented Jun 2, 2025

Copy link
Copy Markdown

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

It looks like your PR touches SQL parser code but doesn't add or edit parser tests. Please make sure you add or edit parser tests if you edit the parser.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity

Copy link
Copy Markdown
Member

This change is Reviewable

@blathers-crl

blathers-crl Bot commented Jun 3, 2025

Copy link
Copy Markdown

It looks like your PR touches SQL parser code but doesn't add or edit parser tests. Please make sure you add or edit parser tests if you edit the parser.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl

blathers-crl Bot commented Jun 3, 2025

Copy link
Copy Markdown

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

Comment thread pkg/sql/parser/sql.y Outdated
@KeithCh
KeithCh marked this pull request as ready for review June 4, 2025 20:05
@KeithCh
KeithCh requested review from a team as code owners June 4, 2025 20:05
@KeithCh
KeithCh requested review from a team, andyyang890 and asg0451 and removed request for a team June 4, 2025 20:05
@KeithCh KeithCh changed the title changefeedccl: Support CREATE DATABASE CHANGEFEED syntax changefeedccl: support CREATE DATABASE CHANGEFEED syntax Jun 5, 2025
Comment thread pkg/sql/sem/tree/changefeed.go Outdated
Comment thread pkg/sql/sem/tree/changefeed.go
Comment thread pkg/sql/parser/sql.y Outdated
Comment thread pkg/sql/parser/sql.y Outdated

@asg0451 asg0451 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some thoughts; also can you get someone from sql foundations to actually review the parser changes?

Comment thread pkg/ccl/changefeedccl/alter_changefeed_stmt.go Outdated
Comment thread pkg/ccl/changefeedccl/changefeed_stmt.go Outdated
tableOnlyTargetList := tree.BackupTargetList{}
for _, t := range changefeedStmt.Targets {
tableOnlyTargetList.Tables.TablePatterns = append(tableOnlyTargetList.Tables.TablePatterns, t.TableName)
target, ok := t.(*tree.ChangefeedTableTarget)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't love having to do these assertions everywhere. is this (having target be an interface) the right abstraction?

@KeithCh KeithCh Jun 9, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess if we don't plan on supporting multiple types of targets per changefeed it's not really needed.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One benefit is that we don't have to modify the signature of functions that takes in tree.ChangefeedTargets, and the way we loop through them. If we didn't have the abstraction we would have to do something like

type ChangefeedTargets struct {
 ChangefeedDatabseTarget ChangefeedDatabseTarget[]
 ChangefeedTableTarget ChangefeedTableTargets[]
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider something like this instead, with validation on CREATE ... CHANGEFEED that ensures that all targets are the same level?

// ChangefeedTargets represents a list of database objects to be watched by a
// changefeed.
type ChangefeedTargets []ChangefeedTarget

// ChangefeedTarget represents a database-level or table-level changefeed
// target.
type ChangefeedTarget struct {
	// Name is the name of a database or table.
	Name
	// FamilyName is the name of a column family to be watched. It is the empty
	// string for database-level targets.
	FamilyName Name
}

@KeithCh KeithCh Jun 10, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is possible, but we can't rely on FamilyName to determine whether it is a DB target. We might have with split_column_families set in which case it would be empty. It would have to be like

type ChangefeedTarget struct {
	// Name is the name of a database or table.
	Name
	// FamilyName is the name of a column family to be watched. It is the empty
	// string for database-level targets.
	FamilyName Name
	
	// TargetType is an enum (table, changefeed)
	TargetType ChangefeedTargetType
}

and then we check the TargetType when we work with the object.

But that doesn't seem very idomatic.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems idiomatic to me, unless I'm missing something. Here's one example of a similar pattern:

// CTE represents a common table expression inside of a WITH clause.
type CTE struct {
Name AliasClause
Mtr CTEMaterializeClause
Stmt Statement
}
// CTEMaterializeClause represents either MATERIALIZED, NOT MATERIALIZED, or an
// empty materialization clause.
type CTEMaterializeClause int8
const (
// CTEMaterializeDefault represents an empty materialization clause.
CTEMaterializeDefault CTEMaterializeClause = iota
// CTEMaterializeAlways represents MATERIALIZED.
CTEMaterializeAlways
// CTEMaterializeNever represents NOT MATERIALIZED.
CTEMaterializeNever
)

I'd bet there are more examples like this if you look around.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that since FamilyName only exists for table targets it would be nice to have different structs to separate them. I'm ok with your approach if you think it is the best way.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked to @asg0451, we're going to have multiple fields inside CreateChangefeed to represent the different target types.

@KeithCh
KeithCh requested review from a team as code owners June 9, 2025 22:26
@KeithCh
KeithCh requested review from a team, michae2 and msbutler and removed request for a team June 9, 2025 22:26
Comment thread pkg/sql/parser/sql.y Outdated
tableOnlyTargetList := tree.BackupTargetList{}
for _, t := range changefeedStmt.Targets {
tableOnlyTargetList.Tables.TablePatterns = append(tableOnlyTargetList.Tables.TablePatterns, t.TableName)
target, ok := t.(*tree.ChangefeedTableTarget)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider something like this instead, with validation on CREATE ... CHANGEFEED that ensures that all targets are the same level?

// ChangefeedTargets represents a list of database objects to be watched by a
// changefeed.
type ChangefeedTargets []ChangefeedTarget

// ChangefeedTarget represents a database-level or table-level changefeed
// target.
type ChangefeedTarget struct {
	// Name is the name of a database or table.
	Name
	// FamilyName is the name of a column family to be watched. It is the empty
	// string for database-level targets.
	FamilyName Name
}

Comment thread pkg/sql/sem/tree/changefeed.go Outdated
@KeithCh
KeithCh marked this pull request as draft June 10, 2025 18:22
@KeithCh
KeithCh marked this pull request as ready for review June 10, 2025 21:41
Comment thread pkg/ccl/changefeedccl/changefeedbase/target.go
@KeithCh
KeithCh requested a review from asg0451 June 23, 2025 17:41
@asg0451
asg0451 requested a review from mgartner June 24, 2025 16:50
Comment thread pkg/sql/parser/sql.y Outdated
Comment thread pkg/sql/sem/tree/changefeed.go Outdated
Comment thread pkg/sql/sem/tree/changefeed.go
Comment thread pkg/sql/sem/tree/changefeed.go Outdated
Comment thread pkg/sql/sem/tree/changefeed.go Outdated
Comment thread pkg/sql/sem/tree/changefeed.go Outdated
Comment thread pkg/cmd/docgen/diagrams.go
@KeithCh
KeithCh requested a review from asg0451 July 1, 2025 16:11
Comment thread pkg/sql/parser/sql.y Outdated
Comment thread pkg/cmd/docgen/diagrams.go Outdated
Comment thread pkg/ccl/changefeedccl/alter_changefeed_stmt.go Outdated
Add basic syntax support for database-level
changefeeds.
CREATE DATABASE CHANGEFEED for foo
This PR only allows the aforementioned statement
to be parsed; no changefeed is created.

Epic: CRDB-1421
Resolves: #147369

Release note: None
@KeithCh

KeithCh commented Jul 2, 2025

Copy link
Copy Markdown
Author

bors r=asg0451

tftr!

@craig

craig Bot commented Jul 2, 2025

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DB-Level Changefeed: Support syntax for DB-level changefeeds with no filters

6 participants