Feature Request: export simple statement parser

[apstndb]

Is your feature request related to a problem? Please describe.

Based on my experience looking at Spanner-related OSS projects such as spanner-cli (and its my fork, spanner-mycli), wrench, yo, and golang-migrate, I've observed that many of them don't require a full-fledged SQL parser. Instead, they typically need a subset of functionalities, specifically:

• Statement splitting that correctly handles (or isn't confused by) semicolons within comments and string literals.
• Statement type identification (often, just the first token after skipping any hints is sufficient).
• Query parameter detection (though this is less commonly required).

Describe the solution you'd like

The implementation in go-sql-spanner/statement_parser.go provides an adequate solution for this use case. Although I understand there might still be potential for breaking changes, would it be possible to export this implementation once it becomes stable?

Describe alternatives you've considered

While cloudspannerecosystem/memefish, which includes a GoogleSQL lexer/parser, could be used to implement these, an additional (or separate) implementation would be necessary if a Spanner-supporting OSS project also needs to support the PostgreSQL interface in addition to the GoogleSQL interface.

Additional context
Add any other context or screenshots about the feature request here.

[apstndb]

I have noticed #513 is merged.

I believe that these functions supporting both dialects are useful.

• ParseParameters
• DetectStatementType

IMO, SplitStatements is natural extension of simple parser and it is very useful for usecases like batch schema/data loading from file.
(Sorry, I have wrongly closed this issue)

[apstndb]

Possible usecase

• wrench
  • completely replace thirdparty memefish lexer(and its helper functions) https://github.com/cloudspannerecosystem/wrench/blob/master/pkg/spanner/memefish.go
  • PostgreSQL support Support POSTGRESQL dialect cloudspannerecosystem/wrench#149
• golang-migrate
  • Replace spansql Proposal: Migrate parser library in Spanner support from spansql to memefish golang-migrate/migrate#1194
• spanner-cli and spanner-mycli
  • They still need their own client side statement(I believe it can be implemented by unexported parser.skipStatementHint() and parser.readKeyword()))

[olavloite]

@apstndb I'll look into exporting more of the various functions in StatementParser and SimpleParser. And a function like SplitStatements should indeed also be relatively straightforward to implement using these.

One additional thing that you could consider, in addition to using these parsers, is to base the various tools and frameworks on top of this database/sql driver, instead of the Spanner Go client library. That would still allow you to use your own client-side parser to pre-process statements, but would also give you automatic support for all new features that are added to this driver. We have already added support for PostgreSQL-dialect Spanner databases, and this support will be further improved in the coming months. This will include support for more PostgreSQL-dialect SQL statements, like for example SET TRANSACTION ... statements (https://www.postgresql.org/docs/current/sql-set-transaction.html).

[apstndb]

I'll look into exporting more of the various functions in StatementParser and SimpleParser. And a function like SplitStatements should indeed also be relatively straightforward to implement using these.

Thank you!

One additional thing that you could consider, in addition to using these parsers, is to base the various tools and frameworks on top of this database/sql driver, instead of the Spanner Go client library.

Exactly. I agree that the client-side statement handling in go-sql-spanner (and java-spanner), which includes not only parsing but also implementation, serves as a good de facto standard for broader tools in the Spanner ecosystem.
Currently, I haven't used go-sql-spanner through database/sql because I prefer to keep a thin abstraction layer, but most tools can leverage either go-sql-spanner or java-spanner (or java-spanner-jdbc).