[Tracking] Complete the Arrow Flight SQL server and ship an official Doris ADBC driver

[morningman]

This issue tracks two halves of one goal: bringing Doris's Arrow Flight SQL server up to the
spec's feature set, and maintaining an official doris driver in the ADBC Driver Foundry, so
that a user runs dbc install doris and connects with a single URI.

Part of #65615. Part A depends on #67577 (one protocol-agnostic session and execution layer) for its
session and prepared-statement items; Part B can start before that and ship a pre-release built on
the subset that works today.

Why

ADBC is becoming for columnar analytics what JDBC is for row-oriented access. The ADBC Driver
Foundry already hosts drivers for ClickHouse, Databricks, Snowflake, Trino, Spark, SingleStore,
Presto, Redshift, MSSQL and BigQuery, all implementing ADBC 1.1 and callable from Go, Python, R and
Rust. Doris is not on that list, so when someone picks a data source in pandas, polars, Ibis or R,
Doris is not among the options.

Doris users are not blocked today - they use the generic adbc_driver_flightsql and then deal with
the gaps themselves. Doris's own FAQ lists twelve of them: BE unreachable from the client unless
public_host or an Nginx reverse proxy is configured, tokens evicted when a client never calls
close(), DATETIME arriving as an integer on the Java side, nested ARRAY failing in JDBC clients,
gRPC message-size limits, parameter ordinal 1 out of range on prepared-statement binding. A driver
is supposed to absorb that.

At the same time the server implements only a small part of the spec. Session actions, parallel
endpoints, native metadata commands, bulk ingestion, cancellation and renewal are all missing, so
users emulate them with SQL strings - SET exec_mem_limit=2000 in place of a session option.

Design principles. One ADBC connection is one Doris session is one Flight SQL connection: no
second channel and no MySQL-protocol fallback, because a fallback path hides missing capabilities
instead of reporting them - a gap should surface as ADBC_STATUS_NOT_IMPLEMENTED and appear in the
capability matrix. Only the spec, no private RPCs: Doris-specific semantics ride on session-option
names and Arrow extension type names, which are the extension points the spec already provides. The
control plane stays in the FE and the data plane in the BE, but both belong to one session - session
state only affects planning, and after planning the FE issues tickets and the BE streams, so a
distributed data plane does not break the session's singularity.

Status convention

Same as #65615: [x] means merged or confirmed complete; [ ] means open or needs follow-up.

Part A - server-side completeness

• A1. Session lifecycle through the standard actions. SetSessionOptions /
  GetSessionOptions / CloseSession mapped onto Doris session variables (SET, SHOW VARIABLES),
  with adbc.connection.current_catalog and current_db_schema as session options (SWITCH,
  USE). CloseSession also invalidates the session's authentication context, as the spec requires.
  The session object is the one from [Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends #67577, held by the Flight adapter, which removes the separate
  token cache and the invalid bearer token failure mode with it. Covers [fix](doris catalog)Support session passthrough for Doris catalog #59742 and the client-close
  half of [Bug] Arrow flight SQL process hanging after closing connection #36331.
  [feature](arrow-flight) Serve the Flight SQL session actions on the Doris session #67966 (opened 2026-09-14, merged 2026-09-17): SetSessionOptions / GetSessionOptions / CloseSession served
  on the Doris session. Two corrections from doing it: the adbc.h constants are
  adbc.connection.catalog / adbc.connection.db_schema (not current_*), and the ADBC Flight SQL
  driver never sends them - it translates them into the Flight session options catalog / schema
  (the JDBC driver sends catalog for its connection property), while every other name is passed
  through as a session variable. Removing the token cache is Stage 3 of [Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends #67577, not this item.
• A2. Parallel result return. GetFlightInfo returns N FlightEndpoints, one per BE result
  sink, each with its own ticket, fetched concurrently. Three shapes: scan-only queries grouped by
  tablet (row bandwidth equals cluster bandwidth); aggregation or join at the top, one sink per
  fragment instance (what enable_parallel_result_sink already does - this proposes making it the
  default rather than an opt-in); a global sort at the top, still multiple endpoints but with
  FlightInfo.ordered = true and endpoint order equal to result order, so clients fetch in parallel
  and concatenate. Tickets must be self-contained and verifiable (query id, fragment instance id,
  expiry, signature) so the BE never has to call back to the FE, which would make the FE the new
  bottleneck. BE addressing uses the spec's arrow-flight-reuse-connection://? location, which is a
  cleaner and standard answer to the reachability problem behind [Bug] When arrow-flight is used and the FE private name name is not resolvable, the client cannot access the FE nodes #62538, [Bug] BE segfault in Arrow Flight SQL DoGet when using public_host (ILB proxy) #62217, [Bug] Connection refused: no further information: /127.0.0.1:8050 JDK1.8 connect error #59490 and [Feature] Improving arrow flight sql with Kubernetes setup #44599.
  Related: [Bug] Arrow Flight SQL: short-circuit point query returns no Flight endpoint (no FlightSqlEndpointsLocations) #67368, where the short-circuit point-query path registers no Flight endpoint at all.
• A3. Native metadata commands. CommandGetCatalogs, GetDbSchemas, GetTables,
  GetTableTypes, GetXdbcTypeInfo, GetPrimaryKeys, GetImportedKeys, GetExportedKeys,
  GetSqlInfo served from catalog metadata directly as Arrow, instead of being rewritten into
  information_schema SQL. Doris's multi-catalog model (internal plus Hive, Iceberg, JDBC, Paimon)
  maps exactly onto ADBC's catalog / db_schema / table hierarchy, so one ADBC connection gives a
  Python client a federated view of the whole lakehouse - something the Snowflake and ClickHouse
  drivers cannot offer.
• A4. Bulk ingestion through CommandStatementIngest. ADBC's adbc.ingest.target_table plus
  BindStream plus ExecuteUpdate is DoPut carrying CommandStatementIngest, with
  DoPutUpdateResult returned in PutResult.app_metadata. Two pieces on the Doris side: an Arrow IPC
  input format for Stream Load, and routing CommandStatementIngest to it, so conn.adbc_ingest()
  lands Arrow memory without a CSV or JSON round trip. The FE should redirect the ingest DoPut to
  the BEs the way Stream Load already redirects, and allow parallel writes, otherwise ingestion
  becomes the new bottleneck. Ingest modes map to CREATE TABLE + load, Stream Load,
  INSERT OVERWRITE, and create-then-append. Covers the ingestion request in [Feature] Doris support Arrow Flight SQL protocol #25514.
• A5. Prepared statements and parameter binding. ActionCreatePreparedStatementRequest
  returns an opaque handle plus, optionally, result and parameter schemas; per the spec the result
  schema may depend on the bound parameters, so it is better to return nothing than something wrong.
  Doris reuses the prepared-statement object extracted in [Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends #67577 and receives Arrow parameter batches
  over DoPut. ExecuteSchema needs plan-without-execute, which the FE planner can already do.
  Covers apache/arrow-java#1009; the
  parameter ordinal 1 out of range error itself is the upstream bug
  apache/arrow#40118 and needs tracking there.
• A6. Cancellation, renewal and polling. AdbcStatementCancel onto CancelFlightInfo with
  the query id taken from FlightInfo.app_metadata, cancelling the associated poll descriptors and
  any issued-but-unconsumed tickets; RenewFlightEndpoint to extend the life of a large result;
  PollFlightInfo for asynchronous submission and progress of long queries.
• A7. Authentication and compression. Bearer token from Handshake as the session credential,
  mTLS, and the existing LDAP and Kerberos plugins; Arrow IPC body compression (LZ4, ZSTD) negotiated
  as a connection option, which matters most across data centers.

Part B - the official driver

• B1. Repository and stack. An adbc-drivers/doris repository under the Foundry (open an
  issue in adbc-drivers/onboarding from the template; Foundry admins create the repo with CI,
  workflows and validation templates, then the Doris community members are invited). Written in Go on
  driverbase-go, built as a shared library exporting AdbcDriverDorisInit.
• B2. Layering. driver/ (ADBC surface), flight/ (handshake, token renewal, endpoint
  routing and parallel scheduling, retries), types/ (type mapping and Arrow extension registration),
  catalog/ (metadata commands to GetObjects), ingest/. The type mapping in types/ shares one
  source of truth with the server via code generation, so the two cannot drift.
• B3. ADBC 1.1 surface mapped onto Flight SQL calls, including ExecutePartitions returning
  serialized FlightEndpoints as partitions and ReadPartition opening one DoGet each, with
  default parallelism min(len(endpoints), GOMAXPROCS).
• B4. Driver options and URI. doris.read.parallelism, doris.read.compression,
  doris.grpc.max_message_size, doris.ingest.format, doris.catalog.include_external, and a single
  URI form doris://user:pass@fe_host:port/catalog.db?param=value - no more grpc:// in Python
  versus jdbc:arrow-flight-sql:// in Java. The initial catalog/schema in the URI covers
  apache/arrow-java#829.

Testing

• Baseline first, before any code. Install the Foundry mysql driver, point it at Doris's
  MySQL port, run the Foundry validation suite and record what fails. That list is both the
  requirements document and good community-issue material. (The Foundry mysql driver already
  documents a non-MySQL backend, Databend, through its mysql.vendor option, so this is a supported
  way to measure.)
• A DriverQuirks implementation declaring Doris's capabilities, a docs/doris.md skeleton
  filled in from validation results, and a compose.yaml so validation runs against a real cluster in
  CI on every PR and release.
• Parallel-read correctness in the Doris repo: merged multi-endpoint results equal the
  single-endpoint result, ordered = true really is ordered, and partial endpoint failure propagates
  the error as expected. The cross-protocol consistency suite from [Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends #67577 is reused as is.
• Throughput and memory baselines on a ClickBench or TPC-H result-fetch, comparing jdbc:mysql,
  the generic flightsql driver and the official driver.

Release and promotion

The Foundry's bar for a first release is Linux, macOS and Windows, the three minimum capabilities
(query, bulk ingest, GetObjects), and a passing standard validation run; anything short of that
ships as a pre-release such as v0.1.0-alpha.1.

Suggested sequence: Part B ships an alpha on the subset that works today, the validation report then
drives Part A's priorities, and each finished Part A item removes one NOT_IMPLEMENTED from the
driver. When the minimum set is complete, tag v0.1.0 and ask the Foundry admins for CDN
distribution.

Arrow Flight SQL is still marked experimental in Doris, which is the right window for the breaking
changes in A1 (session and token lifetime) and A2 (parallel return changes result arrival order for
unordered queries). Both go in the release notes. Proposed condition for promoting Flight SQL out of
experimental: A1 through A6 complete and the Foundry validation minimum set green.

The generic adbc_driver_flightsql keeps working and is not deprecated; the official driver is the
recommended path, not the only one.

Rejected alternatives

• Rename the generic flightsql driver and publish it under the Foundry. Fastest way onto the
  driver list, but it solves none of the user-visible problems - the connection string, BE
  reachability and type conversion all stay with the user - and replacing it later costs more than
  doing it properly once.
• Wrap go-sql-driver/mysql in sqlwrapper and speak the MySQL protocol. Zero kernel changes and
  compatible with every version, but the Arrow conversion happens on the client, so it unifies the
  API without gaining any transport speedup - which was the reason for doing this at all. Valuable as
  the measurement baseline above, not as the destination.
• Two channels on the client: control plane over MySQL, data plane over Flight SQL. The session is
  then not shared; see the same entry in [Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends #67577.
• Private Flight extension RPCs for Doris-specific semantics. Session options and Arrow extension
  types already cover it, and private RPCs would make the driver unusable by generic clients, which
  defeats the point of shipping a standard driver.

References

• Arrow Flight SQL spec: https://arrow.apache.org/docs/format/FlightSql.html
• Building a driver in the ADBC Driver Foundry: https://adbc-drivers.org/building-drivers/
• Existing Foundry drivers: https://adbc-drivers.org/drivers/
• driverbase-go: https://github.com/adbc-drivers/driverbase-go
• Foundry validation suite: https://github.com/adbc-drivers/validation
• Doris Arrow Flight SQL docs: https://doris.apache.org/docs/dev/connection-integration/arrow-flight-sql
• The original Arrow Flight SQL issue: [Feature] Doris support Arrow Flight SQL protocol #25514

[hutiefang76]

I would like to take a small, independently testable part of A3: implement GetTableTypes and make GetTables honor the complete table_types filter. On current master, getStreamTableTypes returns UNIMPLEMENTED, and the helper forwards only the first requested type to listTableStatus (which only specially handles VIEW). That can omit requested tables or include excluded types.

The proposed patch keeps the existing catalog/user privilege checks, uses the same BASE TABLE / SYSTEM VIEW / VIEW names emitted by GetTables, returns the type list sorted as Flight SQL specifies, and filters the authorized table results before optional schema collection. I am preparing focused FE regression tests, including multiple/unknown types and Arrow buffer cleanup. This would be a partial A3 contribution, with no session or ADBC-driver redesign. Please let me know if someone is already working on this slice or if it should be coordinated differently.

[morningman]

I would like to take a small, independently testable part of A3: implement GetTableTypes and make GetTables honor the complete table_types filter. On current master, getStreamTableTypes returns UNIMPLEMENTED, and the helper forwards only the first requested type to listTableStatus (which only specially handles VIEW). That can omit requested tables or include excluded types.

The proposed patch keeps the existing catalog/user privilege checks, uses the same BASE TABLE / SYSTEM VIEW / VIEW names emitted by GetTables, returns the type list sorted as Flight SQL specifies, and filters the authorized table results before optional schema collection. I am preparing focused FE regression tests, including multiple/unknown types and Arrow buffer cleanup. This would be a partial A3 contribution, with no session or ADBC-driver redesign. Please let me know if someone is already working on this slice or if it should be coordinated differently.

Thanks for your contribution. I will review your PR