[feature](lance) support lance catalog in doris by zhangstar333 · Pull Request #65730 · apache/doris

zhangstar333 · 2026-07-16T14:27:21Z

What problem does this PR solve?

Issue #66340
Problem Summary:
doc: apache/doris-website#4024

This PR introduces read support for the Lance format in Doris, including:

Querying Lance datasets through a Lance Catalog.
Pushing supported predicates down to Lance to reduce the amount of data scanned.
Reading Lance datasets directly through S3 and Local TVFs.
Performing vector similarity search through the vector_search() TVF, with the distance exposed through the _distance column.
Lance is currently read-only and uses File Scanner V2. Writing to Lance is not supported yet.

  -- Create a Lance Catalog
  CREATE CATALOG lance_catalog PROPERTIES (
      "type" = "lance",
      "lance.catalog.type" = "filesystem",
      "warehouse" = "s3://warehouse/lance",
      "s3.endpoint" = "http://127.0.0.1:9000",
      "s3.access_key" = "admin",
      "s3.secret_key" = "password",
      "s3.region" = "us-east-1",
      "use_path_style" = "true"
  );

  -- Regular query; supported WHERE predicates are pushed down to Lance
  SELECT user_id, user_name
  FROM lance_catalog.doris.user_profiles
  WHERE engagement_score >= 90;

  -- Read a Lance dataset directly through the S3 TVF
  SELECT user_id, user_name
  FROM s3(
      "uri" = "s3://warehouse/lance/doris/user_profiles.lance",
      "format" = "lance",
      "s3.endpoint" = "http://127.0.0.1:9000",
      "s3.access_key" = "admin",
      "s3.secret_key" = "password",
      "s3.region" = "us-east-1",
      "use_path_style" = "true"
  );

  -- Perform a vector similarity search
  SELECT row_id, label, _distance
  FROM vector_search(
      "table" = "lance_catalog.doris.vector_search",
      "column" = "embedding",
      "query_vector" = "[0,0,0,0]",
      "top_k" = "5",
      "metric" = "l2"
  )
  ORDER BY _distance, row_id;

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-07-16T14:27:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhangstar333 · 2026-07-31T07:37:40Z

/review

Gabriel39

I found several security and correctness issues that should be addressed before merging.

github-actions · 2026-07-31T09:24:11Z

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/30613481239

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

github-actions

This adds a substantial Lance integration, but the current scan path still has blocking correctness, memory-safety, process-safety, security-display, snapshot-consistency, placement, and resource-governance defects. The inline comments identify thirteen distinct issues that remain after deduplicating against the four live comments already on the PR. Review was static-only as required; no build or test commands were run.

Gabriel39 · 2026-07-31T10:11:01Z

在这一版实现里，我们要把Postfilter改成Lance SDK 原生 Postfilter
另外，测试覆盖要把各种索引类型都加上

zhangstar333 · 2026-08-04T08:19:07Z

run buildall

zhangstar333 · 2026-08-04T11:33:21Z

run buildall

hello-stephen · 2026-08-04T13:07:30Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.62% (1904/2453)
Line Coverage	64.53% (34057/52774)
Region Coverage	64.49% (17216/26694)
Branch Coverage	54.03% (9220/17064)

zhangstar333 · 2026-08-04T15:45:46Z

run buildall

hello-stephen · 2026-08-04T16:52:21Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.62% (1904/2453)
Line Coverage	64.45% (34012/52774)
Region Coverage	64.46% (17206/26694)
Branch Coverage	53.96% (9207/17064)

hello-stephen · 2026-08-04T19:19:43Z

BE Regression && UT Coverage Report

Increment line coverage 69.09% (847/1226) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.67% (30364/41215)
Line Coverage	57.62% (333867/579401)
Region Coverage	54.49% (277771/509733)
Branch Coverage	55.46% (124428/224358)

hello-stephen · 2026-08-04T19:33:15Z

FE Regression Coverage Report

Increment line coverage 65.66% (1017/1549) 🎉
Increment coverage report
Complete coverage report

zhangstar333 · 2026-08-05T00:50:26Z

run buildall

…pache#66495) The vector_search() query path is algorithm-agnostic, but until now only flat search was actually exercised: the fixture from apache#65730 never created the vector index its tests claimed to cover (the referenced create_vector_search_index.py was never committed), so Lance silently fell back to flat KNN and the "IVF_PQ" regression proved nothing about indexes. This change adds verified IVF_FLAT query coverage and repairs the IVF_PQ fixture: - Add lance_build_preinstalled_catalog.py, an offline fixture generator pinned to pylance 4.0.1 (matching lance-c v0.1.2 / lance-rs 4.0.1). It builds the Directory V2 catalog through the namespace API (all_types re-registered, doris namespace, one indexed table per algorithm: vector_search_ivf_flat and vector_search_ivf_pq), then self-checks: Lance plans contain ANNSubIndex/ANNIvfPartition, indexes cover both fragments, IVF_FLAT full-probe equals flat search exactly, and the row-256 partition-boundary discriminator holds per table. The generated catalog (404K) is committed; the manifest is compacted back to one version/one data file. - Deterministic data: 1024 rows, two fragments, 16-dim Float32 with embedding[j] = (row_id - 1) + j, so a query equal to row r's vector has exact squared L2 distance 16 * (n - r)^2 to row n - integer ladder, tie-free at the endpoints, exact in Float32. - Rewrite test_lance_vector_search to target the real IVF_PQ table and add test_lance_vector_search_ivf_flat. Both suites carry a silent-fallback discriminator: querying the row-256 boundary with nprobes=1 must differ from the flat result (a pipeline that ignores the index returns exactly the flat rows and fails the assertion). IVF_FLAT additionally asserts indexed == flat programmatically (its algorithm guarantee); IVF_PQ freezes refine_factor-reranked results in goldens without claiming algorithm guarantees. - Remove run07_create_vector_types.sql and the stale golden output. Cross-version compatibility of the generated manifest was verified against lance-java 4.0.0 (lance-spark-bundle 0.4.0, runtime table registration) and lance-java 9.1.0-beta.3 (FE Directory Namespace client): both list, describe, register and deregister tables against it correctly. The remaining algorithms (IVF_SQ, IVF_HNSW_FLAT, IVF_HNSW_SQ, IVF_HNSW_PQ) follow the same pattern in follow-up PRs. Draft until the .out goldens are regenerated by a full docker regression run, which also provides the FE -> BE -> lance-c end-to-end evidence. Part of apache#66495. Claude-Session: https://claude.ai/code/session_014jKR13f5jdqs8NFyEuCnAN

### What problem does this PR solve? Issue Number: apache#66497 Related PR: apache#65730 Problem Summary: Lance external tables can contain logical vector and scalar indexes, but SHOW INDEX currently only handles internal OlapTable metadata and returns an empty result for Lance tables. Read authoritative logical index descriptions from one latest Directory Dataset snapshot, resolve field IDs from the same snapshot schema, and expose deterministic bounded metadata through the existing 13-column SHOW INDEX result without changing internal-table behavior. ### Release note Support SHOW INDEX for Lance filesystem/Directory Catalog tables. Lance REST index inspection remains unsupported in this phase. ### Check List (For Author) - Test: Unit Test and Regression test - Behavior changed: Yes. SHOW INDEX now returns logical indexes for Lance Directory tables and explicitly rejects Lance REST catalogs. - Does this need documentation: Yes. Document the supported Directory-only scope and REST limitation.

### What problem does this PR solve? Issue Number: N/A Related PR: #65730, #66581 Problem Summary: Ordinary Lance scans currently read **every row of a fragment** even when the query only needs the first N rows (e.g. `SELECT ... LIMIT 10`). Lance applies its own LIMIT *after* the scanner's filter, so the query LIMIT can be forwarded to each fragment scanner and let it stop early, cutting IO and decode cost. **How it is fixed** - `thrift`: add an optional `TLanceFileDesc.limit`. - `FE` (`LanceScanNode`): push the query limit into each fragment split via `canPushDownLimit()`, and surface `lanceLimit` in the explain output. - `BE` (`lance_reader`): forward it to the scanner through `lance_scanner_set_limit` for ordinary scans; vector search keeps its own `top_k` limit. **Correctness** The limit is pushed **only when all predicates are already pushed into Lance** (no residual Doris conjunct). Otherwise Doris still re-filters the returned rows, and truncating a fragment early could drop valid results. `OFFSET` needs no special handling: Nereids' `SplitLimit` rewrites `Limit(limit, offset)` into a global `Limit(limit, offset)` over a local `Limit(limit + offset, 0)`, and that local bound is what reaches the scan node. So `getLimit()` already includes the offset; each fragment fetches up to `limit + offset` rows and the upper global LIMIT still applies the offset and the final bound. Per-fragment truncation is therefore always safe. **Behavior change** Query results are unchanged. Only the number of rows scanned per fragment is reduced for LIMIT queries; the explain output shows an extra `lanceLimit=N` line when the limit is pushed. ### Release note Push down LIMIT into Lance fragment scanners to reduce the rows scanned for `LIMIT` / `LIMIT ... OFFSET` queries over Lance tables. ### Check List (For Author) - Test - [x] Unit Test (`LanceThriftContractTest` covers the limit round-trip and the no-limit case) - [ ] Manual test — `SELECT * FROM <lance_tbl> LIMIT 10` returns 10 rows and `EXPLAIN` shows `lanceLimit=10`; a query with a non-pushable predicate keeps the limit out of the scan - Behavior changed: - [x] No. - Does this need documentation? - [x] No.

### What problem does this PR solve? Issue Number: apache#66497 Related PR: apache#65730 Problem Summary: Lance external tables can contain logical vector and scalar indexes, but SHOW INDEX currently only handles internal OlapTable metadata and returns an empty result for Lance tables. Read authoritative logical index descriptions from one latest Directory Dataset snapshot, resolve field IDs from the same snapshot schema, and expose deterministic bounded metadata through the existing 13-column SHOW INDEX result without changing internal-table behavior. ### Release note Support SHOW INDEX for Lance filesystem/Directory Catalog tables. Lance REST index inspection remains unsupported in this phase. ### Check List (For Author) - Test: Unit Test and Regression test - Behavior changed: Yes. SHOW INDEX now returns logical indexes for Lance Directory tables and explicitly rejects Lance REST catalogs. - Does this need documentation: Yes. Document the supported Directory-only scope and REST limitation.

### What problem does this PR solve? Issue Number: #66497 Related PR: #65730 Lance external tables can contain logical vector and scalar indexes, but `SHOW INDEX` currently only handles internal `OlapTable` metadata and returns an empty result for Lance tables. This change reads authoritative logical index descriptions from one latest Lance Directory Dataset snapshot, resolves field IDs against the schema from that same snapshot, and exposes deterministic, bounded metadata through the existing 13-column `SHOW INDEX` result without changing internal-table behavior. ### What changed? - Add immutable Lance logical-index metadata and a loader backed by one latest Dataset snapshot. - Support `SHOW INDEX`, `SHOW INDEXES`, `SHOW KEY`, and `SHOW KEYS` for Lance filesystem/Directory Catalog tables. - Preserve existing internal-table behavior and permission-check ordering (privilege check runs before any catalog initialization). - Return a fixed unsupported error for Lance REST catalogs in this phase. - Sanitize Dataset URI, storage credential, and REST secret values from index metadata errors. - Defensively reject non-Lance tables resolved from a Lance catalog instead of failing on a raw cast. - Add unit and regression coverage for normalization, formatting, permissions, aliases, empty-index tables, and error handling. ### Design walkthrough (diagrams) `SHOW INDEX` for Lance Directory tables is a small read pipeline whose main concern is object ownership across the JNI boundary: SDK objects (`Dataset`, `IndexDescription`, Arrow allocators) never escape the loader, and the command layer only sees immutable `LanceLogicalIndex` values. Layered view — three new classes (`LanceLogicalIndex`, `LanceIndexMetadataLoader`, `LanceMetadataReadExecutor`) and three touch points (`ShowIndexCommand`, `LanceExternalTable`, `LanceExternalCatalog`): ```mermaid flowchart TB subgraph CMD["Command layer · Nereids"] A["ShowIndexCommand analyze() privilege check (:83) handleShowIndex() dispatch (:111) buildLanceRows() row mapping (:159)"] end subgraph EXT["External table layer"] B["LanceExternalTable loadIndexMetadata() (:69) — thin delegate"] end subgraph CAT["Catalog layer · caller thread"] C["LanceExternalCatalog isRestCatalogConfigured() (:172) early REST rejection resolveTableAccess() (:371) namespace resolution"] C2["sanitizedRootCauseMessage() (:473) URI / credential / token redaction"] end subgraph BND["Execution boundary · lance-metadata-read pool"] D["LanceMetadataReadExecutor (:50) 4 concurrent · 16 queued · deadline ≤ 60s"] E["LanceIndexMetadataLoader load() (:82) → normalize() (:206)"] end subgraph NAT["native · JNI"] F["Dataset.open(uri, latest) schema + describeIndices from one snapshot"] end G["LanceLogicalIndex immutable value object · pure-Java boundary"] A --> B --> C --> D --> E --> F E --> G C -.->|any failure| C2 C2 -.->|redacted exception only| A ``` Request timeline and resource ownership — namespace resolution stays on the caller thread (it owns the catalog's shared namespace/allocator), the JNI read runs on a bounded worker with a task-owned allocator, and a timed-out caller never cancels the in-flight native call: ```mermaid sequenceDiagram autonumber participant U as User participant C as Caller thread ShowIndexCommand participant K as Caller thread LanceExternalCatalog participant X as Executor boundary participant W as Worker thread lance-metadata-read participant J as Lance JNI Dataset U->>C: SHOW INDEX FROM lance_db.t C->>C: analyze() → checkTblPriv(SHOW) Note over C: privilege check runs before any catalog initialization C->>K: isRestCatalogConfigured() Note over K: reads normalized properties only, no namespace init — REST catalogs rejected here with a fixed error C->>K: resolveTableAccess() Note over K: shared namespace / allocator owned by the caller thread, so catalog close stays safe K-->>C: ResolvedTableAccess(uri, options) C->>X: execute(task) X->>W: submit (wrapped task checks deadline first) W->>W: expired in queue? fail before entering JNI W->>J: open(latest) + schema + describeIndices Note over W,J: task-owned RootAllocator (256MB) — catalog close cannot release it early J-->>W: IndexDescription + LanceSchema W-->>X: immutable index list X-->>C: future.get(remaining deadline) Note over C,X: timeout / interrupt never cancels the Future — the worker stays sole owner of native resources C->>C: buildLanceRows() → 13 columns C-->>U: ShowResultSet ``` Loader internals — one snapshot feeds both the schema mapping and the index descriptions; normalization is fail-closed and produces deterministic, bounded output: ```mermaid flowchart LR O["Dataset.open(uri, latest) index cache 0 · metadata cache 64MB schema and indexes from one snapshot"] O --> S["getLanceSchema().fields()"] O --> L["listIndexes() physical entries ≤ 16384"] subgraph SB["schema branch"] S --> DFS["buildFieldNamesById() (:148) DFS depth ≤ 64 · fields ≤ 16384 field id → dotted path, quoted when needed"] end subgraph IB["index branch"] L --> FL["drop system indexes __lance_frag_reuse · __lance_mem_wal"] FL --> DD["dedupe by logical name ≤ 256"] DD --> CR["describeUserIndexes() (:95) per-name describeIndices(hasName) must return exactly 1, name must match"] end DFS --> V CR --> V subgraph NB["normalize() (:206) · fail-closed"] V["known, unique field ids columns ≤ 64 · names ≤ 16KB total"] --> P["details JSON allowlist TreeMap-sorted · ≤ 400B"] P --> R["exact duplicate names rejected sorted by (name, position)"] end R --> OUT["LanceLogicalIndex list immutable · deterministic"] ``` ### Fail-closed semantics (intentional) Three strict behaviors are deliberate and locked by unit tests — the command reports an error rather than showing potentially misleading metadata: - An index referencing an unknown or nested field ID fails the whole `SHOW INDEX` (`testRejectsUnknownNestedDuplicateNullAndEmptyFieldIds`). - One malformed index entry (invalid details JSON, oversize fields, duplicate name) fails the whole result instead of silently dropping rows (`testRejectsMalformedAndNonObjectJsonWithoutEchoingInput`). - Index names differing only by case are preserved as distinct entries; only exact duplicates are rejected (`testRejectsExactDuplicateNameButPreservesCaseOnlyNames`). ### User/developer impact Users can inspect logical Lance indexes through the standard 13-column `SHOW INDEX` interface for filesystem/Directory catalogs. Lance REST index inspection remains unsupported. ### Release note Support `SHOW INDEX` for Lance filesystem/Directory Catalog tables. ### Check List (For Author) - [x] Unit tests: 28 tests passed - [x] Full FE build: all 27 modules passed - [x] Lance regression test: generation and normal verification modes passed - [x] `git diff --check` - [x] Behavior changed: `SHOW INDEX` now returns logical indexes for Lance Directory tables and explicitly rejects Lance REST catalogs - [x] Documentation needed: document the supported Directory-only scope and REST limitation

Issue Number: close apache#66496 Related PR: apache#65730 Problem Summary: Lance schema discovery rejected Blob v2, Arrow JSON, BFloat16, Arrow null, and Arrow duration fields as unsupported. Add validated FE and BE mappings, materialize Blob payloads through lance-c, normalize BFloat16 values to FLOAT, preserve duration counts as BIGINT, and materialize Arrow null values safely. Support reading Lance Blob v2, Arrow JSON, BFloat16, null, and duration columns. - Test: Unit Test (added; not run locally at user request) - Behavior changed: Yes (the newly supported Lance columns are queryable) - Does this need documentation: Yes (type matrix is documented in the regression suite; user documentation follow-up is required)

## Summary - add Apache Doris to the Query Engines and SQL integrations table - link to the Apache Doris Lance Catalog documentation and source repository - describe the current read-only catalog, predicate pushdown, and vector search capabilities ## References - Documentation: https://doris.apache.org/docs/4.x/lakehouse/catalogs/lance-catalog/ - Implementation: apache/doris#65730 ## Verification - `make build` - `make check-links`

Issue Number: close apache#66496 Related PR: apache#65730 Problem Summary: Lance schema discovery rejected Blob v2, Arrow JSON, BFloat16, Arrow null, and Arrow duration fields as unsupported. Add validated FE and BE mappings, materialize Blob payloads through lance-c, normalize BFloat16 values to FLOAT, preserve duration counts as BIGINT, and materialize Arrow null values safely. Support reading Lance Blob v2, Arrow JSON, BFloat16, null, and duration columns. - Test: Unit Test (added; not run locally at user request) - Behavior changed: Yes (the newly supported Lance columns are queryable) - Does this need documentation: Yes (type matrix is documented in the regression suite; user documentation follow-up is required)

…ce managed versioning ### What problem does this PR solve? Issue Number: close apache#66491 Related PR: apache#65730 (baseline Lance read integration, which already carried `FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation) Problem Summary: Lance catalogs could select a dataset version on the FE, but nothing proved it end to end, the error messages leaked SDK internals (`NumberFormatException: For input string`, storage paths and Rust source locations), and tables whose versions are managed by a REST Namespace (`DescribeTable` returns `managed_versioning = true`) were rejected outright with "not supported by the current BE reader". This PR completes the four scope items of the issue: 1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF '<timestamp>'` are covered by regression suites on a filesystem catalog and a REST catalog, with a committed three-version fixture (`time_travel.lance`) and the existing `multi_frag.lance` (three appends, then a delete) so that reading an older version across a deletion file is exercised on the BE. The semantics follow the Iceberg and Paimon integrations: the selected version is the latest one whose commit time is not later than the timestamp, parsed in the session time zone with second or millisecond precision and compared with commit times at millisecond precision (the precision a namespace reports them in, so storage and namespace resolution agree); a timestamp earlier than the first version is an error. 2. **REST Namespace managed versioning** – the FE now opens such a table through the SDK's namespace client (`Dataset.open().namespaceClient(..).tableId(..)`), so the latest version comes from `ListTableVersions`, an explicit version from `DescribeTableVersion`, and `FOR TIME AS OF` from the commit times (`timestamp_millis`) the namespace reports. The SDK finalizes a still-staged manifest to its canonical `_versions/` path while opening, so the BE keeps opening the dataset by URI and version number through lance-c exactly as before; no BE or thrift change is needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables. 3. **Selected version fixed for the whole statement** – schema binding, fragment planning, predicate pushdown, metadata COUNT and the BE scan all use the selected version; two references to the same table in one statement can select different versions (self join / union tests). 4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch and version it points at, read from the dataset's `_refs/tags/` for both storage-versioned and managed tables (Lance's namespace tag APIs expose the same refs; the selected version of a managed table is then resolved through the namespace); a tag created on a branch selects that branch's version, not the same number on `main`. As for Iceberg and Paimon, a non-numeric `FOR VERSION AS OF 'v2'` names a tag. `@branch(main)` is `main`; `@tag` / `@branch` take exactly one name and reject extra parameters. Inside a branch, versions and times select among the branch's own versions, which start at the version it was created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`, optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the BE receives the branch directory as the dataset URI and needs no change (a branch is a shallow clone with its own `_versions/`). For managed tables the SDK derives the branch from that directory and sends it in the `branch` field of the version APIs. Previously `@tag` / `@branch` were silently ignored and the query read the latest version of `main`. 5. **Errors** – a missing version (never committed, or removed by `cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')] was not found[ in the namespace]` without the storage path; a missing tag or branch, an out-of-range or non-positive version (a signed number is a version, not a tag name), a timestamp before the first version and an unparsable timestamp have their own messages; a namespace that lists no versions is reported as such; a table that is only declared in the namespace fails with "declared in the namespace but has no data yet"; for a managed table, a namespace `location` that is empty or differs from `table_uri` fails the query before anything is read; every other failure keeps the sanitized provider message. Design decisions worth calling out: - For a managed table the namespace is the source of truth. A version whose manifest is still in storage but which the namespace does not record is reported as not found, and the table's latest version is the namespace's latest even when storage already holds a newer manifest. Reading unrecorded manifests would expose staged or rolled-back commits. - `timestamp_millis` is optional in the Lance Namespace spec. If a namespace lists its versions without it, the only commit times available are the ones in the storage manifests, so `FOR TIME AS OF` uses those, restricted to the versions the namespace lists so that the selected version is always one the namespace records. A manifest records when the writer committed the data, which can be earlier than when the namespace registered the version, so the selection follows the writer's clock in that case. - The SDK opens and checks out a namespace-managed dataset with its own native namespace client (the JNI takes the native handle of a REST or Directory namespace instead of calling back into the Java object), so those operations are not serialized on the FE's namespace lock; only the FE's own namespace requests are. - `FOR VERSION AS OF` on a dataset whose directory is missing altogether reports "version N was not found", because the SDK probes the version's manifest path first; the provider's own message is kept in the exception cause, handled the same way as every other Lance error. - `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its `lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace client, whose `branch` field on the version requests is what managed-table branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled unchanged and every Lance unit test class and regression suite passes on it. The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and never writes a dataset, so the newer FE release does not change what the BE can read. - An expired version (removed by cleanup) is indistinguishable from one that never existed, on storage and in a namespace, so both get the same message. "in the namespace" is only added when the namespace itself reported the miss. - Every selector other than an explicit version on `main` is a checkout from one open of the latest `main`, and a managed `FOR TIME AS OF` whose namespace reports commit times is resolved before anything is opened. The namespace version list is fetched completely, once per read, without a page size: Lance's Directory namespace applies a limit without returning a page token, and neither the order nor monotonic commit times can be relied on to stop early. - Reading a managed table whose manifest is still staged makes the SDK copy it to its canonical path, so read credentials must allow that write; an access-denied failure on a managed table says so. - The BE reads by URI and version number, which assumes the manifest a namespace records ends up at its canonical path `_versions/<u64::MAX - v>.manifest`. A staged manifest from a normal commit is moved there on read; a namespace that keeps finalized manifests elsewhere cannot be detected by the FE, and the BE then either fails to find the version or reads a different manifest with the same number at the canonical path. The documentation lists this as a limitation. - Known limitations: two references to the same branch in one statement resolve the branch head independently, and the optimizer's row count is that of the latest `main` (both shared with the Iceberg integration). `vector_search()`, `full_text_search()` and index inspection use the latest `main` and reject a version, tag or branch in their `table` argument. Index paths open by URI and are only reachable for filesystem catalogs, whose Directory namespace never manages versions; an assertion guards that. - A Lance branch is a shallow clone whose manifests record the parent's location as an absolute URI (`Manifest.base_paths`), so a branch is readable only where it was created. The committed `time_travel.lance` branch was therefore created against the fixture's final location `s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py --create-branch`) and synced back; the fixture's self-check verifies that reference. Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions 1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3 / 4..6 / 7..9): ```sql mysql> SET enable_file_scanner_v2 = true; mysql> SET time_zone = 'UTC'; mysql> CREATE CATALOG lance_demo_fs PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "filesystem", -> "warehouse" = "s3://warehouse/lance", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.access_key" = "admin", -> "s3.secret_key" = "password", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3 -> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a -> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON a.row_id = b.row_id -> ORDER BY a.row_id; +--------+--------+--------+ | row_id | tag_v1 | tag_v3 | +--------+--------+--------+ | 1 | v1 | v1 | | 2 | v1 | v1 | | 3 | v1 | v1 | +--------+--------+--------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 99; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:07'; ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to load Lance table metadata for default.time_travel: IllegalArgumentException: Lance dataset has no version at or before '2026-09-19 13:06:07' ``` A REST Namespace with managed versioning. `time_travel_managed` records versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage already holds version 3, and `time_travel_managed_partial` records 1 and 3: ```sql mysql> CREATE CATALOG lance_demo_rest PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "rest", -> "lance.rest.uri" = "http://127.0.0.1:19102", -> "lance.rest.security.type" = "bearer", -> "lance.rest.bearer-token" = "doris-lance-rest-test-token", -> "lance.namespace.root_database" = "default", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_lagging; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of default.time_travel_managed_lagging was not found in the namespace mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> EXPLAIN SELECT row_id FROM lance_demo_rest.`default`.time_travel_managed_lagging; ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_rest.default.time_travel_managed_lagging lanceCatalogType=rest lanceVersion=2 lanceManagedVersioning=true lanceFragments=2 lanceFragmentGrouping=FRAGMENT ``` Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch `dev` forked from version 2 with one extra row; on `time_travel_managed_partial` the tag `v2` points at the version the namespace no longer records): ```sql mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | +--------+------+ mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | | 100 | dev | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@tag(no_such_tag); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of default.time_travel was not found mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@branch(dev); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 7 | 100 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@tag(v1); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial@tag(v2); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of default.time_travel_managed_partial (tag 'v2') was not found in the namespace mysql> EXPLAIN SELECT row_id FROM lance_demo_fs.`default`.time_travel@branch(dev); ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_fs.default.time_travel lanceCatalogType=filesystem lanceVersion=3 lanceManagedVersioning=false lanceBranch=dev lanceFragments=3 lanceFragmentGrouping=FRAGMENT ``` Test environment changes: the Lance REST fixture (`lance_rest_server.py`) gains `ListTableVersions` / `DescribeTableVersion` for tables listed in `LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed three-version dataset with tags and a branch, carried over as-is by `lance_build_preinstalled_catalog.py` because the suites hard-code its commit times (`lance_build_time_travel.py` regenerates it and prints the times to update; `--create-branch` forks the branch at the uploaded location). The stub additionally serves the `branch` parameter of the version APIs. ### Release note Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)` and `@branch(name)`, including tables whose versions are managed by a REST Namespace (`managed_versioning = true`). Missing versions, tags and branches and invalid selectors now report clear errors. ### Check List (For Author) - Test: Regression test / Unit Test - `external_table_p0/lance/test_lance_time_travel`, `test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture + MinIO) - `LanceManagedVersioningTest` (real SDK dataset behind a REST stub: managed / partial / lagging / staged manifest / untimed / growing / server error / location checks / declared-only / cleaned-up version / tags / branches / paging / empty version list), plus `test_lance_vector_search` and `test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest`, `LanceTableAccessCacheTest`, `LanceRestCatalogTest`, `LanceScanNodeTest`, `LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest` - Behavior changed: Yes. Tables with `managed_versioning = true` were rejected and are now readable; error messages for invalid or missing versions changed to the user-facing form above; `@tag` / `@branch` on a Lance table now select the tag or branch (including a tag that points into a branch) instead of silently reading the latest version of `main`; FE dependency `lance-core` 11.0.0 → 12.0.0. - Does this need documentation: Yes (doris-website PR to follow: feature table, REST catalog caution, new "Time Travel" section with the managed-versioning limitations)

zhangstar333 requested a review from yiguolei as a code owner July 16, 2026 14:27

zhangstar333 force-pushed the branch-4.1-lance-catalog branch 2 times, most recently from ce3b540 to f760bd0 Compare July 30, 2026 12:25

zhangstar333 changed the title ~~Branch 4.1 test~~ [feature](lance) support lance catalog in doris Jul 30, 2026

zhangstar333 force-pushed the branch-4.1-lance-catalog branch from 4dd1bee to eb379a8 Compare July 31, 2026 07:17

morningman mentioned this pull request Jul 31, 2026

[Tracking] Support Lance Catalog #66322

Closed

22 tasks

Gabriel39 reviewed Jul 31, 2026

View reviewed changes

github-actions Bot previously requested changes Jul 31, 2026

View reviewed changes

Gabriel39 mentioned this pull request Jul 31, 2026

[Tracking] Lance feature matrix for Doris 4.2 #66340

Open

85 tasks

zhangstar333 force-pushed the branch-4.1-lance-catalog branch from eb379a8 to 56d7904 Compare August 3, 2026 15:08

Gabriel39 added the dev/5.0.x label Aug 4, 2026

zhangstar333 force-pushed the branch-4.1-lance-catalog branch from c9a4827 to 70516cf Compare August 4, 2026 11:32

zhangstar333 added 6 commits August 5, 2026 08:55

thrift

b43b4f3

be code about lance reader

d724df4

be ut test

47cf8a9

docker lance

38ab475

update format v2

8a919a3

update

19da132

FANNG1 mentioned this pull request Aug 6, 2026

[test](lance) Add query coverage for the remaining ANN index types and vector element types #66512

Merged

Gabriel39 mentioned this pull request Aug 6, 2026

[Feature](lance) Add namespace and table DDL support #66516

Open

14 tasks

Jay-ju mentioned this pull request Aug 10, 2026

[opt](lance) push down LIMIT into Lance fragment scanners #66608

Merged

4 tasks

u70b3 mentioned this pull request Aug 11, 2026

[feature](lance) Support SHOW INDEX for Lance Directory tables #66637

Merged

6 tasks

FANNG1 mentioned this pull request Aug 14, 2026

[Bug](lance) REST catalog against Gravitino: namespace-vended credentials are dropped, and the root database is not listable #66772

Open

3 tasks

Jay-ju mentioned this pull request Aug 23, 2026

[feature](lance) push COUNT(*) down to Lance dataset metadata #66999

Merged

Gabriel39 mentioned this pull request Sep 4, 2026

docs: add Apache Doris integration lance-format/lance#8983

Merged

zy-kkk mentioned this pull request Sep 23, 2026

[feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453

Open

Conversation

zhangstar333 commented Jul 16, 2026 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 16, 2026

Uh oh!

zhangstar333 commented Jul 31, 2026

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gabriel39 commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Cloud UT Coverage Report

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

FE Regression Coverage Report

Uh oh!

zhangstar333 commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhangstar333 commented Jul 16, 2026 •

edited by morningman

Loading

Gabriel39 commented Jul 31, 2026 •

edited

Loading