Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions docs/client-connectivity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Client Connectivity

How to connect various Iceberg clients to `ice-rest-catalog`.

## DuckDB

Connect DuckDB to an Iceberg REST catalog using a Bearer token.

Official reference: [Iceberg REST Catalogs](https://duckdb.org/docs/current/core_extensions/iceberg/iceberg_rest_catalogs.html)

### 1. Install extensions

```sql
INSTALL iceberg;
INSTALL httpfs; -- needed for S3/GCS-backed tables
LOAD iceberg;
LOAD httpfs;
```

### 2. Store the Bearer token in a secret

If you already have a token (from your IdP, catalog UI, `curl`, etc.), put it in an Iceberg secret with `TOKEN`:

```sql
CREATE SECRET iceberg_secret (
TYPE iceberg,
TOKEN 'your_bearer_token_here'
);
```

- Use the raw token only — do **not** include the `Bearer ` prefix.
- DuckDB sends it as `Authorization: Bearer <token>` on REST catalog requests.

Optional: add extra headers if your catalog requires them (e.g. GCP billing project):

```sql
CREATE SECRET iceberg_secret (
TYPE iceberg,
TOKEN 'your_bearer_token_here',
EXTRA_HTTP_HEADERS MAP {
'x-goog-user-project': 'your_gcp_project_id'
}
);
```

### 3. Attach the REST catalog

```sql
ATTACH 'warehouse_name' AS my_catalog (
TYPE iceberg,
SECRET iceberg_secret,
ENDPOINT 'https://your-rest-catalog.example.com'
);
```

| Parameter | Meaning |
|--------------------|----------------------------------------------|
| `'warehouse_name'` | Catalog warehouse name (from your provider) |
| `ENDPOINT` | Base URL of the Iceberg REST catalog |
| `SECRET` | Name of the secret from step 2 |

### 4. Storage access

The Bearer token authenticates to the REST catalog, not necessarily to S3/GCS where data lives.

By default DuckDB uses vended credentials (`ACCESS_DELEGATION_MODE 'vended_credentials'`): the catalog returns temporary storage credentials when you load a table.

If your catalog does not vend credentials, configure storage separately:

```sql
-- Example: direct S3 access
CREATE SECRET s3_secret (
TYPE s3,
KEY_ID '...',
SECRET '...',
REGION 'us-east-1'
);
ATTACH 'warehouse' AS my_catalog (
TYPE iceberg,
SECRET iceberg_secret,
ENDPOINT 'https://catalog.example.com',
ACCESS_DELEGATION_MODE 'none'
);
```

Supported storage backends: S3, S3 Tables, GCS (see [limitations](https://duckdb.org/docs/current/core_extensions/iceberg/iceberg_rest_catalogs.html#limitations)).

### 5. Query tables

```sql
SHOW ALL TABLES;
SELECT * FROM my_catalog.default.my_table LIMIT 10;
```

Tables are referenced as `catalog.schema.table`.
1 change: 1 addition & 0 deletions ice-rest-catalog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,4 @@ If `enabled` is true but the catalog backend is not etcd, the lock is ignored (w
- [Catalog Import/Export](../docs/catalog-import-export.md) -- export and import catalog registry (namespaces and table metadata pointers) via CLI or REST API
- [Catalog migration (1-node to 3-node etcd)](../docs/catalog-export-import-migration.md) -- migrate registry via `ice catalog-export` / `catalog-import`; optional recovery from an etcd snapshot backup first
- [Rewriting table paths](../docs/rewrite-table-path.md) -- using Spark to rewrite paths in metadata files when copying a table to a new location
- [Client Connectivity](../docs/client-connectivity.md) -- connecting to ice-rest-catalog from DuckDB (and other clients)
Loading