lance-format · fightBoxing · Sep 4, 2026
diff --git a/docs/src/.pages b/docs/src/.pages
@@ -0,0 +1,6 @@
+nav:
+  - Welcome: index.md
+  - Install: install.md
+  - Config: config.md
+  - Performance: performance.md
+  - Operations: operations
diff --git a/docs/src/config.md b/docs/src/config.md
@@ -0,0 +1,79 @@
+# Config
+
+Configuration options are grouped by area. The table connector uses `'connector' = 'lance'`; the
+catalog types are `'lance'` (directory/S3) and `'lance-namespace'` (dir/rest).
+
+## Table connector options (`connector = 'lance'`)
+
+### Common
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `path` | ✅ | — | Path to the Lance dataset |
+| `hadoop.*` | ❌ | — | Prefix for Hadoop-family filesystem config (e.g. `hadoop.tbdsfs.meta`); stripped and injected into the Hadoop `Configuration` used for path resolution |
+
+### Read (Source)
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `read.batch-size` | ❌ | 1024 | Read batch size |
+| `read.limit` | ❌ | — | Maximum rows to read (limit pushdown) |
+| `read.columns` | ❌ | — | Columns to read, comma separated |
+| `read.filter` | ❌ | — | SQL `WHERE`-style filter predicate |
+| `read.version` | ❌ | — | Time travel: read a specific dataset version |
+| `read.as-of-timestamp` | ❌ | — | Time travel: read as of an ISO-8601 timestamp (ignored when `read.version` is set) |
+
+### Write (Sink)
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `write.batch-size` | ❌ | 1024 | Write batch size |
+| `write.mode` | ❌ | append | `append` or `overwrite` |
+| `write.max-rows-per-file` | ❌ | 1000000 | Maximum rows per data file |
+
+### Vector index
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `index.type` | ❌ | IVF_PQ | `IVF_PQ`, `IVF_HNSW`, or `IVF_FLAT` |
+| `index.column` | ❌ | — | Vector column name to index |
+| `index.num-partitions` | ❌ | 256 | IVF partition count |
+| `index.num-sub-vectors` | ❌ | — | PQ sub-vector count (auto if unset) |
+| `index.num-bits` | ❌ | 8 | PQ quantization bits (1–16) |
+| `index.max-level` | ❌ | 7 | HNSW max level |
+| `index.m` | ❌ | 16 | HNSW connections per level |
+| `index.ef-construction` | ❌ | 100 | HNSW construction search width |
+
+### Vector search
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `vector.column` | ❌ | — | Vector search column name |
+| `vector.metric` | ❌ | L2 | `L2`, `Cosine`, or `Dot` |
+| `vector.nprobes` | ❌ | 20 | IVF search probe count |
+| `vector.ef` | ❌ | 100 | HNSW search width |
+| `vector.refine-factor` | ❌ | — | Refine factor for recall |
+
+## Catalog options (`type = 'lance'`)
+
+Directory or S3 warehouse.
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `warehouse` | ✅ | — | Warehouse path (local or `s3://…`) |
+| `default-database` | ❌ | default | Default database name |
+| `s3-access-key` | ❌ | — | S3 access key ID |
+| `s3-secret-key` | ❌ | — | S3 secret access key |
+| `s3-region` | ❌ | — | S3 region (e.g. `us-east-1`) |
+| `s3-endpoint` | ❌ | — | S3 endpoint (for S3-compatible storage like MinIO) |
+| `s3-virtual-hosted-style` | ❌ | true | Virtual-hosted-style URLs |
+| `s3-allow-http` | ❌ | false | Allow HTTP (default HTTPS only) |
+
+## Namespace catalog options (`type = 'lance-namespace'`)
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `impl` | ✅ | — | Namespace implementation: `dir` or `rest` |
+| `root` | ❌ | — | Root path for directory namespace |
+| `uri` | ❌ | — | URI for REST namespace |
+| `default-database` | ❌ | default | Default database name |
diff --git a/docs/src/index.md b/docs/src/index.md
@@ -0,0 +1,60 @@
+# Flink Lance Connector
+
+## Introduction
+
+The Apache Flink Connector for Lance allows Apache Flink to read and write datasets stored in the
+[Lance](https://lance.org/) columnar format — an open lakehouse format optimized for multimodal AI
+and vector search workloads.
+
+By using the Flink Connector for Lance, you can run Flink's stream/batch processing, SQL querying,
+and stateful pipelines directly on Lance datasets, including native vector search.
+
+## Features
+
+The connector is built on the Flink Table API (`DynamicTableSource` / `DynamicTableSink`) plus
+`CatalogFactory`. Specifically, you can use the Flink Connector for Lance to:
+
+* **Read & Write Lance Datasets**: append and overwrite datasets via Flink SQL or the DataStream API.
+* **Column, Filter, Limit & Aggregate Pushdown**: push projections, `WHERE` predicates, limits and
+  aggregations down to Lance for efficient scans.
+* **Vector Search**: KNN search over `ARRAY<FLOAT>` columns with `L2`, `Cosine`, and `Dot` metrics,
+  via the `LanceVectorSearchFunction` table function.
+* **Vector Index Building**: create `IVF_PQ`, `IVF_HNSW`, and `IVF_FLAT` indexes.
+* **Time Travel**: read a historical version via `read.version` or `read.as-of-timestamp`.
+* **Catalog Support**: a directory/S3 `lance` catalog and a `lance-namespace` catalog (dir / rest).
+
+## Quick Start
+
+Create a catalog and a table, then insert and query:
+
+```sql
+-- Create a directory-based catalog
+CREATE CATALOG lance_catalog WITH (
+    'type' = 'lance',
+    'warehouse' = '/path/to/warehouse',
+    'default-database' = 'default'
+);
+
+USE CATALOG lance_catalog;
+
+-- Create a Lance table
+CREATE TABLE vectors (
+    id BIGINT,
+    content STRING,
+    embedding ARRAY<FLOAT>
+) WITH (
+    'connector' = 'lance',
+    'path' = '/data/vectors',
+    'write.batch-size' = '1024'
+);
+
+-- Insert data
+INSERT INTO vectors VALUES
+    (1, 'Hello World', ARRAY[0.1, 0.2, 0.3, 0.4]);
+
+-- Query data
+SELECT * FROM vectors WHERE id > 0;
+```
+
+See [Install](install.md) for dependency setup and [Operations](operations/) for the full SQL
+surface.
diff --git a/docs/src/install.md b/docs/src/install.md
@@ -0,0 +1,49 @@
+# Install
+
+## Requirements
+
+* JDK 11 or higher
+* Maven 3.6+
+* Apache Flink 1.18 / 1.19 / 1.20
+
+The connector ships one artifact per supported Flink minor version:
+
+| Flink version | Artifact |
+|---|---|
+| 1.18 | `lance-flink-1.18` |
+| 1.19 | `lance-flink-1.19` |
+| 1.20 | `lance-flink-1.20` |
+
+## Dependencies
+
+The connector depends on `org.lance:lance-core` (7.0.0) and Apache Arrow (18.3.0). These are pulled
+in transitively; you only need to add the connector artifact for your Flink version.
+
+Lance's Java bindings ship a platform-specific JNI native library
+(`liblance_jni.so` / `liblance_jni.dylib`) inside the `lance-core` jar. Ensure you run on a
+supported platform (linux-x86-64, linux-aarch64, darwin-aarch64).
+
+## Maven
+
+```xml
+<dependency>
+    <groupId>org.apache.flink</groupId>
+    <artifactId>lance-flink-1.18</artifactId>
+    <version>0.1.0</version>
+</dependency>
+```
+
+## Build from source
+
+```bash
+mvn clean verify
+```
+
+The build produces a fat jar per module (e.g. `lance-flink-1.18/target/lance-flink-1.18-*.jar`).
+Add the jar to your Flink cluster or job classpath, then use the `lance` / `lance-namespace`
+catalog types in SQL.
+
+## Note on Arrow and Netty
+
+The Arrow allocator defaults are set at runtime; if you see classloader-related SPI issues in
+tests, set the system property `arrow.memory.allocator.type=Netty`.
diff --git a/docs/src/operations/.pages b/docs/src/operations/.pages
@@ -0,0 +1,4 @@
+nav:
+  - DDL: ddl
+  - DQL: dql
+  - DML: dml
diff --git a/docs/src/operations/ddl/.pages b/docs/src/operations/ddl/.pages
@@ -0,0 +1,3 @@
+nav:
+  - CREATE CATALOG: create-catalog.md
+  - CREATE TABLE: create-table.md
diff --git a/docs/src/operations/ddl/create-catalog.md b/docs/src/operations/ddl/create-catalog.md
@@ -0,0 +1,80 @@
+# CREATE CATALOG
+
+The Lance Flink connector ships two catalog types, registered via SPI:
+
+| `type` | Class | Description |
+|---|---|---|
+| `lance` | `LanceCatalogFactory` | Directory-based catalog over a warehouse path (local or S3) |
+| `lance-namespace` | `LanceNamespaceCatalogFactory` | Catalog backed by a Lance namespace (dir or REST) |
+
+## Directory catalog (`type = 'lance'`)
+
+```sql
+CREATE CATALOG lance_catalog WITH (
+    'type' = 'lance',
+    'warehouse' = '/path/to/warehouse',
+    'default-database' = 'default'
+);
+
+USE CATALOG lance_catalog;
+```
+
+### S3 warehouse
+
+```sql
+CREATE CATALOG lance_s3_catalog WITH (
+    'type' = 'lance',
+    'warehouse' = 's3://bucket-name/warehouse',
+    'default-database' = 'default',
+    's3-access-key' = 'your-access-key',
+    's3-secret-key' = 'your-secret-key',
+    's3-region' = 'us-east-1',
+    's3-endpoint' = 'https://s3.amazonaws.com'
+);
+```
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `warehouse` | ✅ | — | Warehouse path (local or `s3://`) |
+| `default-database` | ❌ | `default` | Default database |
+| `s3-access-key` | ❌ | — | S3 access key |
+| `s3-secret-key` | ❌ | — | S3 secret key |
+| `s3-region` | ❌ | — | S3 region |
+| `s3-endpoint` | ❌ | — | S3 endpoint (for MinIO etc.) |
+| `s3-virtual-hosted-style` | ❌ | `true` | Virtual-hosted-style URLs |
+| `s3-allow-http` | ❌ | `false` | Allow HTTP |
+
+## Namespace catalog (`type = 'lance-namespace'`)
+
+```sql
+-- Directory-based namespace
+CREATE CATALOG my_lance WITH (
+    'type' = 'lance-namespace',
+    'impl' = 'dir',
+    'root' = '/tmp/lance-warehouse'
+);
+
+-- REST-based namespace
+CREATE CATALOG my_lance WITH (
+    'type' = 'lance-namespace',
+    'impl' = 'rest',
+    'uri' = 'http://localhost:8080'
+);
+```
+
+| Option | Required | Default | Description |
+|---|---|---|---|
+| `impl` | ✅ | — | `dir` or `rest` |
+| `root` | ❌ | — | Root path for `dir` impl |
+| `uri` | ❌ | — | URI for `rest` impl |
+| `default-database` | ❌ | `default` | Default database |
+
+## Supported DDL
+
+| Statement | Status |
+|---|---|
+| `CREATE DATABASE` / `DROP DATABASE` / `ALTER DATABASE` | ✅ |
+| `SHOW DATABASES` / `SHOW TABLES` | ✅ |
+| `CREATE TABLE` / `DROP TABLE` / `RENAME TABLE` | ✅ |
+| `ALTER TABLE` | ❌ — not supported (structure immutable) |
+| `CREATE INDEX` | ❌ — no SQL DDL; configure `index.*` on the table |
diff --git a/docs/src/operations/ddl/create-table.md b/docs/src/operations/ddl/create-table.md
@@ -0,0 +1,56 @@
+# CREATE TABLE
+
+Lance tables are created through the dynamic table factory (`connector = 'lance'`).
+The actual Lance dataset is created lazily on first write.
+
+## Minimal example
+
+```sql
+CREATE TABLE vectors (
+    id BIGINT,
+    content STRING,
+    embedding ARRAY<FLOAT>
+) WITH (
+    'connector' = 'lance',
+    'path' = '/data/vectors'
+);
+```
+
+## With a vector index
+
+```sql
+CREATE TABLE doc_embeddings (
+    doc_id BIGINT,
+    title STRING,
+    embedding ARRAY<FLOAT>
+) WITH (
+    'connector' = 'lance',
+    'path' = '/data/embeddings',
+    'index.type' = 'IVF_PQ',
+    'index.column' = 'embedding',
+    'index.num-partitions' = '256',
+    'index.num-sub-vectors' = '16',
+    'vector.metric' = 'COSINE'
+);
+```
+
+## Required options
+
+| Option | Description |
+|---|---|
+| `path` | Path to the Lance dataset |
+
+## Type mapping
+
+| Lance / Arrow type | Flink type |
+|---|---|
+| Int8 / Int16 / Int32 / Int64 | TINYINT / SMALLINT / INT / BIGINT |
+| Float32 / Float64 | FLOAT / DOUBLE |
+| String | STRING |
+| Boolean | BOOLEAN |
+| Binary | BYTES |
+| Date32 | DATE |
+| Timestamp | TIMESTAMP |
+| FixedSizeList\<Float\> | ARRAY\<FLOAT\> |
+
+See [Config](../../config.md) for the full option reference.
diff --git a/docs/src/operations/dml/.pages b/docs/src/operations/dml/.pages
@@ -0,0 +1,2 @@
+nav:
+  - INSERT INTO: insert-into.md
diff --git a/docs/src/operations/dml/insert-into.md b/docs/src/operations/dml/insert-into.md
@@ -0,0 +1,39 @@
+# INSERT INTO
+
+The Lance Flink sink appends rows to a Lance dataset. Write mode is controlled by
+the `write.mode` option.
+
+## Write modes
+
+| `write.mode` | Behaviour |
+|---|---|
+| `append` (default) | Append rows to the existing dataset |
+| `overwrite` | Replace the dataset on first write |
+
+## Example
+
+```sql
+INSERT INTO vectors VALUES
+    (1, 'Hello World', ARRAY[0.1, 0.2, 0.3, 0.4]);
+```
+
+## Sink options
+
+| Option | Default | Description |
+|---|---|---|
+| `write.batch-size` | 1024 | Rows buffered before a flush |
+| `write.mode` | `append` | `append` or `overwrite` |
+| `write.max-rows-per-file` | 1000000 | Rows per data file |
+
+## Current limitations
+
+| Statement | Status |
+|---|---|
+| `INSERT INTO` (append) | ✅ |
+| `INSERT OVERWRITE` | ✅ |
+| `UPDATE` | ❌ — not implemented |
+| `DELETE` | ❌ — in progress (see issue #63 / #74) |
+| Primary key / upsert | ❌ — PK declaration and CDC changelog not yet supported |
+
+> The sink currently declares insert-only changelog mode. CDC `UPDATE` / `DELETE`
+> support is tracked in the connector roadmap.
diff --git a/docs/src/operations/dql/.pages b/docs/src/operations/dql/.pages
@@ -0,0 +1,4 @@
+nav:
+  - SELECT: select.md
+  - Vector Search: vector-search.md
+  - Time Travel: time-travel.md