feat: incremental distributed creation for vector index

[summaryzb]

Dependencies

This proposal depends on:

1. lance-core: RFC: Expose IVF Centroids and PQ Codebook as Public Reader APIs lance#7319 — required for the R2 (truly distributed) implementation, which exposes the centroids / PQ codebook readers across Rust / Java / Python so executors can rebuild segments without retraining.
2. lance-spark: feat: distributed vector index creation #605 — the distributed CREATE INDEX baseline this work extends. The mode option introduced here lives on top of that PR's AddIndexExec / VectorIndexJob skeleton.

The R1 (driver-only) implementation in this PR is independent of #7319 and lands first; once #7319 ships, runIncremental(...) switches to the distributed path with no SQL surface change.

Problem

ALTER TABLE … CREATE INDEX … USING IVF_* currently always does a full rebuild: the driver re-trains IVF centroids (and the PQ codebook for PQ variants), every fragment is re-indexed in parallel by executors, and the resulting segments atomically replace the
previous index via commitExistingIndexSegments (see AddIndexExec.scala and VectorIndexJob.scala from #605).

Once a table has data appended after its index was built, the only way to make the index cover the new fragments is to re-run the same CREATE INDEX. Even when only a small number of fragments were added, this:

1. Re-trains centroids / PQ codebook (k-means runs up to max_iters iterations).
2. Rebuilds segments for every fragment, including ones that haven't changed.
3. Replaces all old segments (every Index.uuid rotates), which is wasteful for downstream consumers that cache or materialize per-segment artifacts.

SHOW INDEXES already exposes num_unindexed_fragments / num_unindexed_rows, so users can observe what's missing — but there's no SQL surface to act on it incrementally.

Motivation

• Cost. Re-training + full rebuild is paid on every ingest cycle. For typical production tables (tens of millions of vectors, training cost in the tens of minutes), repeated full rebuilds for small appends are clearly disproportionate.
• Consistency. Incremental builds must reuse the same centroids / codebook; otherwise queries see inconsistent semantics across segments of one logical index. lance-core's Dataset.optimizeIndices(OptimizeOptions.retrain=false) already provides this contract — it
  just isn't exposed via Spark SQL today.
• Segment stability. Incremental builds produce add-only segments (old segments untouched), which is friendlier to downstream consumers that key off Index.uuid.

Proposed change

Add a mode named argument to the WITH clause; values replace (default, preserves current behavior) or incremental:

ALTER TABLE lance.db.items CREATE INDEX vec_idx USING ivf_pq (embedding) WITH (
    mode = 'incremental'
);

Contract

• mode='incremental' requires the named index to already exist; otherwise an IllegalArgumentException("Index '<name>' does not exist; cannot run in incremental mode") is raised. We do not silently fall back to full build — that would defeat the purpose.
• All other WITH(...) arguments are ignored with a warning. Centroids / codebook / num_partitions / etc. are inherited from the existing index.
• When num_unindexed_fragments == 0, returns (0, indexName) and commits no transaction.
• The fragments_indexed output column means "fragments newly covered by this run" in incremental mode.

Implementation (R1, driver-only)

AddIndexExec.run() parses mode first and short-circuits to runIncremental(...), which calls Dataset.optimizeIndices(OptimizeOptions.builder().indexNames([name]).retrain(false).build()) on the driver. Build runs single-threaded on the driver in this stage — not
distributed yet — but the SQL surface, parameter whitelist, observability, and regression tests are all complete.

R2 (truly distributed incremental build) depends on lance-format/lance#7319 to expose centroids / codebook readers in Rust / Java / Python. Design docs:

Once #7319 lands, runIncremental(...) swaps to the distributed path; no SQL surface changes.

Scope (this PR)

• AddIndexExec.scala: add extractMode / runIncremental, plus AddIndexExec companion object (MODE_REPLACE / MODE_INCREMENTAL).
• VectorIndexParamsResolver.scala: include mode in CommonKeys whitelist so it doesn't fail validation as an unknown parameter.
• BaseIncrementalAddVectorIndexTest.java: 7 JUnit 5 cases covering happy path, no-op, missing-index rejection, empty table, IVF_PQ, explicit mode='replace', unrecognized mode.
• Versioned subclasses for lance-spark-3.4_2.12 and lance-spark-3.5_2.12.
• docs/src/operations/ddl/create-index.md: new "Incremental Mode" section + mode row in the options table.
• docs/design/: three design documents (incremental plan / lance-core API design / RFC draft) plus a pyo3 investigation report.

Out of scope / follow-ups

• R2 distributed incremental — gated on RFC: Expose IVF Centroids and PQ Codebook as Public Reader APIs lance#7319.
• Structural parameter changes (e.g. changing num_partitions) — not supported in incremental mode; users must use mode='replace'. Documented.
• Segment compaction — repeated incremental runs grow segment count over time. A follow-up OPTIMIZE INDEX … COMPACT subcommand should be considered as a separate PR.