Support distributed execution for OPTIMIZE INDEX

[majin1102]

Summary

Distribute the physical execution of ALTER TABLE ... OPTIMIZE INDEX without changing its SQL semantics or planning policies.

The driver plans index maintenance against one dataset snapshot. Spark executors build or merge uncommitted physical index segments, and the driver validates and publishes all accepted outputs in one atomic Lance commit.

Execution model

The initial implementation supports two work-unit types:

BuildCoverage {
  work_id
  fragment_ids
  index_configuration
}

MergeSegments {
  work_id
  input_segments
  expected_coverage
}

BuildCoverage creates index coverage for uncovered fragments through fragment-scoped createIndex. MergeSegments calls mergeExistingIndexSegments for an exact planner-selected group.

Each work unit maps to one Spark partition and one executor task. Fragment and merge groups come from the existing layout parameters in #744. Spark controls scheduling and concurrency through existing job and cluster configuration.

Commit and fault tolerance

Executors never update the manifest. Each attempt writes under a unique UUID and returns only uncommitted Index metadata.

The driver accepts at most one result per work_id, validates coverage and index metadata, and atomically publishes all outputs with commitExistingIndexSegments. Unselected segments remain unchanged. Failed or duplicate artifacts remain invisible and can be reclaimed by Lance cleanup.

Newly built coverage is not merged again in the same invocation. It may participate in size-tiered planning during a later OPTIMIZE INDEX.

Initial scope

The first version distributes BuildCoverage and MergeSegments for:

• BTree
• ZoneMap
• FTS / Inverted

Unsupported operations continue through the existing driver-side Dataset.optimizeIndices() path.

Vector maintenance, IVF rebalance, vector retraining, and intra-segment parallel build are follow-up work.

Dependencies

• Builds on Support OPTIMIZE INDEX SQL #743.
• Executes exact merge groups planned by Support size-tiered scalar index optimization #744.
• Uses existing Lance 9.0 fragment-scoped build, segment merge, and atomic commit APIs.
• Uses per-segment size metadata from feat(java): expose index segment sizes lance#8355.

Acceptance criteria

• Multiple independent work units execute as parallel Spark tasks.
• Executors produce uncommitted segments without updating the manifest.
• The driver atomically commits all accepted outputs.
• Retries and duplicate attempts do not affect visible index state.
• Artifact bytes do not pass through the Spark driver.
• Tests cover distributed coverage build, exact segment merge, retry, commit conflict, and preservation of unselected segments.