Delta 4.3.0 RC1

[openinx]

🎉 Delta Lake 4.3.0-RC1 is out for testing!

We're excited to share the first release candidate of Delta 4.3.0. This release lets any Delta engine create, read, write, and stream Unity Catalog managed Delta tables through a stable, versioned API.

Highlights:

📡 [Spark] Unity Catalog Delta REST API integration — Read, write, and create UC-managed Delta tables from any Delta engine through a single versioned API, with UC as the source of truth. On by default; disable with deltaRestApi.enabled=false.

🗺️ [Spark] GeoSpatial support — Create and query GeometryType / GeographyType columns end-to-end, including in generated columns and check constraints.

🧬 [Spark] Schema evolution in INSERT — Add the WITH SCHEMA EVOLUTION clause to a SQL INSERT to auto-evolve the target schema for just that statement — no session-wide config needed.

🔁 [Spark] Selective DataFrame replace — Use replaceUsing to replace rows that match on key columns, or replaceOn to replace rows matching a custom condition.

🧊 [UniForm] Atomic + incremental Iceberg conversion — Iceberg metadata is written atomically with each Delta commit and converted incrementally, now on both Spark 4.0 and 4.1.

📬 [Sharing] Streaming + CDF — Stream shared Delta-format tables, read Change Data Feed, and run backfill-then-stop pipelines with Trigger.AvailableNow.

How to try it

The RC artifacts are published to the Sonatype Central snapshots repository under version 4.3.0-rc1-SNAPSHOT (paired with Unity Catalog 0.5.0-rc1-SNAPSHOT). For example, to launch a SQL shell against a UC catalog with the Delta REST API enabled:

Spark 4.1

export CATALOG_NAME=main
export UC_URI=<your-unity-catalog-uri>
export UC_TOKEN=<your-unity-catalog-token>

bin/spark-sql --name "delta-uc-spark41" \
  --master "local[*]" \
  --packages "org.apache.hadoop:hadoop-aws:3.4.2,io.delta:delta-spark_4.1_2.13:4.3.0-rc1-SNAPSHOT,io.unitycatalog:unitycatalog-spark_4.1_2.13:0.5.0-rc1-SNAPSHOT" \
  --repositories "https://central.sonatype.com/repository/maven-snapshots/" \
  --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
  --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
  --conf "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \
  --conf "spark.sql.catalog.$CATALOG_NAME=io.unitycatalog.spark.UCSingleCatalog" \
  --conf "spark.sql.catalog.$CATALOG_NAME.uri=$UC_URI" \
  --conf "spark.sql.catalog.$CATALOG_NAME.token=$UC_TOKEN" \
  --conf "spark.sql.catalog.$CATALOG_NAME.deltaRestApi.enabled=true" \
  --conf "spark.sql.defaultCatalog=$CATALOG_NAME"

Spark 4.0

export CATALOG_NAME=main
export UC_URI=<your-unity-catalog-uri>
export UC_TOKEN=<your-unity-catalog-token>

bin/spark-sql --name "delta-uc-spark40" \
  --master "local[*]" \
  --packages "org.apache.hadoop:hadoop-aws:3.4.0,io.delta:delta-spark_4.0_2.13:4.3.0-rc1-SNAPSHOT,io.unitycatalog:unitycatalog-spark_4.0_2.13:0.5.0-rc1-SNAPSHOT" \
  --repositories "https://central.sonatype.com/repository/maven-snapshots/" \
  --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
  --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
  --conf "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \
  --conf "spark.sql.catalog.$CATALOG_NAME=io.unitycatalog.spark.UCSingleCatalog" \
  --conf "spark.sql.catalog.$CATALOG_NAME.uri=$UC_URI" \
  --conf "spark.sql.catalog.$CATALOG_NAME.token=$UC_TOKEN" \
  --conf "spark.sql.catalog.$CATALOG_NAME.deltaRestApi.enabled=true" \
  --conf "spark.sql.defaultCatalog=$CATALOG_NAME"

Please give it a try and report any issues — your feedback helps make the GA release solid. 🙏