description
Apache Spark Connector Documentation

Apache Spark Connector

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

Apache Spark as a connector for federated SQL query against a Spark Cluster using Spark Connect

datasets:
  - from: spark:spiceai.datasets.my_awesome_table
    name: my_table
    params:
      spark_remote: sc://my-spark-endpoint

Configuration

spark_remote: A spark remote connection URI. Refer to spark connect client connection string for parameters in URI.

Correlated scalar subqueries are only supported in filters, aggregations, projections, and UPDATE/MERGE/DELETE commands. Spark Docs
The Spark connector does not yet support streaming query results from Spark.