Datastream file source supported?

[mighelone]

Hi,

I am testing Spline with a streaming job reading some json files.

from pyspark.sql import SparkSession
import pyspark.sql.functions as psf
import pyspark.sql.types as pst

if __name__ == "__main__":
    spark = SparkSession.builder.master("local[*]").appName("runExample").getOrCreate()
    spark.sparkContext.setLogLevel("WARN")
    df = (
        spark
        .readStream
        .schema("id_ long, ts timestamp, value double")
        .json("field")
    )
    query = (
        df
        .writeStream
        .option("checkpointLocation", "_checkpoint")
        .option("truncate", False)
        # .option("numRows", 50)
        .format("parquet")
        .option("path", "example")
        .start()
    )
    query.awaitTermination()

    spark.stop()

Spline runs using docker-compose, and it worked fine for the batch version of the script.

I run the script as follows:

spark-submit \
  --packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:0.7.1 \
  --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
  --conf "spark.spline.producer.url=http://localhost:8080/producer" run_example.py

and these are the configuration of the compose:

# component versions

ARANGO_DB_VERSION=3.8.1
SPLINE_CORE_VERSION=0.7.5
SPLINE_AGENT_VERSION=0.7.1
SPLINE_UI_VERSION=0.7.2

# exposed ports

ARANGO_DB_PORT=8530
SPLINE_REST_PORT=8080
SPLINE_UI_PORT=9090

Do I miss something? Is streaming from files supported?

[wajda]

Unfortunately streaming isn't supported yet, but it's on the roadmap. The priority on this feature was raised recently, so it should be implemented in one of future releases. #41