Spline Initialization failing for Console and Logging

[nkarthikn]

Error:

ERROR QueryExecutionEventHandlerFactory: Spline initialization failed! Spark Lineage tracking is DISABLED.
java.util.NoSuchElementException: Missing configuration property spline.lineageDispatcher.http.producer.url

Command Used:

pyspark  --packages za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.7.1  \
         --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
         --conf "spline.lineageDispatcher=console" \
         --conf "spline.lineageDispatcher.console.className=za.co.absa.spline.harvester.dispatcher.ConsoleLineageDispatcher" \
         --conf "spline.lineageDispatcher.console.stream=OUT"

What am I missing?

[wajda]

Spark ignores config keys that don't start with spark. Hence, when setting Spline properties via Spark configuration all keys have to be prepended with the spark. prefix to be properly propagated to the Spline agent.

pyspark  --packages za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.7.1  \
         --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
         --conf "spark.spline.lineageDispatcher=console" \
         --conf "spark.spline.lineageDispatcher.console.className=za.co.absa.spline.harvester.dispatcher.ConsoleLineageDispatcher" \
         --conf "spark.spline.lineageDispatcher.console.stream=OUT"

[wajda]

by the way, these lines are redundant

--conf "spark.spline.lineageDispatcher.console.className=za.co.absa.spline.harvester.dispatcher.ConsoleLineageDispatcher" \
--conf "spark.spline.lineageDispatcher.console.stream=OUT

The console dispatcher is preconfigured in the default config.

[wajda]

The default config is merged with the user one, so there is no need to repeat default settings in your own config,

[nkarthikn]

thank you! it worked.