Spline UI is not able to track Lineage when reading and writing to AWS S3 from EMR

[leadawsprofessional]

Spline UI is not able to track the data lineage when writing to AWS s3. Getting the following errors

Use case: I have an EMR from which I was running a spark job sample
(a) reading from S3 and writing S3 the spark is able to write but no spline data lineage got this bel,ow error
(b) reading from the local file system in EMR and writing local the spark is able to write but no spline data lineage got this same below error

Command used was

spark-submit \
  --packages za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.6.1 \
  --conf spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener \
  --conf spark.spline.producer.url=http://<IP OF THE Spline >:8080/producer \
  test.py

22/06/07 17:49:45 ERROR Utils: uncaught error in thread spark-listener-group-shared, stopping SparkContext
java.lang.NoSuchMethodError: org.json4s.CustomSerializer.<init>(Lscala/Function1;Lscala/reflect/Manifest;)V
        at org.json4s.ext.UUIDSerializer$.<init>(JavaTypesSerializers.scala:29)
        at org.json4s.ext.UUIDSerializer$.<clinit>(JavaTypesSerializers.scala)
        at org.json4s.ext.JavaTypesSerializers$.<init>(JavaTypesSerializers.scala:26)
        at org.json4s.ext.JavaTypesSerializers$.<clinit>(JavaTypesSerializers.scala)
        at za.co.absa.commons.json.format.JavaTypesSupport.formats(JavaTypesSupport.scala:25)
        at za.co.absa.commons.json.format.JavaTypesSupport.formats$(JavaTypesSupport.scala:23)
        at __wrapper$1$37e9b5c4c86f4fdbb9b86e375f3bb650.__wrapper$1$37e9b5c4c86f4fdbb9b86e375f3bb650$$anon$1.formats(<no source file>:2)
        at za.co.absa.commons.json.AbstractJsonSerDe$EntityToJson.<init>(AbstractJsonSerDe.scala:33)
        at za.co.absa.commons.json.AbstractJsonSerDe.EntityToJson(AbstractJsonSerDe.scala:32)
        at za.co.absa.commons.json.AbstractJsonSerDe.EntityToJson$(AbstractJsonSerDe.scala:32)
        at __wrapper$1$37e9b5c4c86f4fdbb9b86e375f3bb650.__wrapper$1$37e9b5c4c86f4fdbb9b86e375f3bb650$$anon$1.EntityToJson(<no source file>:2)
        at za.co.absa.spline.harvester.HashBasedUUIDGenerator.nextId(idGenerators.scala:57)
        at za.co.absa.spline.harvester.HashBasedUUIDGenerator.nextId(idGenerators.scala:55)
        at za.co.absa.spline.harvester.ComposableIdGenerator.$anonfun$nextId$2(idGenerators.scala:52)
        at scala.Function1.$anonfun$andThen$1(Function1.scala:57)
        at za.co.absa.spline.harvester.ComposableIdGenerator.nextId(idGenerators.scala:52)
        at za.co.absa.spline.harvester.converter.DataTypeConverter.convert(DataTypeConverter.scala:41)
        at za.co.absa.spline.harvester.LineageHarvesterFactory$$anon$1.za$co$absa$commons$lang$CachingConverter$$super$convert(LineageHarvesterFactory.scala:55)
        at za.co.absa.commons.lang.CachingConverter.$anonfun$convert$1(converters.scala:47)
        at scala.collection.mutable.MapLike.getOrElseUpdate(MapLike.scala:209)
        at scala.collection.mutable.MapLike.getOrElseUpdate$(MapLike.scala:206)
        at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:85)
        at za.co.absa.commons.lang.CachingConverter.convert(converters.scala:47)

[cerveada]

What version of Spark do you use?

[cerveada]

It looks similar to #423 that was already fixed.

Could you try the lates version of the agent (0.7.8)?

[wajda]

As @cerveada said, the issue was fixes in the Agent 0.7.8 (#423)