Databricks with spline to kafka

[zacayd]

Hi Alex

I have a project to read lineage data of spline from spark databricks jobs to kafka
From what I saw on the https://absaoss.github.io/spline/
There Is an ability to connect the spine to kafka -
So I created a yml file to up the containers of spline_ui_, spline_spline-kafka, spline_rest , spline_arangodb
See attached yml

Her is the result of docker ps

CONTAINER ID   IMAGE                               COMMAND                   CREATED             STATUS             PORTS                                                 NAMES
311c5e8214b3   absaoss/spline-web-ui:0.7.4         "catalina.sh run"         About an hour ago   Up About an hour   8009/tcp, 0.0.0.0:9090->8080/tcp, :::9090->8080/tcp   spline_ui_1
f7e72dfe56ba   absaoss/spline-kafka-server:0.7.7   "catalina.sh run"         About an hour ago   Up About an hour   8009/tcp, 0.0.0.0:7070->8080/tcp, :::7070->8080/tcp   spline_spline-kafka_1
e414f7a99b36   absaoss/spline-rest-server:0.7.7    "bash -c '\n  until c…"   About an hour ago   Up About an hour   8009/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   spline_spline_1
f6183a32d788   arangodb/arangodb:3.9.4             "/entrypoint.sh aran…"    About an hour ago   Up About an hour   0.0.0.0:8529->8529/tcp, :::8529->8529/tcp             spline_arangodb_1

I also create 2 containers of zookeeper and broker of kafka (kafka-compose.yml)

On Databricks cluster I put on advanced settings:

spark.spline.mode REQUIRED
spark.spline.lineageDispatcher.kafka.topic spline-zacay
spark.spline.lineageDispatcher kafka
spark.spline.lineageDispatcher.kafka.producer.bootstrap.servers 192.168.100.11:9092

On notebook I put on the first rows:

sc._jvm.za.co.absa.spline.harvester.SparkLineageInitializer.enableLineageTracking(spark._jsparkSession)

I run the notebook- it finishes successfully but – the topic contains no messages
On logs of spline Kafka container I get

14:38:21.202 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN  o.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-spline-group-1, groupId=spline-group] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

What else should I check in order to make it work?
Please advise
docker-compose.txt
kafka-compose.txt

[cerveada]

You can take a look how it is done in this repository: https://github.com/AbsaOSS/spline-openlineage

Here is docker compose:
https://github.com/AbsaOSS/spline-openlineage/tree/master/docker/ol-integration-and-spline

You don't need the open-lineage integration part (rest-proxy and aggregator), but the rest is just whole spline with Kafka.

[zacayd]

@cerveada - can you share the correct yml- since it fails on

Pulling rest-proxy (my/spline-open-lineage-rest-proxy:latest)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

[cerveada]

As I said:

You don't need the open-lineage integration part (rest-proxy and aggregator)

just delete the services form docker compose and it should work.