[BUG]: Watermarking in a streaming scenario Spark 2.4.x with Kafka on a Raspberry PI does not work.

[marcoparenzan]

Hello,

I have written this code: [(https://github.com/marcoparenzan/kafka-spark-on-pi-with-dotnet/blob/main/src/KafkaWordCountDotNetSpark/Program.cs)]
I'm executing this code on a Raspberry PI 4/4Gb with Spark 2.4.1 or 2.4.8. I have not yet tried 3.x.

Executing on Spark, that calls a Kafka server (on another Raspberry PI) does not work. In general communication with Kafka and aggregation (on console) works. The problem is have CSV/Parquet file generation with aggregation. It seems it does not "see" that I call Watermarking.

I have read lot of articles with equivalent code and they work. I have not yet read the inners of Spark Streaming. But my code seems correct.

Any Idea?

Thanks

Marco Parenzan