[BUG]: Running a job with ForeachBatch under Databricks causes the Job to never stop

[muhammad-s-zainal]

Describe the bug
Using ForeachBatch in a spark job deployed to Databricks causes the job to never terminate.

To Reproduce

Steps to reproduce the behavior:

1. Follow this tutorial to setup dotnet spark on Databricks:
   https://learn.microsoft.com/en-us/dotnet/spark/tutorials/databricks-deployment
2. Include ForeachBatch in your structured streaming query, for example:

var query = df
      .WriteStream()
      .Format("delta")
      .OutputMode("append")
      .ForeachBatch(UpsertToDelta)
      .Option("checkpointLocation", $"{destPath}/checkpoint")
      .Option("path", destPath)
      .Trigger(Trigger.Once())
      .Start();

For more info, refer to this page on implementing ForeachBatch and upserting into a delta table:
https://docs.databricks.com/_static/notebooks/merge-in-streaming.html

4. Run the job. I can confirm that the streaming query will actually proceed and finish however the Databricks UI will continue to show the job as running until it is manually terminated.

Expected behavior
Databricks Job terminates after streaming query is complete.

Additional context

• FYI, I have raised a support ticket with Databricks and Azure and both have expressed that the issue is with the dotnet spark APIs.
• Using Microsoft.Spark.Extensions.Delta 2.0.0 for delta table support

[GeorgeS2019]

@muhammad-s-zainal

I am tracking why this is .NET for Spark

Currently Microsoft.Spark.Extensions.Delta is 2.11.
Have u resolved this issue?

[muhammad-s-zainal]

I've tried it with Microsoft.Spark.Extensions.Delta 2.1.1, same issue unfortunately.