ClassNotFoundException when using SparkBatchClient

[ShahdNara]

My goal is to trigger a .NET Spark job programmatically. I'm following this tutorial , but I keep getting ClassNotFoundException (for my WordCount C# class).
It works only when I trigger the Spark job using Synapse Studio UI (by creating an Apache Spark Job definition).

The .zip file for the .NET application has the .dll for the class.

Code

SparkBatchClient client = new SparkBatchClient(new Uri(endpoint), sparkPoolName, new DefaultAzureCredential());

string name = "name";
string file = "abfss://xxx.core.windows.net/jobs/WordCount.zip";

SparkBatchJobOptions request = new SparkBatchJobOptions(name, file)
{
    ClassName = "WordCount",
    DriverMemory = "28g",
    DriverCores = 4,
    ExecutorMemory = "28g",
    ExecutorCores = 4,
    ExecutorCount = 2
};

SparkBatchOperation createOperation = await client.StartCreateSparkBatchJobAsync(request);
SparkBatchJob jobCreated = await createOperation.WaitForCompletionAsync();

---

Error
Livy:
22/06/14 20:36:58 WARN DependencyUtils: Skip remote jar abfss://xxx.core.windows.net/jobs/WordCount.zip.

Driver (stderr):

22/06/13 19:54:27 ERROR ApplicationMaster: Uncaught exception: 
java.lang.ClassNotFoundException: WordCount
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:722)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:496)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:268)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:905)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:904)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1732)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:904)
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

[ylashin]

I had an identical issue but I was submitting the job via the UI. My initial mistake was that the UI defaults job type to Scala so I had to change it to .NET Spark.

In a nutshell, you need to have:

• A .net core 3.1 executable (console app)
• Modify Language to .NET Spark
• Upload your zipped published folder content and specify executable file name without the extension

Hopefully REST API and the SDK have knobs for all the above.

HTH