[BUG]: Spark has to be restarted if using UDFs and recompiling

[borgdylan]

Whenever I use UDFs in an application utilizing .NET for Spark, and recompile the application regardless of whether I change the UDF code or not, the Spark worker no longer manages to load the UDF when it has to be executed as part of a query. The only workaround I found is to restart the Spark process before each run. This is OK when doing data science on a desktop, but not so if using Spark for server side processing where one expects Spark to keep running when updating the binaries that leverage Spark (there my be old and new instances during a roll-out for example). This is consistently reproducible for me when running Spark in debug mode (as a standalone server) regardless of which UDF I use or which dataset I use.

Desktop (please complete the following information):

• OS: Linux x64 (Ubuntu 19.10)

[elvaliuliuliu]

Thanks for your question. Can you please provide minimum repro (e.g. the command you use, a code snippet, etc.)?

[borgdylan]

I will code up a minimum repro after work and submit it here. When I encountered this I was using a non c# programming language so the repro will be in that same language. You will need desktop/mono msbuild to compile the code and .net core to run it. The language has no VS support so I will post the relevant command line steps to run it or leave a tasks.json for use with VSCode.

[borgdylan]

I has encoutntered this issue on v0.6.0. SInce there were UDF changes in v0.9.0, should I try with that version and see if it persists?

/cc @elvaliuliuliu

[elvaliuliuliu]

I has encoutntered this issue on v0.6.0. SInce there were UDF changes in v0.9.0, should I try with that version and see if it persists?

/cc @elvaliuliuliu

The v0.9.0 has the new feature to support UDF that returns Row object, if you want to try the new feature, you can use v0.9.0, otherwise I would recommend to provide a minimum repro. Thanks!

[imback82]

@borgdylan can you try setting spark.python.use.daemon to false? This will start a new process for each task, so it should pick up the new dlls (but at a bit of perf cost).

[borgdylan]

@imback82 How and where do I set it? On the spark-submit command line using --conf?

[imback82]

Yes, with --conf or when you create a SparkSession.

[borgdylan]

@imback82 That works. Using the config I coudlchange my code without restarting spark and I did not get the usual BadImageFormatException. The eception only happens when using UDFs that link to external libraries such as Newtonsoft or BouncyCastle.

[borgdylan]

sparkrepro.zip

[imback82]

What are the exception messages? Are you shipping those external libraries along with UDF dlls?

[borgdylan]

To run the repro you will need Desktop MSBuild. The project has a .vscode subfolder set up to build with msbuild found via PATH. It is using .NET for Spark v0.9.0. You will need to change the default RID in the .dylproj file to the one suitable for your machine. After building, you have to copy the people.json (dataset) file to the output directory where all the binaries are and from that folder start spark in debug mode. The exception triggers if running the code, leaving spark running in teh background, commenting out the second call to WIthColumn(), recompiling and re-running the code. There is a snippet that can be added to use @imback82 's workaround which works wonderfully. Below please find an archive with an xcopy-able vscode extension for syntax highlighting the code in the repro. The extension defines a new language 'dylandotnet' which allows you to change the C# extension package.json in such a way to allow debugging the repro code in the case running via dotnet is not enough.

[borgdylan]

dylandotnet-vsc.zip