SqlClient issue related to DOTNET environment variable

[dbeavon]

I'm getting this annoying error from SQL.
System.PlatformNotSupportedException: 'System.Data.SqlClient is not supported on this platform.'

Things work fine on the driver while debugging via org.apache.spark.deploy.dotnet.DotnetRunner. But when the same type of code is delegated to the Spark.Worker.TaskRunner process, those keeps hitting these exceptions.

I think the problem is related to either the .Net Core version, or related to the DOTNET_ASSEMBLY_SEARCH_PATHS.

The driver reports the System.Environment.Version as 3.1.10 and the worker reports it as 3.1.8. This seems like it could be an issue (although probably unlikely).

I think I found a better clue related to the origin of the assembly System.Data.SqlClient.dll. The worker is loading it from a path like so:

c:\LoadEpicorJournalEntryData\bin\Debug\netcoreapp3.1\System.Data.SqlClient.dll

... which is in the directory that is set by DOTNET_ASSEMBLY_SEARCH_PATHS.

... But the driver loads System.Data.SqlClient.dll from a path like so:

c:\LoadEpicorJournalEntryData\bin\Debug\netcoreapp3.1\runtimes\win\lib\netcoreapp2.1\System.Data.SqlClient.dll

I'll continue playing with this but would appreciate any help. Honestly I'm not that familiar with .Net Core ... so I don't know how it loads assemblies in the normal case, let alone when there is an environment variable being used (like DOTNET_ASSEMBLY_SEARCH_PATHS).

I may try copying the "good" assembly over the "bad" one and see what happens. But manual steps like that aren't great as a long-term solution.

[dbeavon]

By copying the file from the "runtimes\win\lib" back down to DOTNET_ASSEMBLY_SEARCH_PATHS, I was able to execute the entire SqlConnection constructor.

But things subsequently fail when opening the connection, with .Open():

System.DllNotFoundException
  HResult=0x80131524
  Message=Unable to load DLL 'sni.dll' or one of its dependencies: The specified module could not be found. (0x8007007E)

I was able to find that file and copy it back to DOTNET_ASSEMBLY_SEARCH_PATHS as well. Now everything works.

Would love to hear how others handle this issue. I'm assuming the problem is related to unmanaged code for specific platforms.

[dbeavon]

I switched from using the System.Data.SqlClient nuget to Microsoft.Data.SqlClient. But the issue still remains.

I think the problem is in the way that the .Net core Worker is handling the loading of assemblies.

There is a "deps.json" file in the output directory of the app (also the same directory as DOTNET_ASSEMBLY_SEARCH_PATHS). My understanding is that it is supposed to be used by the .Net core assembly loader to help it load the correct assemblies at runtime. Only I'm guessing that the our spark worker (Microsoft.Spark.Worker-1.0.0) is either ignoring that file or is assuming the paths are relative to where this (Microsoft.Spark.Worker-1.0.0) process lives.

Here is a comment that pointed me to "deps.json"...
dotnet/runtime#23639 (comment)

I'd appreciate any guidance on the best approach for handling this type of dependency.