Issues with full Framework build

[0shi]

Hi there,
I've been evaluating a couple of MozJpeg / LibJpeg libraries for some post-processing work, and had some issues getting the full Framework (I've tried in console applications with net45 and net471) build working locally. I encountered the following issues:

1. The dlls are not copied from the packages folder into the bin folder as part of a build unless you are using both a new style .csproj AND you have included a RuntimeIdentifier in the csproj. This does, however, work regardless of your values for AppendTargetFrameworkToOutputPath and AppendRuntimeIdentifierToOutputPath.

2. The dlls cannot be read from the location they are copied to using the existing code. This is because they look in a subfolder for the dll, rather than in the root, as you can see here:
   

   AS.TurboJpegWrapper/Quamotion.TurboJpegWrapper/TurboJpegImport.cs

   Line 125 in f754900

   if (Environment.Is64BitProcess)

This appears to fail because AppDomain.CurrentDomain.SetupInformation.ApplicationBase already includes the RuntimeIdentifier portion of the path, and it does not need to be appended again (Meanwhile, the dll is now located in the folder located by AppDomain.CurrentDomain.SetupInformation.ApplicationBase).

I've worked around this issue by creating those folders manually and pasting the dlls into them as part of my build process, so this is not a blocker.

Thanks!

[georg-jung]

Did you encounter the same behaviour when evaluating my fork? (context: I created MozJpegSharp, which is a fork of this repo that uses MozJPEG and comes with prebuilt binaries too. @0shi opened an issue at MozJpegSharp, which is why I know he tried it.)

When I do a short test

• create a new default 4.7.2 console app (AnyCPU)

• install from NuGet as packages.config dependency

• put the following in Main

  using (var tjc = new TJCompressor())
  {
  
  }
  

I observe that I get a System.TypeInitializationException with Quamotion.TurboJpegWrapper (details below) but it works fine with MozJpegSharp. Thus, I assume that some of my changes fixed this issue. Maybe I'm able to help fixing it here too. Afaik (sadly I don't have a link right now) native binaries are copied automatically from NuGet packages for SDK-style projects but not for older ones. To overcome this issue I took some inspiration from SQLitePCL.raw, which has to handle this too and is very broadly used (i.e. as base of EF Core SQLite) and seems very carefully maintained. As a result I created MozJpegSharp.targets, that becomes part of the final nuget package and contains instructions for copying the files around (I liked NuGetPackageExplorer for taking a look).

To summarize, I think adding an appropriate .targets file to the NuGet package would solve this. Edit: I also changed the directory structure a bit, see TurboJpegImport.cs. It seems that this is what gets created automatically when using new csprojs, so I adapted the code and manual copying for old csprojs to behave the same.

Exception details:

System.TypeInitializationException: "Der Typeninitialisierer für "TurboJpegWrapper.TurboJpegImport" hat eine Ausnahme verursacht."

Inner Exception:
ArgumentOutOfRangeException: The directory 'C:\Users\georg\source\repos\LjtTest\LjtTest\bin\Debug\' does not contain a subdirectory for the current architecture. The directory 'C:\Users\georg\source\repos\LjtTest\LjtTest\bin\Debug\win7-x86' does not exist.
Parametername: directory

Stack Trace of Inner Exception:
   bei TurboJpegWrapper.TurboJpegImport.Load(String directory)
   bei TurboJpegWrapper.TurboJpegImport.Load()
   bei TurboJpegWrapper.TurboJpegImport..cctor()

Working directory structure using MozJpegSharp (see also here)

bin\Debug\MjsTest.exe
bin\Debug\MjsTest.exe.config
bin\Debug\MjsTest.pdb
bin\Debug\MozJpegSharp.dll
bin\Debug\MozJpegSharp.pdb
bin\Debug\runtimes
bin\Debug\runtimes\win-x64
bin\Debug\runtimes\win-x64\native
bin\Debug\runtimes\win-x64\native\turbojpeg.dll
bin\Debug\runtimes\win-x64\native\vcruntime140.dll
bin\Debug\runtimes\win-x86
bin\Debug\runtimes\win-x86\native
bin\Debug\runtimes\win-x86\native\turbojpeg.dll
bin\Debug\runtimes\win-x86\native\vcruntime140.dll

[0shi]

@georg-jung - Thanks for confirming a repro! I did not experience this with your fork.

If it helps I have a few sample projects which can easily produce the issue, but you should be able to just create a .NET Framework console application from the template (To test with old .CSPROJ style) or create a new .NET Core console application from the template and change the target framework (To test with the new .CSPROJ style)

[0shi]

@georg-jung - Unfortunately I spoke a little too soon. It seems that AppDomain.CurrentDomain.SetupInformation.ApplicationBase returns the Project's folder rather than the bin folder for ASP.Net applications.
I put together a quick repro project that reproduces the issue for both AS.TurboJpegWrapper and MozJpegSharp
For simplicity I've just put some code to create the respective TCompressors in a controller, but that's not our use case.
Note that this is a full Framework project, so the TurboJpegWrapper dll is not even copied into the bin folder, but the MozJpegSharp dlls are.

I tried a few common methods for resolving dlls in this context and wasn't able to get anything useful:

• AppDomain.CurrentDomain.SetupInformation.ApplicationBase Returns the project folder (e.g. "C:\Repos\0shi\ASPNetJpegTest\Web\")
• System.Web.HttpContext.Current.Server.MapPath("") Returns the project folder followed by the controller (e.g. "C:\Repos\0shi\ASPNetJpegTest\Web\LibJpegTurbo")
• System.Reflection.Assembly.GetExecutingAssembly().Location Returns the path of a temporary folder containing only a single dll (e.g "C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\vs\be1e12e9\69624d91\assembly\dl3\117134b5\005f5af2_3df8d501\MozJpegSharp.dll"). Same for System.Reflection.Assembly.GetCallingAssembly().Location

I finally found some success with System.Reflection.Assembly.GetExecutingAssembly().CodeBase, which returns e.g. "file:///C:/Repos/0shi/ASPNetJpegTest/Web/bin/MozJpegSharp.DLL". Obviously this would need some parsing, but looks promising. Under a console application this returns the path to the executing exe, so seems like it would fit there as well.

If this isn't suitable, not sure how to get around this without letting users override the importer. Again, I can work around this by copying the folders to the correct place manually, but now that correct place is our Project folder rather than a bin folder, which isn't desirable. My workaround would be to add those folders to .gitignore and copy them there as a build step.

EDIT: In case anyone else needs the build steps I'm using:

  <ItemGroup
    <MozJpegDlls Include="..\packages\MozJpegSharp.1.0.27\runtimes\**">
      <InProject>false</InProject>
    </MozJpegDlls>
  </ItemGroup>
  <Target Name="BeforeBuild">  
    <Copy SourceFiles="@(MozJpegDlls)" DestinationFiles="@(MozJpegDlls->'.\runtimes\%(RecursiveDir)%(Filename)%(Extension)')" SkipUnchangedFiles="true" />
  </Target>

[georg-jung]

Thanks for all the research! I somehow got the feeling there needs to be an easier solution to this. I mean, it can't be the intention of the framework authors that every .Net Standard lib author who consumes native code has to juggle with paths in this way. Maybe it just got that hard because we started thinking about paths in the first place. I'll look into it...

[georg-jung]

While looking at the solutions other projects took, I came across multiple approaches. I started summarizing what I found, but it's still very much a WIP. I'll post my notes here though a) to keep them in the right place and b) because they might give some starting points if someone else looks for this:

For anything that's not .Net Core 3+, there does not seem to be the way to go.

1. SQLitePCLRaw

   • put binaries in ./runtimes/<rid>/native/<lib> on every platform (.targets copies there, ProjectReference does so automatically too obviously)
   • the .targets file just copies binaries for the specific platform as whenever it is used, the build is platform specific anyway (is this true?)
     • on Windows it does copy x86, x64 and arm though, as we could be building an AnyCPU assembly that needs specific binaries during runtime
   • use System.Runtime.InteropServices.NativeLibrary on core 3+, implement own logic everywhere else, multitarget library
   • thats what EF Core SQLite does internally, so it can not be so wrong, still see also here

2. libgit2sharp

   • copy binaries to ./lib in .targets (all bins for every platform)
     • ProjectReference certainly still goes the runtimes way so the build folders have a different structure dependeing on the target
   • custom loading logic that uses S.R.IS.NativeLibrary when available via reflection to circumvent multi-targeting

3. NativeLibraryLoader package

   • out-source all the loading logic
   • is kind of outdated as it never uses S.R.IS.NativeLibrary; uses it's own logic in any case
   • adds a dependency to the library (probably this isn't strictly necessary for achieving what this does, thinking of i.e. Nullable)

4. ImGui

   • don't use any custom loading logic
     • on core, ./runtimes/... will be used automatically
     • the system defaults apply, so the .targets just copies the binaries to .
   • while this approach is very simple and covers most of the use-cases, I guess it breaks execution of AnyCPU assemblies on non-64bit machines that were build on 64bit machines.
   • I really don't know what happens to mono. As .targets just copies the binary for every platform, it might just work well.

5. SkiaSharp

6. System.IO.Compression

   • part of the core runtime itself

Left to do for me:

• look into SkiaSharp and System.IO.Compression
• how does core3+ itself do it? backport that?
• compare the approaches and decide for a way to proceed
• from what I've seen so far I'm tempted to go the (1.) way
• it could be possible to create a library that simplifies the chosen approach. Do I want to do this?
• ask this on StackOverflow?

[robeving]

Also experiencing this issue after upgrade

[ruicaramalho]

I tried creating the folder win7-x64 and copied the file Quamotion.TurboJpegWrapper.dll but now complains about turbojpeg.dll
Could not load libturbojpeg from ....\bin\Debug\win7-x64\turbojpeg.dll

[georg-jung]

You could try placing the Quamotion....dll directly besides your .exe and the native lib below an additional runtimes folder (in addition to your existing structure). See also above.

You can also try installing MozJpegSharp (which is based on this project; disclaimer: I created it) and see if that works. If it does you can look at the build output that is created while it is installed and replicate that manually when switching back to this package.

[ruicaramalho]

I found the file turbojpeg.dll added to the folder win7-x64 now it works.

Don't forget to change the propertiy Copy to Output Directory to Copy if newer for the dll files inside folder win7-x64