Extend support for using AOT libraries in other AOT .NET libraries/executables

[LouChiSoft]

Is your feature request related to a problem? Please describe.

There are plenty of reasons why someone might want to compile a DLL to machine code, but then also still be able to use it in a .NET assembly. Currently the only way to do this is to treat the native DLL as if it was a DLL compiled by any other AOT language, exporting the function on one end and then importing it on the other.

It would be nice if in the event of a .NET to .NET linking, there was additional support to streamline and simplify the process, as well as be able to make certain assumptions that you can't make when you only know that the library is a native library and not a .NET AOT library.

Describe the solution you'd like

A new compilation target for AOT libraries designed to be used with other .NET projects, for the purposes of this ticket I will just call it a .netlib. The goal of a .netlib would be that someone (1st part/3rd party/etc.) compiles it, then the consumer of the library adds a reference it it in their .NET project. Whether it is a library or an executable referencing the .netlib shouldn't matter, and once the reference is made it should function exactly the same as if you were linking to a JIT assembly.

My assumption is that something like that could be achieved by taking the source code of the library you are compiling to a .netlib and creating a secondary non-AOT bridging library. The bridging library would only contain wrapper classes/functions that wrap around the publicly accessible classes/functions from the source code. Then when compiling the consumer executable/library the wrapper functions would be embedded in the consumer calling directly into the AOT library. During publishing of an application like this it would copy the executable and the DLL from the .netlib into the output folder.

The contents of a .netlib might contain something along the lines of a:

• Header: the section fo the .netlib that tells the compiler all the relevant information it needs to know, os-arch target. Size of the bridging library/aot library for copying the data later, etc. And other features, like what version of C# the code was written in to know what features are available when consuming the library.
• Bridging Library: The bridging library would just be a standard .NET assembly that would be linked to the consumer.
• The AOT library: the fully compiled machine code to be linked to at runtime..

Additional context

I want to stress I am not a compiler engineer, I am just a regular programmer. Solving (or otherwise simplifying) the AOT barrier within .NET would go a long way to making me want to write more code in .NET. Additionally, I would assume that when you can make the assumption that the DLL you are linking to was written in .NET you can call the DLL functions as if they are managed functions rather than unmanaged types (maybe I am wrong on this, but my current understanding is that AOT code is still somewhat managed)
As this feature would be intended to be used with other AOT targets, I also wonder if the overhead associated with calling DLL functions can be minimised. Currently, if you want to pass data to and from a native library there is a high chance that you have to go through a marshalling layer of some kind. Instead of you could make the assumption that the string type in the library is the same string type in the executable all of the marshalling can be removed and the data can be directly accessed.

I would love feedback on the proposal, if it's even possible and if so what the state of .NET AOT is in and if it would be close to being able to support such a feature. Thanks

[MichalStrehovsky]

There are plenty of reasons why someone might want to compile a DLL to machine code, but then also still be able to use it in a .NET assembly.

What is your scenario exactly?

There are a lot of problems with enabling something like this. Just one (non-sensical, but illustrative) example: say you are using System.Drawing.Point. A function in the native DLL takes Point as a parameter and just returns it back. At some point someone decides that Point should have a third field named Z. This is a non-breaking change in IL terms. It happens all the time. It's non breaking, because IL assemblies that reference a type from a different assembly don't encode things like "size of struct", "fields of struct". Instead they just refer to the type by name and the calculation of sizes, offsets, etc. happens at runtime. The assembly could have been compiled to IL 10 years ago, but it would still work on the latest runtime, even if we add 10 fields to Point.

If you have a DLL that was compiled ahead of time, the calculation of sizes/offsets/etc. happens at the time of AOT compilation. The type Point essentially disappears in the natively compiled DLL because native CPUs don't care about type or field names. If someone adds a new field after code generation happened, the code that uses the struct is now invalid and there's going to be mismatch between code in the natively compiled DLL and just-in-time calculated layout. This is a problem one could anticipate for a DllImport (types used in DllImport signatures need to follow different constraints and it is known that adding a field is a breaking change there), but one would not anticipate that for "regular .NET method calls".

[LouChiSoft]

What is your scenario exactly?

My scenario professionally is that we are evaluating alternative languages to C++ for some of our projects to a more memory safe language to fall in line with new safety requirements. But given the nature of our IP we can't share "easily decompilable code" which .NET would fall under, but .NET AOT doesn't have. We share our code both as a fully compiled DLLand as an executable so maintaining the two becomes a chore even in C++ and would not be made any easier in C#, at least in it's current sate.

Personally I also maintain several small tools that I use for other projects that all share a "common behaviour" DLL. If I wanted to swap those to .NET I would have to write the boiler plate for linking to the DLL manually. I figured that since all the information I am writing in the boiler plate already exists when you compile the original library I thought it might make sense to reuse that information to offer a more robust AOT library ecosystem.

There are a lot of problems with enabling something like this.

I absolutely expect there would be many issues with something like this. Like I said, I am not a compiler engineer so I am certain there will be things obvious to you guys that would make a feature like this difficult to implement. The main goal of my feature request was to get a discussion going around this. Maybe it can't be implemented exactly like I described above, but there can be a compromise that get's some of the functionality, I would certainly take that over the current sitation.

As a show of intent for discussing the issues that a feature like this could bring, let's take your example about System.Drawing.Point. You're right, if the internals of the type change, but the interface of the type is the same then we can run into incompatability issues. So maybe in the .netlib it will be required to include a hash of any time that shows up in a publicly accesible function. So at compile time anytime a public function that is called can do a quick hash check to see if the two types which have the same name are actually compatible and throw a compile error if they are not. Maybe this would be enough to solve that issues, maybe it wouldn't. It's just a suggestion

As more of an opinion than anything else, I think the creation and consumption of shared libraries has lagged behind the rest of the programming world in terms of modernisation. Take C++ for example, the process for creating a shared library in C++ hasn't really changed in God knows how long. You still have to write all your code, compile it to a library and then manually sanitize your headers to strip out any private information you don't want to give to the consumer of your library. On top of that if you are creating a cross-platform shared library you have other issues like having to include __declspec(dllimport), etc. in the windows version that you wouldn't in a linux version which leads to having to use preprocessor macros adding yet another layer of "shared library management" on top of your code.

If you have anymore questions please feel free to ask. I didn't write every idea I had revolving around this feature request for fear of making my initial post too long.

[MichalStrehovsky]

Is the motivation for AOT obfuscation, basically?

I think I don't understand why the DLL would need to be AOT compiled if it's going to be used from .NET.

If one is shipping a reusable .NET component, the IL assembly format was designed specifically for this (all the versioning, etc. is designed to allow someone to build an IL assembly and ship it to someone else for consumption with very few concerns about versionability).

[NCLnclNCL]

aot is not cross-platform, needs 3 files for 3 OS and 32/64 bit, not to mention aot is compiled all for one and trimming for aot so the file will be very large due to duplicate libraries if using many dlls

[kaleb-itm]

at a past job we had a source generator that would use a custom attribute to make method that used UnmanagedCallersOnlyAttribute to expose the method we want
then we have a build task that would make a header file to use it

[LouChiSoft]

Apologies for the delayed reponse:

Is the motivation for AOT obfuscation, basically?

While not personally a concern, internally for private IPs it's preferred by management for when you can say it's compiled to machine code. Yes I know this can still be reversed engineered, that's just how management is.

I think I don't understand why the DLL would need to be AOT compiled if it's going to be used from .NET.

Maybe I did a poor job of explaining, that's my bad. It doesn't have to be used from .NET via nuget or something. It could be distributed anyway and you would just add a reference and point to where on disk it is

If one is shipping a reusable .NET component, the IL assembly format was designed specifically for this (all the versioning, etc. is designed to allow someone to build an IL assembly and ship it to someone else for consumption with very few concerns about versionability).

Yes, unfortunately that is an issue with AOT compiling. And to get around it there would have to be some kind of ABI that tells the compiler how to link to the library when using it. Carbon (Google's C++ replacement experiment) have talked about having defined ABIs on the library boundary, with no internal ABI. So the only thing that is fixed would be library interface. I think a similar thing could be achieved in .NET by bundling the ABI "spec" in the .netlib. Then during the linker phase, instead of assuming a fixed ABI, it would use the one that library came with. There are definitely deatils here that would need to be properly explored to get something workable, but I do think that there is a solution that could be implemented.

aot is not cross-platform, needs 3 files for 3 OS and 32/64 bit, not to mention aot is compiled all for one and trimming for aot so the file will be very large due to duplicate libraries if using many dlls

Correct, but this would be a dev dependancy. End users wouldn't be affected as when you publish whatever application you are using the library with, it would make a copy of only the version of the DLL that is used by that platform.

at a past job we had a source generator that would use a custom attribute to make method that used UnmanagedCallersOnlyAttribute to expose the method we want
then we have a build task that would make a header file to use it

This is kind of what I meant, but instead of having a build task that would generate a header the compiler would do it all for you and bundle it into a nice single file