AOT compilation support

[umlx5h]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

I'm trying to build an Avalonia desktop application using Lucene.NET. but I want to publish as an AOT executable.
However, when I perform an AOT build on an app using Lucene, the following runtime error occurs.

I used the sample code below.
https://lucenenet.apache.org/

Application: AvaloniaAOTTest.exe
CoreCLR Version: 10.0.6
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
   at System.Reflection.DynamicInvokeInfo.InvokeWithFewArguments(IntPtr, Byte&, Byte&, Object[], BinderBundle, Boolean) + 0x133
   at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0xbf
   at System.Delegate.DynamicInvokeImpl(Object[]) + 0x50
   at Avalonia.Interactivity.EventRoute.RaiseEventImpl(RoutedEventArgs) + 0x14b
   at Avalonia.Interactivity.EventRoute.RaiseEvent(Interactive, RoutedEventArgs) + 0xfd
   at Avalonia.Interactivity.Interactive.RaiseEvent(RoutedEventArgs) + 0x3c
   at Avalonia.Controls.Control.<>c.<.cctor>b__66_0() + 0xdf
   at Avalonia.Threading.DispatcherOperation.InvokeCore() + 0x192
   at Avalonia.Threading.DispatcherOperation.Execute() + 0x5f
   at Avalonia.Threading.Dispatcher.ExecuteJob(DispatcherOperation) + 0x78
   at Avalonia.Threading.Dispatcher.ExecuteJobsCore(Boolean) + 0xf0
   at Avalonia.Win32.Win32Platform.WndProc(IntPtr, UInt32, IntPtr, IntPtr) + 0x58
   at AvaloniaAOTTest!<BaseAddress>+0x82c99a

I can run the program by commenting out the sections that use Lucene.
Does this project support AOT? If not, are there any plans to add support?

Thanks.

Describe the solution you'd like

AOT available

Additional context

    <PackageReference Include="Lucene.Net" Version="4.8.0-beta00017" />
    <PackageReference Include="Lucene.Net.Analysis.Common" Version="4.8.0-beta00017" />

[NightOwl888]

Thanks for your inquiry. I did some research and it seems we already have all of the targets and dependencies we need to support AOT on net10.0, but it requires some additional guarantees that we either:

1. Provide no dynamic code
2. Mark sections that are dynamic "AOT safe"
3. Provide alternative AOT-safe implementations
4. Throw in places that do not support AOT

The issue is that some of our code:

1. Uses Reflection to create instances and scan for types
2. Provides dynamic code generation at runtime (such as in Lucene.Net.Expressions)
3. Uses the dynamic keyword (J2N)

It seems that we can work around this somewhat without designing any additional public APIs, but ultimately the best way is to add factory DI registrations of "external" components rather than using Reflection to scan for them. That would be a win for both regular and AOT users.

The Java implementation leaned heavily into a service locator called the Service Provider Interface (SPI) to scan for types in all packages that Java had loaded and make them available in a common way. This was utilized in a few specific places:

Codecs

We have already built proper dependency injection extensions in the Codecs, but the DefaultCodecFactory only loads types and creates instances lazily only if needed. For AOT, you would need to provide custom codec registration for the codec formats you use that don't rely on Reflection. That would probably get you a long way toward a solution.

Do note that the codecs also register themselves by name using .NET attributes to provide the name. Since getting the name requires Relection, you may need to put the names (strings) in a lookup table or switch case statement in your factory implementation.

FSDirectories

AOT has no support for MMapDirectory. This is the default setting when you call FSDirectory.Open() overloads on Windows and Linux environments. The workaround here is to use one of the other FSDirectory implementations if you want your index on disk (which is the normal thing to do).

• SimpleFSDirectory - uses FileStream for both read and write
• NIOFSDirectory - uses RandomAccess for read and FileStream for write

Alternatively, having an in-memory index is feasible for some use cases using RAMDirectory.

Analysis attributes

We changed the implementation from using Reflection exclusively to newing up the built-in attributes. I don't think you will have a problem here, but let us know if you are seeing something like AttributeSource in your logs.

Analysis factories

These aren't required by Lucene.NET at all, since we can new up analyzers separate from the factories. I think you can just avoid them, but they do depend on the SPI loader that was ported to scan all loaded assemblies for these types. I don't think this should be a problem for you now, but I will be working on adding better support for AOT so these are properly handled.

RamUsageEstimator

The RamUsageEstimator class also contains some Reflection code to analyze type structures of various objects. Some of these unsafe APIs are called from various codecs and a few other places. It looks like most of the Reflection calls are confined to the constructors. I am not sure whether this will affect runtime behavior, but I have my doubts the RAM usage estimates will be accurate.

J2N

We also have some dynamic calls in J2N that frankly, we would be better off without. I am looking for a way to remove them now.

This has now been addressed in NightOwl888/J2N#181. However, there was some other Reflection code in J2N that is triggered automatically by using AsReadOnly() method calls from BCL collection types or by using the Aggresssive mode of the DictionaryEqualityComparer<TKey, TValue>, ListEqualtiyComparer<T> or SetEqualityComparer<T>. Unfortunately, Lucene.NET a few BCL collections internally and may call AsReadOnly() on them to get into dangerous territory. There is no problem until either the Equals() or GetHashCode() methods are called on those colllections, which may trigger a chain of Reflection calls. We still haven't replaced some of the BCL collections (see #1272 and #1273, which are both up for grabs to anyone who wants to work on them).

Of course, to be on the safe side, it is recommended that if you pass a collection into a Lucene.NET API, make sure it is a J2N collection, which natively supports structural equality comparisons without any Reflection at all.

Experiment

I attempted to compile a quick example app using AOT and I got some warnings, but it ultimately compiled. When are you getting the error, compile time or runtime? Here is what I tried.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <PublishAot>true</PublishAot>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Lucene.Net" Version="4.8.0-beta00017" />
    <PackageReference Include="Lucene.Net.Analysis.Common" Version="4.8.0-beta00017" />
  </ItemGroup>

</Project>

using System;
using Lucene.Net.Analysis.Standard;
using Lucene.Net.Documents;
using Lucene.Net.Index;
using Lucene.Net.Store;
using Lucene.Net.Util;

namespace AOTCompilationCheckConsole
{
    internal class Program
    {
        static void Main(string[] args)
        {
            var dir = new RAMDirectory();
            var analyzer = new StandardAnalyzer(LuceneVersion.LUCENE_48);

            var config = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer);
            using var writer = new IndexWriter(dir, config);

            var doc = new Document();
            doc.Add(new TextField("field", "hello world", Field.Store.YES));
            writer.AddDocument(doc);
            writer.Commit();

            Console.WriteLine("OK");
        }
    }
}

dotnet publish -c Release -r win-x64 -p:PublishAot=true

Restore complete (1.7s)
  AOTCompilationCheckConsole net10.0 win-x64 succeeded with 5 warning(s) (42.1s) → bin\Release\net10.0\win-x64\publish\
    F:\Users\shad\.nuget\packages\j2n\2.1.0\lib\net8.0\J2N.dll : warning IL2104: Assembly 'J2N' produced trim warnings. For more information see https://aka.ms/il2104
    F:\Users\shad\.nuget\packages\j2n\2.1.0\lib\net8.0\J2N.dll : warning IL3053: Assembly 'J2N' produced AOT analysis warnings.
    F:\Users\shad\.nuget\packages\lucene.net\4.8.0-beta00017\lib\net8.0\Lucene.Net.dll : warning IL2104: Assembly 'Lucene.Net' produced trim warnings. For more information see https://aka.ms/il2104
    F:\Users\shad\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.win-x64\10.0.6\runtimes\win-x64\lib\net10.0\Microsoft.CSharp.dll : warning IL3053: Assembly 'Microsoft.CSharp' produced AOT analysis warnings.
    F:\Users\shad\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.win-x64\10.0.6\runtimes\win-x64\lib\net10.0\System.Linq.Expressions.dll : warning IL3053: Assembly 'System.Linq.Expressions' produced AOT analysis warnings.

Build succeeded with 5 warning(s) in 45.6s

[umlx5h]

@NightOwl888
Thank you for your detailed response.

I attempted to compile a quick example app using AOT and I got some warnings, but it ultimately compiled. When are you getting the error, compile time or runtime? Here is what I tried.

it's a runtime error.
If I run the sample code in the event handler triggered by a button click, it crashes with a unhandled exception.

If AOT isn't supported, it should throw a compile-time error, but that doesn't seem to be the case.
Adding an AOT-related attribute will likely cause a compile-time error.

    <StackPanel>
		<TextBlock Text="{Binding Greeting}" HorizontalAlignment="Center" VerticalAlignment="Center"/>
        <Button Content="Click Me" Click="Button_Click" />
    </StackPanel>

    private void Button_Click(object? sender, Avalonia.Interactivity.RoutedEventArgs e)
    {
        var dir = new RAMDirectory();
        var analyzer = new StandardAnalyzer(LuceneVersion.LUCENE_48);

        var config = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer);
        using var writer = new IndexWriter(dir, config);

        var doc = new Document();
        doc.Add(new TextField("field", "hello world", Field.Store.YES));
        writer.AddDocument(doc);
        writer.Commit();
    }

Click Me button causes following exception, got by event viewer.

Application: AvaloniaAOTTest.exe
CoreCLR Version: 10.0.6
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
   at System.Reflection.DynamicInvokeInfo.InvokeWithFewArguments(IntPtr, Byte&, Byte&, Object[], BinderBundle, Boolean) + 0x133
   at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0xbf
   at System.Delegate.DynamicInvokeImpl(Object[]) + 0x50
   at Avalonia.Interactivity.EventRoute.RaiseEventImpl(RoutedEventArgs) + 0x14b
   at Avalonia.Interactivity.EventRoute.RaiseEvent(Interactive, RoutedEventArgs) + 0xd7
   at Avalonia.Interactivity.Interactive.RaiseEvent(RoutedEventArgs) + 0x3c
   at Avalonia.Controls.Button.OnClick() + 0xc7
   at Avalonia.Controls.Button.OnPointerReleased(PointerReleasedEventArgs) + 0xdf
   at Avalonia.Reactive.LightweightObservableBase`1.PublishNext(T) + 0x23f
   at Avalonia.Interactivity.EventRoute.RaiseEventImpl(RoutedEventArgs) + 0xcc
   at Avalonia.Interactivity.EventRoute.RaiseEvent(Interactive, RoutedEventArgs) + 0xd7
   at Avalonia.Interactivity.Interactive.RaiseEvent(RoutedEventArgs) + 0x3c
   at Avalonia.Input.MouseDevice.MouseUp(IMouseDevice, UInt64, IInputRoot, Point, PointerPointProperties, KeyModifiers, IInputElement) + 0x140
   at Avalonia.Input.MouseDevice.ProcessRawEvent(RawPointerEventArgs) + 0x148
   at Avalonia.Input.InputManager.ProcessInput(RawInputEventArgs) + 0x30
   at Avalonia.Controls.PresentationSource.HandleInputCore(Object) + 0xfa
   at Avalonia.Threading.Dispatcher.Send(SendOrPostCallback, Object, Nullable`1) + 0xf1
   at Avalonia.Controls.PresentationSource.HandleInput(RawInputEventArgs) + 0xdf
   at Avalonia.Win32.WindowImpl.AppWndProc(IntPtr, UInt32, IntPtr, IntPtr) + 0x6ae
   at Avalonia.Win32.WindowImpl.WndProc(IntPtr, UInt32, IntPtr, IntPtr) + 0x7e
   at Avalonia.Win32.WindowImpl.WndProcMessageHandler(IntPtr, UInt32, IntPtr, IntPtr) + 0xba
   at AvaloniaAOTTest!<BaseAddress>+0x80d8fa

[NightOwl888]

@umlx5h

Yes, thanks for that, but the stack trace isn't actually telling us anything about what is wrong. However, being that J2N 2.1.0 uses the dynamic keyword, it is probably completely incompatible with AOT. I just pushed NightOwl888/J2N#181 with some refactoring and added the attributes and exceptions for both compile-time and runtime AOT trimming support. You can download the nuget asset here: https://dev.azure.com/NightOwl888/J2N/_build/results?buildId=597&view=artifacts&pathAsName=false&type=publishedArtifacts and drop it into a local folder as a package feed. Then you just need to add a direct PackageRefernce to the new version of J2N to upgrade it.

As for getting more info out of the build, I have found that a configuration like this will trigger the trimming build warnings that show line numbers and types that are triggering them.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <PublishAot>true</PublishAot>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>
  
  <PropertyGroup>
    <AnalysisLevel>latest</AnalysisLevel>
    <IsTrimmable>true</IsTrimmable>
    <EnableTrimAnalyzer>true</EnableTrimAnalyzer>
    <SuppressTrimAnalysisWarnings>false</SuppressTrimAnalysisWarnings>
    <TrimmerSingleWarn>false</TrimmerSingleWarn>
    <IlcGenerateCompleteTypeMetadata>true</IlcGenerateCompleteTypeMetadata>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Lucene.Net" Version="4.8.0-beta00017" />
    <PackageReference Include="Lucene.Net.Analysis.Common" Version="4.8.0-beta00017" />
    <PackageReference Include="J2N" Version="2.2.0-alpha-0083-g981d86e95a" />
  </ItemGroup>

</Project>

Warning: EnableTrimAnalyzer tends to use a lot of RAM and slow down MSBuild.

Here is a list of the issues I got for a very basic application. Unfortunately, the search functionality is completely dependent upon "attributes", which are not the same thing as the .NET System.Attribute class. They contain a lot of Reflection code and I have my doubts searching or even building an index will work at this point.

F:\Users\shad\source\repos\AOTCompilationCheckConsole\AOTCompilationCheckConsole>dotnet publish -c Release -r win-x64 -p:PublishAot=true -v:diag
Restore complete (3.2s)
    Determining projects to restore...
    Restored F:\Users\shad\source\repos\AOTCompilationCheckConsole\AOTCompilationCheckConsole\AOTCompilationCheckConsole.csproj (in 1.29 sec).
  AOTCompilationCheckConsole net10.0 win-x64 succeeded with 8 warning(s) (35.4s) → bin\Release\net10.0\win-x64\publish\
    Generating native code
    ILC : Trim analysis warning IL2067: Lucene.Net.Codecs.DefaultCodecFactory.NewCodec(Type): 'type' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicParameterlessConstructor', 'DynamicallyAccessedMemberTypes.NonPublicConstructors' in call to 'System.Activator.CreateInstance(Type,Boolean)'. The parameter 'type' of method 'Lucene.Net.Codecs.DefaultCodecFactory.NewCodec(Type)' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
    ILC : Trim analysis warning IL2026: Lucene.Net.Codecs.DefaultCodecFactory.ScanForCodecs(Assembly): Using member 'System.Reflection.Assembly.GetTypes()' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. Types might be removed.
    ILC : Trim analysis warning IL2072: Lucene.Net.Util.AttributeSource.AttributeFactory.DefaultAttributeFactory.CreateAttributeInstance<S>(): 'type' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicParameterlessConstructor' in call to 'System.Activator.CreateInstance(Type)'. The return value of method 'Lucene.Net.Util.AttributeSource.AttributeFactory.DefaultAttributeFactory.GetClassForInterface<S>()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
    ILC : Trim analysis warning IL2070: Lucene.Net.Util.AttributeSource.<>c__DisplayClass14_0.<GetAttributeInterfaces>b__0(Type): 'this' argument does not satisfy 'DynamicallyAccessedMemberTypes.Interfaces' in call to 'System.Type.GetInterfaces()'. The parameter 'clazz' of method 'Lucene.Net.Util.AttributeSource.GetAttributeInterfaces(Type)' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
    ILC : Trim analysis warning IL2070: Lucene.Net.Util.RamUsageEstimator.CreateCacheEntry(Type): 'this' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicFields', 'DynamicallyAccessedMemberTypes.NonPublicFields' in call to 'System.Type.GetFields(BindingFlags)'. The parameter 'clazz' of method 'Lucene.Net.Util.RamUsageEstimator.CreateCacheEntry(Type)' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
    ILC : Trim analysis warning IL2075: Lucene.Net.Util.Attribute.ReflectWith(IAttributeReflector): 'this' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicFields', 'DynamicallyAccessedMemberTypes.NonPublicFields' in call to 'System.Type.GetFields(BindingFlags)'. The return value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
    ILC : Trim analysis warning IL2075: Lucene.Net.Util.Attribute.ToString(): 'this' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicFields', 'DynamicallyAccessedMemberTypes.NonPublicFields' in call to 'System.Type.GetFields(BindingFlags)'. The return value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
    ILC : Trim analysis warning IL2026: Lucene.Net.Util.AttributeSource.AttributeFactory.DefaultAttributeFactory.CreateAttributeWeakReference(Type,Type&): Using member 'System.Reflection.Assembly.GetType(String,Boolean)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. Types might be removed by trimming. If the type name is a string literal, consider using Type.GetType instead.

Build succeeded with 8 warning(s) in 41.1s

Of course, these messages are just what come out of a basic minimal app. There is probably going to be more to fix than this to make both Lucene.Net and Lucene.Net.Analysis.Common safe to use. It would be helpful if we had a PR or two to move the needle forward on this. Note that I updated the list of known issues in my comment above and included some links to open issues that are related to this PR.