NativeAOT and XmlSerializer

[danielo-unity3d]

Description

When publishing a test application which uses XML serialization, vi code generated via xscgen (see here) along with PublishAot, the application crashes because the serialization code relies heavily upon reflection.

The stack trace is, then:

bin/Debug/net8.0/linux-x64/publish/XmlSerializationTest
Unhandled Exception: System.InvalidOperationException: There is an error in the XML document.
 ---> System.InvalidOperationException: There was an error reflecting type 'ACME.Package.Xml.Schema.PackageACL'.
 ---> System.InvalidOperationException: You must implement a default accessor on System.Collections.ObjectModel.Collection`1[[ACME.Package.Xml.Schema.Package, XmlSerializationTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]] because it inherits from ICollection.
   at System.Xml.Serialization.TypeScope.GetDefaultIndexer(Type, String) + 0x4e9
   at System.Xml.Serialization.TypeScope.GetCollectionElementType(Type, String) + 0x44
   at System.Xml.Serialization.TypeScope.ImportTypeDesc(Type, MemberInfo, Boolean) + 0x732
   at System.Xml.Serialization.TypeScope.GetTypeDesc(Type, MemberInfo, Boolean, Boolean) + 0x172
   at System.Xml.Serialization.StructModel.GetPropertyModel(PropertyInfo) + 0xf6
   at System.Xml.Serialization.StructModel.GetFieldModel(MemberInfo) + 0xb0
   at System.Xml.Serialization.XmlReflectionImporter.InitializeStructMembers(StructMapping, StructModel, Boolean, String, RecursionLimiter) + 0x777
   at System.Xml.Serialization.XmlReflectionImporter.ImportStructLikeMapping(StructModel, String, Boolean, XmlAttributes, RecursionLimiter) + 0x57f
   at System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(TypeModel, String, XmlReflectionImporter.ImportContext, String, XmlAttributes, Boolean, Boolean, RecursionLimiter) + 0x90d
   --- End of inner exception stack trace ---
   at System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(TypeModel, String, XmlReflectionImporter.ImportContext, String, XmlAttributes, Boolean, Boolean, RecursionLimiter) + 0xb9f
   at System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(TypeModel, String, XmlReflectionImporter.ImportContext, String, XmlAttributes, RecursionLimiter) + 0x57
   at System.Xml.Serialization.XmlReflectionImporter.ImportElement(TypeModel, XmlRootAttribute, String, RecursionLimiter) + 0x15e
   at System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(Type, XmlRootAttribute, String) + 0xa5
   at System.Xml.Serialization.XmlSerializer.GenerateXmlTypeMapping(Type type, XmlAttributeOverrides overrides, Type[] extraTypes, XmlRootAttribute root, String defaultNamespace) + 0xba
   at System.Xml.Serialization.XmlSerializer.GetMapping() + 0x5e
   at System.Xml.Serialization.XmlSerializer.DeserializeUsingReflection(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) + 0x33
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) + 0x1e9
   --- End of inner exception stack trace ---
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) + 0x4ed
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle) + 0x83
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader) + 0x27
   at ACME.Xml.XmlExtensions.<ParseTag>d__0`1.MoveNext() + 0x222
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1, Boolean&) + 0x16d
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1) + 0x4c
   at ACME.XmlSerializationTest.Program.Main(String[] args) + 0x125
   at XmlSerializationTest!<BaseAddress>+0x9e6cc7
   at XmlSerializationTest!<BaseAddress>+0x9e6d4d
[1]    500757 IOT instruction (core dumped)  bin/Debug/net8.0/linux-x64/publish/XmlSerializationTest

However, I have (in the real application) also seen the following stack:

 ---> System.NotSupportedException: 'System.Xml.Serialization.ReflectionXmlSerializationReaderHelper.GetSetMemberValueDelegateWithType[ACME.Package.Xml.Schema.PackageACL,System.DateTime](System.Reflection.MemberInfo)' is missing native code. MethodInfo.MakeGenericMethod() is not compatible with AOT compilation. Inspect and fix AOT related warnings that were generated when the app was published. For more information see https://aka.ms/nativeaot-compatibility
   at System.Reflection.Runtime.MethodInfos.RuntimeNamedMethodInfo`1.GetUncachedMethodInvoker(RuntimeTypeInfo[], MemberInfo) + 0x61
   at System.Reflection.Runtime.MethodInfos.RuntimeConstructedGenericMethodInfo.get_UncachedMethodInvoker() + 0x3b
   at System.Reflection.Runtime.MethodInfos.RuntimeMethodInfo.get_MethodInvoker() + 0x60
   at System.Reflection.Runtime.MethodInfos.RuntimeNamedMethodInfo`1.MakeGenericMethod(Type[]) + 0x363
   at System.Xml.Serialization.ReflectionXmlSerializationReader.GetSetMemberValueDelegate(Object, String) + 0x40a
   at System.Xml.Serialization.ReflectionXmlSerializationReader.SetOrAddValueToMember(Object, Object, MemberInfo) + 0xc5
   at System.Xml.Serialization.ReflectionXmlSerializationReader.<>c__DisplayClass53_1.<WriteLiteralStructMethod>g__Wrapper|1(Object value) + 0x61
   at System.Xml.Serialization.ReflectionXmlSerializationReader.WriteAttribute(ReflectionXmlSerializationReader.Member, Object) + 0x4be
   at System.Xml.Serialization.ReflectionXmlSerializationReader.WriteAttributes(ReflectionXmlSerializationReader.Member[], ReflectionXmlSerializationReader.Member, UnknownNodeAction, Object&) + 0x35b
   at System.Xml.Serialization.ReflectionXmlSerializationReader.WriteLiteralStructMethod(StructMapping, Boolean, Boolean, String) + 0x13dc
   at System.Xml.Serialization.ReflectionXmlSerializationReader.WriteStructMethod(StructMapping, Boolean, Boolean, String) + 0x72
   at System.Xml.Serialization.ReflectionXmlSerializationReader.WriteElement(ElementAccessor, Boolean, Boolean, String, Int32, XmlSerializationReader.Fixup, ReflectionXmlSerializationReader.Member) + 0xc34
   at System.Xml.Serialization.ReflectionXmlSerializationReader.WriteMemberElementsIf(ReflectionXmlSerializationReader.Member[], ReflectionXmlSerializationReader.Member, UnknownNodeAction, XmlSerializationReader.Fixup, ReflectionXmlSerializationReader.CheckTypeSource) + 0x718
   at System.Xml.Serialization.ReflectionXmlSerializationReader.WriteMemberElements(ReflectionXmlSerializationReader.Member[], UnknownNodeAction, UnknownNodeAction, ReflectionXmlSerializationReader.Member, ReflectionXmlSerializationReader.Member, XmlSerializationReader.Fixup, List`1) + 0x116
   at System.Xml.Serialization.ReflectionXmlSerializationReader.GenerateTypeElement(XmlTypeMapping) + 0x3f7
   at System.Xml.Serialization.ReflectionXmlSerializationReader.ReadObject() + 0xd4
   at System.Xml.Serialization.XmlSerializer.DeserializeUsingReflection(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) + 0xc6
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) + 0x1e9
   --- End of inner exception stack trace ---
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) + 0x4ed
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle) + 0x83
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader) + 0x27

The test project, which I have tried to prune as much as possible, can be found here: XmlSerializationTest.zip

• Can the linker be forced to instantiate the pruned functions? (private/internal classes and methods, with generic types)
  • Is it generally preferable to use ILLink Descriptor Format or the Rd.xml File Format (marked as internal)?
• In the test project. does it matter what flags are used for xscgen, and what attributes are finally used to mark the generated classes and fields, and
  • what collections are eventually used (default is System.Collections.ObjectModel.Collection`1)
• Are there any plans to support this scenario? (I could only find issue Annotate remaining runtime libraries for NativeAOT #75480)
• How can System.Private.Xml be modified (to support this scenario)?
  • Is this just a question of addressing all linker warnings (IL3050) that are printed during the link phase (when TrimmerSingleWarn is true), or are other warnings/issues (perhaps being masked)?
  • Occurrences of IL3050 are:
    • src/libraries/System.Private.Xml/src/System/Xml/Serialization/Types.cs: 25 hits
    • src/libraries/System.Private.Xml/src/System/Xml/Serialization/CodeGenerator.cs: 1 hit
    • src/libraries/System.Private.Xml/src/System/Xml/Serialization/XmlSerializationReader.cs: 6 hits
    • src/libraries/System.Private.Xml/src/System/Xml/Serialization/XmlSerializationWriter.cs: 1 hit
    • src/libraries/System.Private.Xml/src/System/Xml/Serialization/NameTable.cs: 1 hit
• Would the support ever be available in .Net 8, or just .Net 9? (could the solution be back-ported if we get to that point?)

Reproduction Steps

Compile and run attached project:

dotnet publish -c Debug
bin/Debug/net8.0/linux-x64/publish/XmlSerializationTest

Expected behavior

The test program should successfully serialize and deserialize a basic data structure to and from XML.

Actual behavior

See stack trace of crash in the description.

Regression?

No.

Known Workarounds

No response

Configuration

dotnet --info
.NET SDK:
 Version:           8.0.303
 Commit:            29ab8e3268
 Workload version:  8.0.300-manifests.c915c39d
 MSBuild version:   17.10.4+10fbfbf2e

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  24.04
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /home/user/.dotnet/sdk/8.0.303/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.7
  Architecture: x64
  Commit:       2aade6beb0

.NET SDKs installed:
  6.0.424 [/home/user/.dotnet/sdk]
  7.0.410 [/home/user/.dotnet/sdk]
  8.0.303 [/home/user/.dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.32 [/home/user/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.20 [/home/user/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.7 [/home/user/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.32 [/home/user/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.20 [/home/user/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.7 [/home/user/.dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_ROOT       [/home/user/.dotnet]

global.json file:
  /home/user/Unity/licensing.entitlement4/global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Other information

See also this issue in xscgen

[RenderMichael]

It would be amazing if XML serialization could get the same treatment as JSON and have a source-generated AOT-safe {de}serializer.

Such a thing would be a large undertaking, and although I think it would be worth it (a shocking amount of people and enterprises use XML and have no plans on transitioning away), I can understand any hesitancy in committing to that kind of project.

[filipnavara]

We were facing the same issue, and I had some short discussion with people on the .NET team. There's no immediate solution right now, and in the end we ended up rewriting most of our code to manually implement IXmlSerializable interface and manually [de]serialize through XmlReader and XmlWriter. For cases where that was not viable we use
DynamicDependency attribute at the moment to tell the AOT compiler what to preserve. One option that was discussed was to extend Serde.NET to support some of the XML scenarios and have people migrate to it.

[filipnavara]

Such a thing would be a large undertaking

The thing is, there's already a code for the source generation (SGen) but there's no way to integrate it into NativeAOT executable easily. The traditional SGen (available on modern .NET as NuGet) generates an additional assembly and the XmlSerializer has code paths that find this assembly and load it at runtime. That obviously doesn't work on NativeAOT since you cannot load assembly dynamically, you need to precompile it, and the code to do the loading in XmlSerializer is stripped anyway.

SGen uses reflection to load the original assembly and to form an object model for the XML schema. It then uses the XML schema model to source generate the code. Unfortunately, the reflection code is quite old and depends on the runtime loading of the assembly. It's not possible to simply migrate it to work on top of System.Reflection.Metadata.

I had the idea to use a modified Roslyn.Reflection to create a source generator that would simply reuse most of the existing infrastructure for source generation. The problem turned out to be that lot of the schema reading code depends on the types existing in the same runtime context, much like the issue with System.Reflection.Metadata linked above.

(Now, if you start from an XML schema in the first place, you can write a code that load it, runs the same SGen internal code to produce the source code, and you will be 90% there with "nice" C# code. It still doesn't fix everything since the source generated code still relies on some reflection logic that doesn't quite work in NativeAOT scenarios, but that would be fixable.)

[filipnavara]

After some back and forth with @Suchiman here's a minimal sample of using SGen in NativeAOT: https://gist.github.com/filipnavara/1e8831c256498bad53b3aae94af87a20

There are few things that were done there:

1. Added SGen using official documentation
2. Added LinkSerializationAssembly target in .csproj to add the SGen output as ILC input w/ rooting
3. Added hack in .csproj / ILLink.Substitutions.xml to override XmlSerializer.Mode to ReflectionAsBackup. The default for NativeAOT is ReflectionOnly which breaks loading the pre-generated assemblies.
4. Added [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] to the root XML class.

Presumably,
2) can be scripted in the Microsoft.XmlSerializer.Generator NuGet,
3) should be fixable by adjusting the behavior of existing ReflectionOnly mode, and,
4) can be fixed with the right annotations in .NET Runtime

[danielo-unity3d]

Thanks for your comments, @filipnavara.

I took your gist, and ran with it.
I appear to have hit a bug caused by SGen (or rather, the XmlReflectionImporter inside System.Private.Xml) not taking the namespace into account when building its typemap (such that two classes with the same name in two separate XML namespaces, mapped into two distinct C# namespaces, will collide).

I have a fix, and can generate the serialization code (for my assembly), but:

• it needs to be integrated (ie. the csproj needs to run the fixed version of the tool), and I'm have issues with the workflow.
• I need to check that the generated code actually works with AOT.

Stay tuned.

[alex-everitt-2277]

Also running into this issue, unfortuantely looks like AOT will not be possible for our usecase until some sort of support comes along for XmlSerialization. Praying for that JSON treatment and a proper source generator 🤞

[ProviceUnify]

Same issue on NET 8 and NET 9 RC 1. But AOT can deal with simple XML files. But i got dead end error with XmlArray on List public fields

public class ClassName
{
    [DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(ClassName))]
    public ClassName() { }

    [XmlArray("ItemsName")]
    [XmlArrayItem("ItemName")]
    public List<ItemType> Item { get; init; }
}

You must implement a default accessor on System.Collections.Generic.List`1[[ItemName, Namespace, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]] because it inherits from ICollection

[Suchiman]

That exception is being raised from here

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/Types.cs

Line 1356 in c4ebd38

throw new InvalidOperationException(SR.Format(SR.XmlNoDefaultAccessors, type.FullName));

If i read the code correctly, it's trying to find the indexer on the List<ItemType> (that is list[42] = ...) but its probably trimmed away or omitted from reflection info.
Try adding typeof(List<ItemType>).GetDefaultMembers(); somewhere in your code that is guaranteed to execute.
That should hint NAOT sufficiently to make sure to include that information.

[ProviceUnify]

Yes. It works, thank you @Suchiman. I have large and complex xml with few levels of nesting. Good way to hint NAOT what types are using in model is adding new TypeName = new() or typeof(TypeName).GetDefaultMembers() in constructors (or other place. I haven't checked other variants).

Pseudo-code for clarification what i mean. You should recheck it if you suppose to use it but main conception saved:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Root>
	<ChildNode/>
	<AnotherChildNodes>
		<AnotherChildNode/>
		<AnotherChildNode/>
		<AnotherChildNode/>
		<AnotherChildNode/>
	</AnotherChildNodes>
</Root>

// this code won't work in NativeAot

[XmlRoot]
public class Root {
   public ChildNodeType ChildNode {get;set;}

  [XmlArray("AnotherChildNodes")]
  [XmlArrayItem("AnotherChildNode")]
  public List<AnotherChildNodeType> AnotherChildNodes {get;set;}
}

public class AnotherChildNodeType {
  // fields
}

public class ChildNodeType {
  // fields
}

// this code will

[XmlRoot]
public class Root {
  [DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(Root))]
  public Root() {
   typeof(List<AnotherChildNodeType>).GetDefaultMembers();
   typeof(AnotherChildNodeType).GetDefaultMembers();
   typeof(ChildNodeType).GetDefaultMembers();
  }  

   public ChildNodeType ChildNode {get;set;}

  [XmlArray("AnotherChildNodes")]
  [XmlArrayItem("AnotherChildNode")]
  public List<AnotherChildNodeType> AnotherChildNodes {get;set;}
}

public class AnotherChildNodeType {
  public AnotherChildNodeType() {
    // typeof(FieldOfClass).GetDefaultMembers(); ...
  }
  // fields
}

public class ChildNodeType {
  public ChildNodeType() {
    // typeof(FieldOfClass).GetDefaultMembers(); ...
  }
  // fields
}

[ThaDaVos]

Any updates on this?
I am running into the same issue when trying to migrate my Library to NativeAoT (It's used as a library for a Clarion project, the only part left is the XML stuff)