What changes was applied to 2.78.x?

[gimlichael]

Hello,

When calling docfx from within a container using both platform linux/arm64,linux/amd64 everything works fine PRIOR to 2.78.x.

Hereafter I receive this error for linux/arm64:

I think its related to xref/href lookup. After some complains from toc.yml that it cannot find the reference, it fails with: qemu: uncaught target signal 11 (Segmentation fault) - core dumped

Capturing some more info resolves in this:

18.75 /build/docfx/toc.yml(0,1): warning InvalidFileLink: Invalid file link:(~/packages).
18.75 /build/docfx/toc.yml(0,1): warning InvalidFileLink: Invalid file link:(~/api/Codebelt.Extensions.Xunit.html).
18.77 /build/docfx/api/namespaces/Codebelt.Extensions.Xunit.Hosting.AspNetCore.Http.Features.md: warning DuplicateUids: Uid(Codebelt.Extensions.Xunit.Hosting.AspNetCore.Http.Features) has already been defined in api/Codebelt.Extensions.Xunit.Hosting.AspNetCore.Http.Features.yml.
18.77 /build/docfx/api/namespaces/Codebelt.Extensions.Xunit.Hosting.AspNetCore.Http.md: warning DuplicateUids: Uid(Codebelt.Extensions.Xunit.Hosting.AspNetCore.Http) has already been defined in api/Codebelt.Extensions.Xunit.Hosting.AspNetCore.Http.yml.
18.78 /build/docfx/api/namespaces/Codebelt.Extensions.Xunit.Hosting.AspNetCore.md: warning DuplicateUids: Uid(Codebelt.Extensions.Xunit.Hosting.AspNetCore) has already been defined in api/Codebelt.Extensions.Xunit.Hosting.AspNetCore.yml.    
18.78 /build/docfx/api/namespaces/Codebelt.Extensions.Xunit.Hosting.md: warning DuplicateUids: Uid(Codebelt.Extensions.Xunit.Hosting) has already been defined in api/Codebelt.Extensions.Xunit.Hosting.yml.
18.78 /build/docfx/api/namespaces/Codebelt.Extensions.Xunit.md: warning DuplicateUids: Uid(Codebelt.Extensions.Xunit) has already been defined in api/Codebelt.Extensions.Xunit.yml.
38.26 47 external references found in xref maps.
38.27 41 uids are unresolved.
38.27 Normalizing all the object to weak type
39.47 Feeding options from template...
42.24 Updating href...
42.30 /build/docfx/toc.yml: Unable to find file "api/Codebelt.Extensions.Xunit.html" for Href referenced by TOC file "toc.yml"
42.30 /build/docfx/toc.yml: Unable to find file "api/Codebelt.Extensions.Xunit.html" for TopicHref referenced by TOC file "toc.yml"
42.30 /build/docfx/toc.yml: Unable to find file "packages" for Href referenced by TOC file "toc.yml"
42.30 /build/docfx/toc.yml: Unable to find file "packages" for TopicHref referenced by TOC file "toc.yml"
42.31 Applying system metadata to manifest...
42.54 Disposing processor ManagedReferenceDocumentProcessor ...
42.55 Disposing build step ApplyOverwriteDocumentForMref ...
42.55 Disposing build step BuildManagedReferenceDocument ...
42.55 Disposing build step FillReferenceInformation ...
42.55 Disposing build step SplitClassPageToMemberLevel ...
42.55 Disposing build step ValidateManagedReferenceDocumentMetadata ...
42.55 Disposing processor RestApiDocumentProcessor ...
42.55 Disposing build step ApplyOverwriteDocumentForRestApi ...
42.55 Disposing build step BuildRestApiDocument ...
42.55 Disposing build step ValidateRestApiDocumentMetadata ...
42.55 Disposing processor UniversalReferenceDocumentProcessor ...
42.55 Disposing build step ApplyOverwriteDocumentForUref ...
42.55 Disposing build step BuildUniversalReferenceDocument ...
42.55 Disposing build step FillReferenceInformation ...
42.55 Disposing processor TocDocumentProcessor ...
42.55 Disposing build step BuildTocDocument ...
42.55 Disposing processor ResourceDocumentProcessor ...
42.55 Disposing build step ValidateResourceMetadata ...
42.55 Disposing processor ConceptualDocumentProcessor ...
42.55 Disposing build step BuildConceptualDocument ...
42.55 Disposing build step ValidateConceptualDocumentMetadata ...
42.55 Disposing processor ApiPageDocumentProcessor ...
42.56 Disposing processor html ...
43.20 AggregateException: One or more errors occurred. (Object reference not set to an
43.20 instance of an object.) (Object reference not set to an instance of an object.)
43.20 (Object reference not set to an instance of an object.) (Object reference not
43.20 set to an instance of an object.) (Object reference not set to an instance of an
43.20 object.) (Object reference not set to an instance of an object.) (Object
43.20 reference not set to an instance of an object.) (Object reference not set to an
43.20 instance of an object.) (Object reference not set to an instance of an object.)
43.20 (Object reference not set to an instance of an object.) (Object reference not
43.20 set to an instance of an object.) (Object reference not set to an instance of an
43.20 object.) (Object reference not set to an instance of an object.) (Object
43.20 reference not set to an instance of an object.) (Object reference not set to an
43.20 instance of an object.) (Object reference not set to an instance of an object.)
43.20 (Object reference not set to an instance of an object.) (Object reference not
43.20 set to an instance of an object.) (Object reference not set to an instance of an
43.20 object.) (Object reference not set to an instance of an object.) (Object
43.20 reference not set to an instance of an object.) (Object reference not set to an
43.20 instance of an object.) (Object reference not set to an instance of an object.)
43.20 (Object reference not set to an instance of an object.)
43.20      NullReferenceException: Object reference not set to an instance of an      
43.21      object.
43.21        at StringComparer get_CurrentCulture()
43.25        at EnumerableSorter<TElement> GetEnumerableSorter(
43.25           EnumerableSorter<TElement> next)
43.25        at int[[]] SortedMap(TElement[] buffer)
43.25        at bool MoveNext()
43.25        at void <ApplySystemMetadata>b__0(ManifestItemWithContext m) in
43.25           ManifestProcessor.cs:164
43.25        at void <RunAll>b__0(TElement s) in DocumentExceptionExtensions.cs:95
43.25        at void <ForWorker>b__1(ref RangeWorker currentWorker, long timeout, out
43.25           bool replicationDelegateYieldedBeforeCompletion)
43.25        at void <ForWorker>b__1(ref RangeWorker currentWorker, long timeout, out
43.25           bool replicationDelegateYieldedBeforeCompletion)
43.25        at void Execute()
43.25   at void Run<TState>(ReplicatableUserAction<TState> action, ParallelOptions
43.25      options, bool stopOnFirstFailure)
43.25   at ParallelLoopResult ForWorker<TLocal,TInt>(TInt fromInclusive, TInt
43.25      toExclusive, ParallelOptions parallelOptions, Action<TInt> body,
43.25      Action<TInt, ParallelLoopState> bodyWithState, Func<TInt,
43.25      ParallelLoopState, TLocal, TLocal> bodyWithLocal, Func<TLocal> localInit,
43.25      Action<TLocal> localFinally)
43.25   at ParallelLoopResult ForWorker<TLocal,TInt>(TInt fromInclusive, TInt
43.26      toExclusive, ParallelOptions parallelOptions, Action<TInt> body,
43.26      Action<TInt, ParallelLoopState> bodyWithState, Func<TInt,
43.26      ParallelLoopState, TLocal, TLocal> bodyWithLocal, Func<TLocal> localInit,
43.26      Action<TLocal> localFinally)
43.26   at ParallelLoopResult ForEachWorker<TSource,TLocal>(IEnumerable<TSource>
43.26      source, ParallelOptions parallelOptions, Action<TSource> body,
43.26      Action<TSource, ParallelLoopState> bodyWithState, Action<TSource,
43.26      ParallelLoopState, long> bodyWithStateAndIndex, Func<TSource,
43.26      ParallelLoopState, TLocal, TLocal> bodyWithStateAndLocal, Func<TSource,
43.26      ParallelLoopState, long, TLocal, TLocal> bodyWithEverything, Func<TLocal>
43.26      localInit, Action<TLocal> localFinally)
43.26   at ParallelLoopResult ForEach<TSource>(IEnumerable<TSource> source,
43.26      ParallelOptions parallelOptions, Action<TSource> body)
43.26   at void RunAll<TElement>(IEnumerable<TElement> elements, Action<TElement>
43.26      action, int parallelism, CancellationToken cancellationToken) in
43.26      DocumentExceptionExtensions.cs:84
43.26   at void RunAll<TElement>(IReadOnlyList<TElement> elements, Action<TElement>
43.26      action, int parallelism, CancellationToken cancellationToken) in
43.27      DocumentExceptionExtensions.cs:68
43.27   at void ApplySystemMetadata() in ManifestProcessor.cs:148
43.27   at void Process() in ManifestProcessor.cs:51
43.27   at void Handle(List<HostService> hostServices, int maxParallelism) in
43.27      LinkPhaseHandler.cs:32
43.27   at Manifest Build(DocumentBuildParameters parameters, IMarkdownService
43.27      markdownService, CancellationToken cancellationToken) in
43.27      SingleDocumentBuilder.cs:71
43.27   at void Build(IList<DocumentBuildParameters> parameters, string
43.27      outputDirectory, CancellationToken cancellationToken) in DocumentBuilder.cs
43.27      :124
43.27   at void BuildDocument(BuildJsonConfig config, BuildOptions options,
43.27      TemplateManager templateManager, string baseDirectory, string
43.27      outputDirectory, string templateDirectory, CancellationToken
43.27      cancellationToken) in DocumentBuilderWrapper.cs:42
43.27   at string Exec(BuildJsonConfig config, BuildOptions options, string
43.27      configDirectory, string outputDirectory, CancellationToken
43.27      cancellationToken) in RunBuild.cs:39
43.27   at void <Execute>b__0() in BuildCommand.cs:25
43.27   at int Run(LogOptions options, Action run) in CommandHelper.cs:48
43.27   at int Execute(CommandContext context, BuildCommandOptions settings) in
43.27      BuildCommand.cs:15
43.27   at Task<int> Execute(CommandContext context, CommandSettings settings) in
43.27      CommandOfT.cs:40
43.27   at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext
43.27      context, ITypeResolver resolver, IConfiguration configuration) in
43.27      CommandExecutor.cs:166

The linux/amd64 build works fine - same complains, but no errors.

I hope this can shed some light on the changes for supporting arm-based systems up till 2.78.x ..

[filzrev]

What's version of .NET SDK is used for docfx 2.78.x ? (it can'be confirmed with dotnet --list-sdks command)

If .NET 9 is used. Try following settings

• Set DOTNET_JitEnablePhysicalPromotion=0 environment variable

Background
There is known JIT problem on .NET 9 SDK when using IDictionary<string, object>.

• https://github.com/dotnet/runtime/issues/108969
• https://github.com/dotnet/aspnetcore/issues/59341

I've confirmed CI tests works as expected when running tests with ubuntu-24.04-arm environment.
But it can't reproduced and it expected to be container environment specific issues.
So I'm suspecting above JIT related issues.

[gimlichael]

I have used both .NET 8 and .NET 9 - same problem - and only >= v2.78.

Setting the proposed ENV did not solve the issue.

For now I will remain on v2.77 - and when time allows it - dig a bit deeper.

Keep up the otherwise good work.

[filzrev]

Thanks for your confirmations.

As far as I've confirmed.
There are no code changed that are relating stacktrace information on docfx (>= v2.78) otherwise target frameworks.
And StringComparer get_CurrentCulture() getter property should not throw NullReferenceException.
So I'm suspecting known JIT issues.

I don't have a linux/arm64 environment, so it would be helpful if someone could identify the cause in more details.

[gimlichael]

I am using docker buildx, so if you are up for it, feel free to clone https://github.com/codebeltnet/xunit and changing https://github.com/codebeltnet/xunit/blob/main/.docfx/Dockerfile.docfx from 2.77.0 to 2.78.2.

After change of said mentioned version, run BuildDocfxImage.ps1 (https://github.com/codebeltnet/xunit/blob/main/.docfx/BuildDocfxImage.ps1).

Thank you for investigating this issue.

---

For full transparency, this is the source of the docker image: https://github.com/codebeltnet/docfx (I did not commit the 2.78.2 as I was still puzzled by the error), but only version have changed.

[filzrev]

qemu: uncaught target signal 11 (Segmentation fault) - core dumped

The above error seems to occur on QEMU emulated environment.
(Running arm64 container image on docker host that running on x64)

Currently .NET SDK don't support QEMU environment.

The QEMU emulator is not supported to run .NET apps. QEMU is used, for example, to emulate Arm64 containers on x64, and vice versa.

So I though the error is occurred because it's not supported by .NET SDK.

[gimlichael]

Oh, this is interesting. Oddly enough it works with v2.77 (and versions prior to this) .. and I can see you kept support for .NET 8, so my assumption would be it that is should work here - hence, only .NET 9 would fail.

As mentioned, I will try to investigate further, but your investigation for sure shed some new light to the challenge ahead.

As I recall, i received qemu: uncaught target signal 11 (Segmentation fault) - core dumped when using default log level. Extending to a more verbose showed the actual error. Findings related to this did not match my scope of experience.

My original suspicion was actually surrounding your recent Playwright changes as seen here: v2.77.0...v2.78.0#diff-87db21a973eed4fef5f32b267aa60fcee5cbdf03c67fafdc2a9b553bb0b15f34 / https://github.com/dotnet/docfx/blob/618e71b424a265cff7cd140dfbafbe56a379371c/.github/workflows/release.yml where you disregard the build-platform.

But I don't have the insights to decided if this is only related to PDF generation or general for DocFX engine.

Thanks.

[filzrev]

On my windows Docker Desktop environment.
Both docfx build and docfx metadata command failed with following error.

qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

So I though it's not related to Playwright relating pdf generations.

And these commands are successfully completed when disable QuickJIT by setting following environment.

export DOTNET_TieredCompilation=0

So it's expected to be QuickJIT related issues on QEMU emulated environment.

As noted above comment. QEMU emulated environment is not supported by .NET.
These behaviors is not consistent between environments.

[gimlichael]

But as I understand the QEMU its triggered by Node.js and not .NET SDK. I mean, I have a lot of projects being build for .NET using ARM64 through docker buildx build without any problems. Again, just my findings and suspicion for versions greater or equal to v2.78.

[gimlichael]

Not quite sure what changed .. but as a reference, I got the issued solve by using --platform=$BUILDPLATFORM: https://github.com/codebeltnet/xunit/blob/main/.docfx/Dockerfile.docfx