Failure to build .NET 8 preview 4 from source

[mateusrodrigues]

Describe the Bug

I tried building .NET 8 preview 4 from source by cloning the dotnet/dotnet repository and checking out the tag v8.0.0-preview.4.23259.5.

The first time, I tried building using the command: VERBOSE=1 ./build.sh --clean-while-building --with-sdk ./previously-built-dotnet -- /v:n /p:SkipPortableRuntimeBuild=true /p:LogVerbosity=n /p:MinimalConsoleLogOutput=false /p:ContinueOnPrebuiltBaselineError=true

Which resulted in:

...
                  Build FAILED.
                  
                         "/home/ubuntu/dn8/dotnet8-preview4/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj" (default target) (1) ->
                         "/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf" (Restore target) (8) ->
                         (Restore target) -> 
                           /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Ref with version (= 7.0.2) [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 5 version(s) in previously-source-built [ Nearest version: 8.0.0-preview.3.23174.8 ] [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 5 version(s) in reference-packages [ Nearest version: 7.0.0 ] [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 0 version(s) in source-built [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 0 version(s) in prebuilt [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                           /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Ref with version (= 7.0.2) [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 4 version(s) in previously-source-built [ Nearest version: 8.0.0-preview.3.23177.8 ] [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 3 version(s) in reference-packages [ Nearest version: 7.0.0 ] [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 0 version(s) in source-built [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                         /home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/src/Shared/Microsoft.AspNetCore.Razor.Utilities.Shared/Microsoft.AspNetCore.Razor.Utilities.Shared.csproj : error NU1102:   - Found 0 version(s) in prebuilt [/home/ubuntu/dn8/dotnet8-preview4/src/razor/artifacts/source-build/self/src/Razor.Slim.slnf]
                  
                      0 Warning(s)
                      2 Error(s)
...

Full log file: https://drive.google.com/file/d/1CpGIlq1bDBVHr1onCCrpKCDPasZn3RBW/view?usp=sharing

Then, on a clean clone of the same repo and tag, I tried adding the --online flag to the same command as above, which resulted in a failed build a bit later in the process:

...
         Build FAILED.
         
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.ubuntu.23.10-x64 with version (= 7.0.2) [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Found 1 version(s) in source-built [ Nearest version: 8.0.0-preview.4.23259.5 ] [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Found 1 version(s) in previously-source-built [ Nearest version: 8.0.0-preview.3.23174.8 ] [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Found 0 version(s) in prebuilt [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Found 0 version(s) in reference-packages [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Found 0 version(s) in dotnet-eng [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Found 0 version(s) in dotnet-public [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Found 0 version(s) in myget-legacy@Local [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Versions from nuget-build were not considered [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.Build.Tasks.Console/NuGet.Build.Tasks.Console.csproj : error NU1102:   - Versions from vside were not considered [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.ubuntu.23.10-x64 with version (= 7.0.2) [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Found 1 version(s) in source-built [ Nearest version: 8.0.0-preview.4.23259.5 ] [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Found 1 version(s) in previously-source-built [ Nearest version: 8.0.0-preview.3.23174.8 ] [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Found 0 version(s) in prebuilt [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Found 0 version(s) in reference-packages [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Found 0 version(s) in dotnet-eng [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Found 0 version(s) in dotnet-public [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Found 0 version(s) in myget-legacy@Local [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Versions from nuget-build were not considered [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
         /home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/src/NuGet.Core/NuGet.CommandLine.XPlat/NuGet.CommandLine.XPlat.csproj : error NU1102:   - Versions from vside were not considered [/home/ubuntu/dn8/dotnet8-preview4-online/src/nuget-client/build/restorehelper.targets]
             0 Warning(s)
             2 Error(s)
...

Full log file: https://drive.google.com/file/d/1XPrtvpec-eI8uj8oJ3J2V6Qjj_BS7Pzi/view?usp=sharing

Steps to Reproduce

1. Launch a clean Ubuntu 23.10 LXD container with lxc launch ubuntu:23.10
2. Install the .NET build prerequisites for Linux as documented
3. Add this PPA to the apt sources
4. Run sudo apt update && sudo apt upgrade
5. Run sudo apt install dotnet-sdk-8.0 dotnet-sdk-8.0-source-built-artifacts
6. git clone https://github.com/dotnet/dotnet && cd dotnet
7. git checkout v8.0.0-preview.4.23259.5
8. Run ln -s /usr/lib/dotnet/source-built-artifacts/Private.SourceBuilt.Artifacts.8.0.100.ubuntu.23.10-x64.tar.gz ./prereqs/packages/archive/ from the repo root
9. Run cp -RLp /usr/lib/dotnet/ ./previously-built-dotnet from the repo root
10. Run either of the ./build.sh commands described above

Other Information

• The dotnet-sdk-8.0-source-built-artifacts includes the pre-built binaries from the bootstrap package, which are placed under /usr/lib/dotnet/source-built-artifacts/Private.SourceBuilt.Artifacts.8.0.100.ubuntu.23.10-x64.tar.gz

cc @MichaelSimons

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[omajid]

There are a few packages (such as Microsoft.NETCore.App.Ref) that are not part of previously-source-built (aka Private.SourceBuilt.Artifacts), but are still currently needed to build .NET 8. In an online source build, they are placed at Private.SourceBuilt.Prebuilts*-x64.tar.gz

$  tar tf ./prereqs/packages/archive/Private.SourceBuilt.Prebuilts.0.1.0-8.0.100-25.centos.8-x64.tar.gz
Microsoft.CodeCoverage.IO.17.7.1-beta.23213.1.nupkg
Microsoft.NETCore.App.Host.centos.8-x64.7.0.2.nupkg
Microsoft.NETCore.App.Host.centos.8-x64.7.0.5.nupkg
Microsoft.NETCore.App.Host.fedora.36-x64.7.0.2.nupkg
Microsoft.NETCore.App.Host.fedora.36-x64.7.0.5.nupkg
microsoft.aspnetcore.app.ref.7.0.2.nupkg
microsoft.aspnetcore.app.ref.7.0.5.nupkg
microsoft.internal.testplatform.remote.17.6.33617.297.nupkg
microsoft.netcore.app.host.linux-arm64.7.0.2.nupkg
microsoft.netcore.app.host.linux-arm64.7.0.5.nupkg
microsoft.netcore.app.host.linux-x64.7.0.2.nupkg
microsoft.netcore.app.host.linux-x64.7.0.5.nupkg
microsoft.netcore.app.ref.7.0.2.nupkg
microsoft.netcore.app.ref.7.0.5.nupkg

In your reproducer steps, you don't run ./prep.sh and don't link the prebuilts package to ./prereqs/packages/archive/, so that archive is missing and the build fails.

I don't yet understand the failure with --online, though.

[omajid]

Can you try the --online steps on any older Ubuntu version? The 23.10 RID is not present in https://github.com/dotnet/dotnet/blob/release/8.0.1xx-preview4/src/runtime/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json which makes me wonder if it's just NuGet having trouble trying to figure out that Microsoft.NETCore.App.Host.ubuntu.23.10-x64 can be substituted by Microsoft.NETCore.App.Host.linux-x64?

[MichaelSimons]

To reiterate @omajid's comments and add a little...

To build offline you will need a set of prebuilts (we are working on removing these) which can be mostly be obtained via running prep.sh --no-sdk --no-artifacts. The exception is the rid specific apphost package which is what you are seeing in your online build. This is a 7.0 rid specific prebuilt that MSFT cannot provide for all RIDs. Since Ubuntu source-builds 7.0, you can retrieve it from your own 7.0 artifacts.