[SDK diff] Fix 3.1 ASP.NET shared framework containing unnecessary base sharedfx DLLs

[dagood]

Filing issue for Microsoft-built vs. source-built SDK diff in 3.1.109: #1770

I got this: https://gist.github.com/dagood/5b523381aa7200fdfe5fd7e9635853f9

I scanned through and saw that the Microsoft-built SDK doesn't have a few DLLs that source-build does have:

--- /work/testing-smoke/builtCli
+++ /work/microsoft-built-dotnet-sdk-3.1.109.tar.gz
@@ -1781,17 +2487,13 @@
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Extensions.WebEncoders.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.JSInterop.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Net.Http.Headers.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Win32.Registry.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Win32.SystemEvents.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Diagnostics.EventLog.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Drawing.Common.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.IO.Pipelines.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.AccessControl.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Cryptography.Cng.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Cryptography.Pkcs.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Cryptography.Xml.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Permissions.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Principal.Windows.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Windows.Extensions.dll

ResolvePackageFileConflicts is kicking in while building the ASP.NET Core sharedfx. In the binlog for Microsoft.AspNetCore.App.Runtime.csproj:

Encountered conflict between
  'Platform:Microsoft.Win32.Registry.dll' and
  'CopyLocal:/work/packages/restored/microsoft.win32.registry/4.7.0/runtimes/unix/lib/netcoreapp3.1/Microsoft.Win32.Registry.dll'.
  Choosing 'CopyLocal:/work/packages/restored/microsoft.win32.registry/4.7.0/runtimes/unix/lib/netcoreapp3.1/Microsoft.Win32.Registry.dll'
  because file version '4.700.20.47203' is greater than '4.700.19.56404'.
Encountered conflict between
  'Platform:System.Security.AccessControl.dll' and
  'CopyLocal:/work/packages/restored/system.security.accesscontrol/4.7.0/lib/netstandard2.0/System.Security.AccessControl.dll'.
  Choosing 'CopyLocal:/work/packages/restored/system.security.accesscontrol/4.7.0/lib/netstandard2.0/System.Security.AccessControl.dll'
  because file version '4.700.20.47203' is greater than '4.700.19.56404'.
...

4.700.20.47203 is the version we just source-built in dotnet/corefx. The ASP.NET sharedfx and base sharedfx both carry a copy.
4.700.19.56404 is the version in Microsoft.NETCore.App.Ref/3.1.0.

I think that in theory, these DLLs should be excluded from the ASP.NET Core sharedfx because they're in the base sharedfx. Not sure yet why that happens in the Microsoft build but not source-build.

I looked at a Microsoft build binlog for aspnet, and it looks like the only difference is that the Microsoft.Win32.Registry/4.7.0 it downloads (built back around 3.1.0) has the lower version of the DLL, 4.700.19.56404. The build uses Microsoft.NETCore.App.Ref/3.1.0's platform manifest, listing the old version, 4.700.19.56404. When they are an exact match, ResolvePackageFileConflicts chooses to use the Platform version and doesn't copy it into the sharedfx.

On our side, we rebuild Microsoft.Win32.Registry/4.7.0 with every single source-build, so the DLLs inside get a newer version corresponding with 3.1.9, 4.700.20.47203. The targeting pack isn't rebuilt, and it still has 4.700.19.56404 in the platform manifest. ASP.NET uses the platform manifest in the targeting pack, so it copies the DLLs from Microsoft.Win32.Registry/4.7.0 rather than using Platform's "old" DLLs.

This is yet another way that the Microsoft build reuse of old nupkgs is making a difference in source-build. My current granularity plan/proposal could help with this by testing that the file versions are the same between the old, reused Microsoft-built nupkg and the fresh bits that source-build produces. We already assume and rely on the DLLs being compatible, so making the file version the same so it's treated as part of the platform seems reasonable to me.

As far as I know, the only difference this makes is the 235 kB of extra DLLs...

[dseefeld]

@ericstj Can you look at this issue and advise if you think there will be a functional issue besides the larger file size with the difference between the source-build build and the "official" build?

[ericstj]

There shouldn't be a difference, just an extra copy of the DLLs. @dougbu might be able to explain why this doesn't happen in Microsoft servicing builds, perhaps ASP.NET pins the package versions to GA?

[dougbu]

perhaps ASP.NET pins the package versions to GA?

Yes, this is exactly what our 3.1 infrastructure does. We discussed this during the stablization servicing exercise and decided that pinning wasn't necessary for 5.0 because dotnet/runtime would keep the assembly versions in the "special" dozen packages consistent after GA. I mean "special" in the sense the packages are in Microsoft.AspNetCore.App.Ref or …Runtime but come from dotnet/runtime.

The discrepancies will go away entirely once dotnet/runtime ships all ref/ assemblies that dotnet/aspnetcore needs in Microsoft.NETCore.App.Ref or Microsoft.Extensions.Internal.Transport because we hope never to rev those packages after GA (unlike the loose packages that are so special).

[crummel]

[Scrub] This appears to still be an issue and may be related to the poisoning issues that @NikolaMilosavljevic is currently looking at.