Non-Deterministic missing libraries in artifacts for same codebase (no changes)

[humble-codes]

Description

During random builds generated through GitHub actions on Amazon EKS, upon running these specific builds always get ReflectionTypeLoadException thrown saying - Could not load file or assembly 'SomeAssemblyName, Version=VersionNumber, Culture=neutral, PublicKeyToken=SomeToken'. The system cannot find the file specified.

Project Details

1. It is a multi-project based solution (API, Infra, Core, etc. )
2. API project is the start-up project
3. The solution is based upon dotnet 8
4. We use GitHub actions with self hosted runner to build and deploy the project and later run it on lambda.

Necessary files and commands used

• API project's .csproj

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <AWSProjectType>Lambda</AWSProjectType>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <PublishReadyToRun>false</PublishReadyToRun>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="9.0.0" />
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer.Hosting" Version="1.7.0" />
    <PackageReference Include="CsvHelper" Version="33.1.0" />
    <PackageReference Include="MediatR.Contracts" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="6.0.16" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Versioning" Version="5.1.0" />
    <PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.5" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />
    <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.7.0" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\platform-jjz-service-core\platform-jjz-service-core.csproj" />
    <ProjectReference Include="..\platform-jjz-service-domain\platform-jjz-service-domain.csproj" />
    <ProjectReference Include="..\platform-jjz-service-Infrastructure\platform-jjz-service-Infrastructure.csproj" />
  </ItemGroup>
  <ItemGroup>
    <Content Update="appsettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
  </ItemGroup>
</Project>

• dotnet CLI commands used in GitHub pipeline to publish project/artifact

  dotnet restore
  dotnet publish -c Release -r linux-x64 --self-contained false -o ./publish

Expected Behaviour

When running the code via dotnet cli, the published artifact should run without any errors for every build created using same codebase on lambda (given that there are no other errors in codebase).

Actual Behaviour

When lambda tries to run the published artifacts, some artifacts (that were generated via same codebase) always throw ReflectionTypeLoadException exception. Only after couple of attempts to re-publishing same code, this issue gets resolved on its own.
I get the same error as the lambda error, if I try running the GitHub artifact on my local windows machine using dotnet platform_jjz_service_api.dll

Stack trace for runtime exception

Unhandled exception. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at MediatR.Registration.ServiceRegistrar.<>c.<ConnectImplementationsToTypesClosing>b__1_0(Assembly a)
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at MediatR.Registration.ServiceRegistrar.ConnectImplementationsToTypesClosing(Type openRequestInterface, IServiceCollection services, IEnumerable`1 assembliesToScan, Boolean addIfAlreadyExists, MediatRServiceConfiguration configuration)
   at MediatR.Registration.ServiceRegistrar.AddMediatRClasses(IServiceCollection services, IEnumerable`1 assembliesToScan, MediatRServiceConfiguration configuration)
   at MediatR.ServiceCollectionExtensions.AddMediatR(IServiceCollection services, IEnumerable`1 assemblies, Action`1 configuration)
   at MediatR.ServiceCollectionExtensions.AddMediatR(IServiceCollection services, Assembly[] assemblies)
   at platform_jjz_service_api.Startup.ConfigureServices(IServiceCollection services) in /runner/_work/clin/clin/platform_jjz_service_api/Startup.cs:line 292
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass7_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.InitializeServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at platform_jjz_service_api.LocalEntryPoint.Main(String[] args) in /runner/_work/clin/clin/platform_jjz_service_api/LocalEntryPoint.cs:line 10
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5'. The system cannot find the file specified.
File name: 'Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5'

My findings

1. This is a non-deterministic error, which randomly occurs (only some builds throw this runtime exception, other builds for same code work totally fine)
2. When compared an erroneous artifact with a correct artifact, noticed that number of libraries in platform_jjz_service_api.deps.json is lesser in erroneous artifact.
3. Changed the dotnet commands slightly to publish the release artifact, but no luck there :

  dotnet nuget locals all --clear
  dotnet restore --locked-mode
  dotnet publish -c Release -r linux-x64 --self-contained false -o ./publish

Known Workarounds

Keep re-deploying the same code mutiple times, eventually in one of the deployments, this issue goes away

[jkotas]

When compared an erroneous artifact with a correct artifact, noticed that number of libraries in platform_jjz_service_api.deps.json is lesser in erroneous artifact.

I assume that Microsoft.Data.SqlClient is missing in the erroneous artifact. Is that correct?

Moving to dotnet/sdk since it is a build problem.

[humble-codes]

@jkotas I thought so too and went ahead to check the artifacts, but Microsoft.Data.SqlClient was present there and same assembly version is present in both good as well as bad artifact - 5.0.0.0

[humble-codes]

Also I did side by side folder comparison for both good & bad artifacts and found that api.deps.json is what actually differs.
Upon further comparison, saw some noticeable diff under target property of platform_jjz_service_api.deps.json file in healthy artifact and the bad artifact.

• bad artifact

"Microsoft.Data.SqlClient/5.1.5": {
  "dependencies": {
    "Azure.Identity": "1.10.3",
    "Microsoft.Data.SqlClient.SNI.runtime": "5.1.1",
    "Microsoft.Identity.Client": "4.56.0",
    "Microsoft.IdentityModel.JsonWebTokens": "6.35.0",
    "Microsoft.IdentityModel.Protocols.OpenIdConnect": "6.35.0",
    "Microsoft.SqlServer.Server": "1.0.0",
    "System.Configuration.ConfigurationManager": "6.0.1",
    "System.Diagnostics.DiagnosticSource": "6.0.1",
    "System.Runtime.Caching": "6.0.0",
    "System.Security.Cryptography.Cng": "5.0.0",
    "System.Security.Principal.Windows": "5.0.0",
    "System.Text.Encoding.CodePages": "6.0.0",
    "System.Text.Encodings.Web": "8.0.0"
  },
  "runtime": {
    "lib/net6.0/Microsoft.Data.SqlClient.dll": {
      "assemblyVersion": "5.0.0.0",
      "fileVersion": "5.15.24027.2"
    }
  },
  "runtimeTargets": {
    "runtimes/unix/lib/net6.0/Microsoft.Data.SqlClient.dll": {
      "rid": "unix",
      "assetType": "runtime",
      "assemblyVersion": "5.0.0.0",
      "fileVersion": "5.15.24027.2"
    },
    "runtimes/win/lib/net6.0/Microsoft.Data.SqlClient.dll": {
      "rid": "win",
      "assetType": "runtime",
      "assemblyVersion": "5.0.0.0",
      "fileVersion": "5.15.24027.2"
    }
  }
},
"Microsoft.Data.SqlClient.SNI.runtime/5.1.1": {
  "runtimeTargets": {
    "runtimes/win-arm/native/Microsoft.Data.SqlClient.SNI.dll": {
      "rid": "win-arm",
      "assetType": "native",
      "fileVersion": "0.0.0.0"
    },
    "runtimes/win-arm64/native/Microsoft.Data.SqlClient.SNI.dll": {
      "rid": "win-arm64",
      "assetType": "native",
      "fileVersion": "0.0.0.0"
    },
    "runtimes/win-x64/native/Microsoft.Data.SqlClient.SNI.dll": {
      "rid": "win-x64",
      "assetType": "native",
      "fileVersion": "0.0.0.0"
    },
    "runtimes/win-x86/native/Microsoft.Data.SqlClient.SNI.dll": {
      "rid": "win-x86",
      "assetType": "native",
      "fileVersion": "0.0.0.0"
    }
  }
},

• good artifact

"Microsoft.Data.SqlClient/5.1.5": {
  "dependencies": {
    "Azure.Identity": "1.10.3",
    "Microsoft.Data.SqlClient.SNI.runtime": "5.1.1",
    "Microsoft.Identity.Client": "4.56.0",
    "Microsoft.IdentityModel.JsonWebTokens": "6.35.0",
    "Microsoft.IdentityModel.Protocols.OpenIdConnect": "6.35.0",
    "Microsoft.SqlServer.Server": "1.0.0",
    "System.Configuration.ConfigurationManager": "6.0.1",
    "System.Diagnostics.DiagnosticSource": "6.0.1",
    "System.Runtime.Caching": "6.0.0",
    "System.Security.Cryptography.Cng": "5.0.0",
    "System.Security.Principal.Windows": "5.0.0",
    "System.Text.Encoding.CodePages": "6.0.0",
    "System.Text.Encodings.Web": "8.0.0"
  },
  "runtime": {
    "runtimes/unix/lib/net6.0/Microsoft.Data.SqlClient.dll": {
      "assemblyVersion": "5.0.0.0",
      "fileVersion": "5.15.24027.2"
    }
  }
},
"Microsoft.Data.SqlClient.SNI.runtime/5.1.1": {},

[humble-codes]

More clue hidden in deps.json file :
As shared in the post our GitHub CI/CD pipeline uses following commands to publish the project :

dotnet restore
dotnet publish -c Release -r linux-x64 --self-contained false -o ./publish

So the healthy/good artifact's platform_jjz_service_api.deps.json file looks like this at high level:

{
  "runtimeTarget": {
    "name": ".NETCoreApp,Version=v8.0/linux-x64",
    "signature": ""
  },
  "compilationOptions": {},
  "targets": {
    ".NETCoreApp,Version=v8.0": {},
    ".NETCoreApp,Version=v8.0/linux-x64": {
      "platform_jjz_service_api/1.0.0": {
        "dependencies": {
          "Amazon.Lambda.AspNetCoreServer": "9.0.0",
          "Amazon.Lambda.AspNetCoreServer.Hosting": "1.7.0",
          "CsvHelper": "33.1.0",
          "MediatR.Contracts": "2.0.1",
          "Microsoft.AspNetCore.Authentication.JwtBearer": "6.0.16",
          "Microsoft.AspNetCore.Mvc.Versioning": "5.1.0",
          "Microsoft.Data.SqlClient": "5.1.5",
          "Microsoft.Extensions.DependencyInjection": "8.0.0",
          "Newtonsoft.Json": "13.0.3",
          "Swashbuckle.AspNetCore": "6.7.0",
          "platform-jjz-service-Infrastructure": "1.0.0",
          "platform-jjz-service-core": "1.0.0",
          "platform-jjz-service-domain": "1.0.0"
        },
        "runtime": {
          "platform_jjz_service_api.dll": {}
        }
      },
      "Amazon.Lambda.APIGatewayEvents/2.7.0": {
        "runtime": {
          "lib/net8.0/Amazon.Lambda.APIGatewayEvents.dll": {
            "assemblyVersion": "1.0.0.0",
            "fileVersion": "1.2.0.0"
          }
        }
      },
      "other packages ....": {}
    }
  },
  "libraries": {
    "Amazon.Lambda.APIGatewayEvents/2.7.0": {},
    "package02": {},
    "and so on ...": {}
  } 
}

Whereas bad artifact's high level structure looks like this:

{
  "runtimeTarget": {
    "name": ".NETCoreApp,Version=v8.0",
    "signature": ""
  },
  "compilationOptions": {},
  "targets": {
    ".NETCoreApp,Version=v8.0": {
      "platform_jjz_service_api/1.0.0": {
        "dependencies": {
          "Amazon.Lambda.AspNetCoreServer": "9.0.0",
          "Amazon.Lambda.AspNetCoreServer.Hosting": "1.7.0",
          "CsvHelper": "33.1.0",
          "MediatR.Contracts": "2.0.1",
          "Microsoft.AspNetCore.Authentication.JwtBearer": "6.0.16",
          "Microsoft.AspNetCore.Mvc.Versioning": "5.1.0",
          "Microsoft.Data.SqlClient": "5.1.5",
          "Microsoft.Extensions.DependencyInjection": "8.0.0",
          "Newtonsoft.Json": "13.0.3",
          "Swashbuckle.AspNetCore": "6.7.0",
          "platform-jjz-service-Infrastructure": "1.0.0",
          "platform-jjz-service-core": "1.0.0",
          "platform-jjz-service-domain": "1.0.0"
        },
        "runtime": {
          "platform_jjz_service_api.dll": {}
        }
      },
      "Amazon.Lambda.APIGatewayEvents/2.7.0": {
        "runtime": {
          "lib/net8.0/Amazon.Lambda.APIGatewayEvents.dll": {
            "assemblyVersion": "1.0.0.0",
            "fileVersion": "1.2.0.0"
          }
        }
      },
      "other packages ....": {}
    }
  },
  "libraries": {
    "Amazon.Lambda.APIGatewayEvents/2.7.0": {},
    "package02": {},
    "and so on ...": {}
  } 
}

[humble-codes]

@jkotas as per the last message off mine, could this issue be because - some published artifacts (healthy ones) include a RuntimeIdentifier (RID) .NETCoreApp,Version=v8.0/linux-x64 while the bad artifacts do not ?

Any help/insights would be appreciated here !!

[jkotas]

@baronfel