Skip to content

DLLs can be missing after update to newer .NET patches #42445

New issue
New issue
Open
Open
DLLs can be missing after update to newer .NET patches#42445
Labels
area-HostuntriagedRequest triage from a team member
@johnearnshaw

Description

@johnearnshaw
johnearnshaw
opened on Oct 26, 2023

I run a long-running .NET 6 application in a remote Ubuntu server that generally runs fine for long periods of time (Currently tested to 226 days).

Recently the application became unresponsive and looking into the applications error logging, it was throwing the following Exception while we were requesting data from it:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

Restating the application fixed the issue so it's not a major problem, but just thought I'd bring this to your attention as this is the only reference of a similar issue I can find online. The full Exception details I have available can be seen below.

As I say, this is not a major problem but I would prefer my application to run longer without any issue, I thought it quite strange that the application starts and runs long-term without any issues and, after a long while, decides System.Runtime.Serialization.Formatters assembly can't be found. Though a following restart of the application fixes the issue.

It's a strange one!? Any suggestion why this may have occurred would be helpful even if solely for informational purpose...

`2023-10-23-13:36:59.498 Modules.Dashboard.DashboardEventPlaygroundModule
Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback.
getEventOutput
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type)
at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates)
at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates)
at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates)
at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization)
at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception:

2023-10-23-13:37:43.263 Modules.Dashboard.DashboardEventPlaygroundModule
Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback.
getEventOutput
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type)
at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates)
at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates)
at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates)
at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization)
at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception:`

Activity

vitek-karas
transferred this issue fromdotnet/linkeron Oct 27, 2023
ghost

ghost commented on Oct 29, 2023

@ghost
ghost
on Oct 29, 2023

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

I run a long-running .NET 6 application in a remote Ubuntu server that generally runs fine for long periods of time (Currently tested to 226 days).

Recently the application became unresponsive and looking into the applications error logging, it was throwing the following Exception while we were requesting data from it:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

Restating the application fixed the issue so it's not a major problem, but just thought I'd bring this to your attention as this is the only reference of a similar issue I can find online. The full Exception details I have available can be seen below.

As I say, this is not a major problem but I would prefer my application to run longer without any issue, I thought it quite strange that the application starts and runs long-term without any issues and, after a long while, decides System.Runtime.Serialization.Formatters assembly can't be found. Though a following restart of the application fixes the issue.

It's a strange one!? Any suggestion why this may have occurred would be helpful even if solely for informational purpose...

`2023-10-23-13:36:59.498 Modules.Dashboard.DashboardEventPlaygroundModule
Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback.
getEventOutput
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type)
at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates)
at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates)
at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates)
at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization)
at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception:

2023-10-23-13:37:43.263 Modules.Dashboard.DashboardEventPlaygroundModule
Modules.Dashboard.DashboardEventPlaygroundModule threw an exception in callback.
getEventOutput
System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

File name: 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at NetTools.Serialization.ReflectionUtils.GetInstanceOf(Type type)
at NetTools.Serialization.JsonTools.JsonReader.ParseObject(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates) at NetTools.Serialization.JsonTools.JsonReader.ParseValue(Type type, String json, StringSerializerFactory serializerFactory, StringBuilder stringBuilder, Stack1 splitArrayPool, Boolean includePrivates)
at NetTools.Serialization.JsonTools.JsonReader.FromJson(Type t, String json, StringSerializerFactory serializerFactory, Boolean includePrivates)
at NetTools.Serialization.Json.ObjectFromJson(Object this, String jsonString, Boolean includePrivates)
at Modules.Dashboard.EventPlayground.Classes.AjaxCallbacks.<>c__DisplayClass2_0.b__0(String[] path, DataRequestEventArgs request, String session, String authorization)
at Modules.WebServer.DashboardModule.DashboardModule.ProcessAjaxRequest(String[] path, DataRequestEventArgs request, String session, String authorization), Inner exception:`

Author: johnearnshaw
Assignees: -
Labels:

area-System.Runtime, untriaged, needs-area-label
Milestone: -
danmoseley

danmoseley commented on Oct 29, 2023

@danmoseley
danmoseley
on Oct 29, 2023
Member

Is it possible to compare 8.0? (But perhaps it would take months to repro again..)
Does the code at the stack trace get exercised successfully in normal operation, or is the failure when it first gets exercised?
Is the app deployed self contained or using shared framework?
Are there any ApplicationLoadContexts in use?
Is it trimmed?
Of course, any way to get this to repro on demand would be helpful.

moellerm

moellerm commented on Oct 30, 2023

@moellerm
moellerm
on Oct 30, 2023

You could check if an update to dotnet (.NET 6.0 runtime) has been installed recently or if rolling back dotnet, restarting the application and then updating dotnet reproduces the issue. See dotnet/runtime#60144 (comment) for a similar case. Obviously without further detail, there could be many other reasons.

LukeDearden

LukeDearden commented on Nov 15, 2023

@LukeDearden
LukeDearden
on Nov 15, 2023

while installing updates on an Ubuntu 22.04.3 VM I noticed that the dotnet update from https://packages.microsoft.com/ubuntu/22.04/prod repo actually triggered a restart of our dotnet application service

`sudo apt upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
apt apt-utils libapt-pkg6.0
The following packages will be upgraded:
aspnetcore-runtime-6.0 dotnet-host dotnet-hostfxr-6.0 dotnet-runtime-6.0 dotnet-runtime-deps-6.0 libprocps8 procps
7 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
2 standard LTS security updates
Need to get 30.0 MB of archives.
After this operation, 50.2 kB disk space will be freed.
Get:1 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libprocps8 amd64 2:3.3.17-6ubuntu2.1 [36.2 kB]
Get:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 procps amd64 2:3.3.17-6ubuntu2.1 [378 kB]
Get:3 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-host amd64 8.0.0-1 [37.7 kB]
Get:4 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-hostfxr-6.0 amd64 6.0.25-1 [144 kB]
Get:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-runtime-deps-6.0 amd64 6.0.25-1 [2800 B]
Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 dotnet-runtime-6.0 amd64 6.0.25-1 [22.8 MB]
Get:7 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 aspnetcore-runtime-6.0 amd64 6.0.25-1 [6621 kB]
Fetched 30.0 MB in 0s (120 MB/s)
(Reading database ... 64881 files and directories currently installed.)
Preparing to unpack .../0-libprocps8_2%3a3.3.17-6ubuntu2.1_amd64.deb ...
Unpacking libprocps8:amd64 (2:3.3.17-6ubuntu2.1) over (2:3.3.17-6ubuntu2) ...
Preparing to unpack .../1-procps_2%3a3.3.17-6ubuntu2.1_amd64.deb ...
Unpacking procps (2:3.3.17-6ubuntu2.1) over (2:3.3.17-6ubuntu2) ...
Preparing to unpack .../2-dotnet-host_8.0.0-1_amd64.deb ...
Unpacking dotnet-host (8.0.0-1) over (7.0.13-1) ...
Preparing to unpack .../3-dotnet-hostfxr-6.0_6.0.25-1_amd64.deb ...
Unpacking dotnet-hostfxr-6.0 (6.0.25-1) over (6.0.24-1) ...
Preparing to unpack .../4-dotnet-runtime-deps-6.0_6.0.25-1_amd64.deb ...
Unpacking dotnet-runtime-deps-6.0 (6.0.25-1) over (6.0.24-1) ...
Preparing to unpack .../5-dotnet-runtime-6.0_6.0.25-1_amd64.deb ...
Unpacking dotnet-runtime-6.0 (6.0.25-1) over (6.0.24-1) ...
Preparing to unpack .../6-aspnetcore-runtime-6.0_6.0.25-1_amd64.deb ...
Unpacking aspnetcore-runtime-6.0 (6.0.25-1) over (6.0.24-1) ...
Setting up dotnet-host (8.0.0-1) ...
Setting up dotnet-runtime-deps-6.0 (6.0.25-1) ...
Setting up dotnet-hostfxr-6.0 (6.0.25-1) ...
Setting up libprocps8:amd64 (2:3.3.17-6ubuntu2.1) ...
Setting up dotnet-runtime-6.0 (6.0.25-1) ...
Setting up aspnetcore-runtime-6.0 (6.0.25-1) ...
Setting up procps (2:3.3.17-6ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.4) ...
Scanning processes...
Scanning candidates...
Scanning linux images...

Running kernel seems to be up-to-date.

Restarting services...
systemctl restart frends-agent.service`

I checked this out on an older Ubuntu 20.04.6 VM but updating to the same dotnet version from repo https://packages.microsoft.com/ubuntu/20.04/prod did not trigger a service restart

dotnet --list-runtimes Microsoft.AspNetCore.App 6.0.25 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.25 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

arcoba

arcoba commented on Feb 15, 2024

@arcoba
arcoba
on Feb 15, 2024

Hi,
I've the same issue with my application that start every day at 7.00 AM.
It's a console app that running on Ubuntu 22.04 LTS using Microsoft.AspNetCore.App 6.0.27 and Microsoft.NETCore.App 6.0.27 runtimes. The application is framework dependent and deployed as single x64 executable file.

Usually work well, but only sometimes when Newtonsoft.Json.JsonSerializer (13.0.3) try to load System.Runtime.Serialization.Formatters it throw the exception System.IO.FileNotFoundException with message:

"Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified."

I think that this kind of problem was not present until runtime version 6.0.8; is it possible?
Do you need any logs or additional information?

Fad.

Sacha-K

Sacha-K commented on Apr 8, 2024

@Sacha-K
Sacha-K
on Apr 8, 2024 · edited by Sacha-K

I saw this one on Windows.

Configuration:

  • net6.0-windows website running on 6.0.28
  • I have a BackgroundService that calls a "data export" method that sits in a netstandard2.0 DLL
  • That DLL has a dependency on Newtonsoft.Json 13.0.3 and it's needed for this method

What happened:

  • Website running on 6.0.27
  • Website stopped and updated to 6.0.28
  • The first time the background service called my method after the 6.0.28 update I got the exception System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
  • No issue after that

In March my method was called about 13k times with website running on version 6.0.27 with no error, failed on the first call after the 6.0.28 update, was called about 8k times with website running on 6.0.28 with no error.

Edit:

I went and read dotnet/runtime#60144 (opened for 2.5 years) and according to comment dotnet/runtime#60144 (comment) the .NET update forces a restart of websites at a point where it is dotnet 6.0.28 and aspnet-runtime 6.0.27, just after that you get aspnet-runtime 6.0.28 installed and aspnet-runtimes 6.0.27 removed causing the exception.

So either I just accept the single crash after updates, or I "manually" stop website / update / start website instead of letting WU do the updates. 😕

tannergooding
removed
untriagedRequest triage from a team member
on Jun 24, 2024
jeffhandley
added
area-Host
on Jul 27, 2024
dotnet-policy-service

dotnet-policy-service commented on Jul 27, 2024

@dotnet-policy-service
dotnet-policy-servicebot
on Jul 27, 2024
Contributor

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

jeffhandley

jeffhandley commented on Jul 27, 2024

@jeffhandley
jeffhandley
on Jul 27, 2024
Member

@agocke This seems more like a Host type issue. What are the expectations for long-running applications where the application is still running when a runtime update is installed?

agocke

agocke commented on Jul 29, 2024

@agocke
agocke
on Jul 29, 2024
Member

These symptoms look exactly like what could currently happen during an update (especially the comment about the app crashing at exactly one specific time, when the computer might be scheduled to install updates).

I don't think this is a problem with assembly loading, I think it's a problem with the older framework being uninstalled while the latest patch is being installed. I'm going to move this over to the SDK to further analyze the issue.

agocke
changed the title [-]System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime.Serialization.Formatters.[/-] [+]DLLs can be missing after update to newer .NET patches[/+] on Jul 29, 2024
ghost added
untriagedRequest triage from a team member
on Jul 29, 2024
agocke
transferred this issue fromdotnet/runtimeon Jul 29, 2024
FlsZen

FlsZen commented on Aug 16, 2024

@FlsZen
FlsZen
on Aug 16, 2024

We've been experiencing issues similar to this seemingly when Windows Update applies a security path. It appears to delete the .NET 6.0.x runtime out from under our running WPF app and relocates the already-loaded assemblies into C:\Config.Msi. I received reports when .NET 6.0.31 was released and again now when 6.0.33 was released.

The issue is specifically that the user performs an operation in the application that they hadn't performed yet (during the process's lifetime) and the runtime needs to load another DLL, but can't find it. For example:

Type: System.IO.FileNotFoundException
Message: Could not load file or assembly 'Accessibility, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
Source: PresentationFramework
Stack Trace:
   at System.Windows.Controls.Primitives.Popup.PopupSecurityHelper.ForceMsaaToUiaBridge(PopupRoot popupRoot)
   at System.Windows.Controls.Primitives.Popup.CreateWindow(Boolean asyncCall)
   at System.Windows.Controls.Primitives.Popup.OnIsOpenChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
   at System.Windows.DependencyObject.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
...

and

Could not load file or assembly 'System.IO.Compression.ZipFile, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The system cannot find the file specified.

I like the concept of running the latest security patch delivered by Windows Update, but is there anything we can do to prevent it from pulling the rug out from under our app? At this point I feel like the best way forward to avoid our software appearing unstable to our customers is to deploy the .NET runtime to a custom folder and run from there.

LukeDearden
mentioned this on Nov 14, 2024
joeloff

joeloff commented on Nov 14, 2024

@joeloff
joeloff
on Nov 14, 2024
Member

@FlsZen please take a look at the new option we made available in November. Perhaps these can help address some of the issues you've encountered. By deferring the removal until the next restart, you'll have more control and hopefully reduce crash incidents.

The problems described isn't isolated to the SDK. Unlike .NET Framework that services as part of the OS using CBS that has access to special kernel transactions to deal with in-use files, .NET is serviced through MU. There are a number of factors that impact what happens with running applications and services.

  1. The installer service and restart manager interact with each other. That interaction can be impacted by things like machine policies that will fall back to files-in-use logic that predates Restart Manager / Vista. There are additional nuances, for example, the installer service will do special checks to see whether individual MSIs include the required files-in-use dialog set and if not, modify the actions it takes. This can impact things like the message sent back by the installer service to the installer.
  2. The SIDs associated with the running application/service and the installation. Updates typically run under SYSTEM, but applications do not. Services may run under system, but sometimes they run under dedicated accounts. RM also takes into account the type of application and sessions. It won't stop/restart applications running in a different session and won't touch things when it doesn't have permissions. Critical services are never stopped/restarted.
  3. Applications may or may not have registered with RM for restarts. Even if they did register, if the SIDs don't line up, RM will report restart failures. You'd typically see events similar to "The application SID does not match the conductor SID". Registered applications can provide proper command lines one how to initiate a restart. The default behavior when using something like dotnet.exe to launch an application will not restart properly because RM only detects the host and not the original command line so it ends up executing dotnet.exe on a restart which is effectively a no-op.
  4. Each MSI creates and ends a session with RM. A .NET installation like the runtime consists of multiple MSIs. This means that at the end of each MSI it may try to restart applications only to shut it down immediately when the next MSI executes. The documentation is unclear whether this is asynchronous or whether the installer service waits for RM to complete all the restarts.
  5. Whether or not the .NET update runs with a UI. Messages from the installer services are directed back to the installer engine that may surface additional UI to choose between closing/restarting applications or continue running and restart the machine. When running quietly as is the case in MU, there are defaults that would be selected.
VAllens

VAllens commented on Apr 10, 2025

@VAllens
VAllens
on Apr 10, 2025

No progress at all?
I'm having the same problem with .NET 9.0.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-HostuntriagedRequest triage from a team member

    Type

    No type

    Projects

    AppModel

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @agocke@jeffhandley@VAllens@FlsZen@danmoseley

        Issue actions
          DLLs can be missing after update to newer .NET patches · Issue #42445 · dotnet/sdk