DpapiXmlDecryptor - An exception occurred while trying to decrypt the element.

[vkirienko]

Describe the bug

We have a pretty much the same issue as described in #9447. Two of our .NET Core 3.1.1 applications running in IIS with hostingModel="InProcess" started to throw errors below when app pool gets recycled and application restarts.

It looks like it started to happen right after the following patches were installed on our production server. Unfortunately problem show up only in our production environments even if all environments are kept the same in terms or patches.

---

Category: Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor
EventId: 43

An exception occurred while trying to decrypt the element.

Exception:

System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
   at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapiCore(Byte* pbProtectedData, UInt32 cbProtectedData, Byte* pbOptionalEntropy, UInt32 cbOptionalEntropy)
   at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapi(Byte[] protectedSecret)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor.Decrypt(XElement encryptedElement)

---

Category: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager
EventId: 24

An exception occurred while processing the key element ''.

Exception:

System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
   at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapiCore(Byte* pbProtectedData, UInt32 cbProtectedData, Byte* pbOptionalEntropy, UInt32 cbOptionalEntropy)
   at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapi(Byte[] protectedSecret)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)

---

Category: Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver
EventId: 12

Key {9e609506-dcbd-4191-a937-bc514cf9f564} is ineligible to be the default key because its CreateEncryptor method failed.

Exception:

System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
   at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapiCore(Byte* pbProtectedData, UInt32 cbProtectedData, Byte* pbOptionalEntropy, UInt32 cbOptionalEntropy)
   at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapi(Byte[] protectedSecret)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor()
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key)

---

.NET 3.5
https://support.microsoft.com/en-us/help/4578981/kb4578981 - Security Only Update for .NET Framework 3.5 for Windows 8.1 and Windows Server 2012 R2 (KB4578981)
https://support.microsoft.com/en-us/help/4569737/kb4569737 - Security Only Update for .NET Framework 3.5 for Windows 8.1 and Windows Server 2012 R2 (KB4569737)
https://support.microsoft.com/en-us/help/4565580/kb4565580 - Security Only Update for .NET Framework 3.5 for Windows 8.1 and Windows Server 2012 R2 (KB4565580)
https://support.microsoft.com/en-us/help/4578953/kb4578953 - Security and Quality Rollup for .NET Framework 3.5 for Windows 8.1, RT 8.1, and Windows Server 2012 R2 (KB4578953)

.NET 4.8
https://support.microsoft.com/en-us/help/4576489/kb4576489 - Security Only Update for .NET Framework 4.8 for Windows 8.1 and Windows Server 2012 R2 (KB4576489)
https://support.microsoft.com/en-us/help/4565588/kb4565588 - Security Only Update for .NET Framework 4.8 for Windows 8.1 and Windows Server 2012 R2 (KB4565588)
https://support.microsoft.com/en-us/help/4569732/kb4569732 - Security Only Update for .NET Framework 4.8 for Windows 8.1 and Windows Server 2012 R2 (KB4569732)
https://support.microsoft.com/en-us/help/4578989/kb4578989 - Security Only Update for .NET Framework 4.8 for Windows 8.1 and Windows Server 2012 R2 (KB4578989)
https://support.microsoft.com/en-us/help/4578976/kb4578976 - Security and Quality Rollup for .NET Framework 4.8 for Windows 8.1, RT 8.1, and Windows Server 2012 R2 (KB4578976)

Further technical details

• ASP.NET Core version

3.1.1

• Include the output of dotnet --info

It was not possible to find any installed .NET Core SDKs
Did you mean to run .NET Core SDK commands? Install a .NET Core SDK from:
https://aka.ms/dotnet-download

Host (useful for support):
Version: 3.1.1
Commit: a1388f194c

.NET Core SDKs installed:
No SDKs were found.

.NET Core runtimes installed:
Microsoft.AspNetCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNe
tCore.App]
Microsoft.NETCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.
App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[blowdart]

@GrabYourPitchforks any ideas what might be going on?

[vkirienko]

What's interesting that there is no pattern related to number of errors per app pool recycle or what keys have problem.

[mkArtakMSFT]

Ping @GrabYourPitchforks

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[GrabYourPitchforks]

These errors are coming from the Win32 API CryptUnprotectData. I wonder if the IIS application pool worker identity has been reset or if the application pool is now configured not to load the user profile? (If there's no user profile, there's no place for DPAPI to store the key material.)

For historical context, in a traditional Full Framework ASPNET app, the expectation is that the IIS worker process does not load the user profile. Instead, the IIS orchestration service detects that the app pool is configured to load aspnet, it carves out a special section of the registry to allow aspnet to hold its key material, and System.Web uses that registry key instead of the user profile to store the auto-generated <machineKey> element.

[vkirienko]

You guess is correct - application pool is configured not to load the user profile. But I was using script below to provision the HKLM registry so that the specified user account can persist auto-generated machine keys.

https://github.com/dotnet/aspnetcore/blob/main/src/DataProtection/Provision-AutoGenKeys.ps1

But it strange that these errors showed up just recently.

[GrabYourPitchforks]

@vkirienko Did the identity of the application pool change recently? For instance, switching from NETWORK SERVICE to APP POOL IDENTITY (or vice versa)? If the app pool user account is set to "app pool identity", the identity is based on the name of the application pool. In that case, if the name of the app pool changes, the provisioning script will need to be rerun.

[vkirienko]

No, it did not change recently. It was our application service account from day one. Application pool name did not change as well.

Re-running provisioning script does not help. It was the first thing that I tried when we got these errors.

[GrabYourPitchforks]

The DataProtection startup routine attempts to detect capabilities of the current hosting environment at startup (see code) and will choose an appropriate encryption mechanism based on that. It will log something like "Using the user profile with DPAPI", "Using the user profile without DPAPI", or "Using registry path [blah] for key storage".

Can you let us know which of these it's writing to the log? You may have to turn on informational or verbose logging to see these messages.

[vkirienko]

Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: 2021-01-25 15:59:40.3371 INFO User profile is available. Using 'C:\Windows\system32\config\systemprofile\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository: 2021-01-25 15:59:40.3464 DEBUG Reading data from file 'C:\Windows\system32\config\systemprofile\AppData\Local\ASP.NET\DataProtection-Keys\key-0d8f0300-c301-48a7-bf94-b97d44f653d7.xml'.
... skip 50 similar lines
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: 2021-01-25 15:59:40.3795 DEBUG Found key {0d8f0300-c301-48a7-bf94-b97d44f653d7}.
... skip 50 similar lines
Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver: 2021-01-25 15:59:40.4092 DEBUG Considering key {8bc09e78-829b-4211-a4ed-e00d04e9469b} with expiration date 2021-04-25 20:58:35Z as default key.
Microsoft.AspNetCore.DataProtection.TypeForwardingActivator: 2021-01-25 15:59:40.4092 DEBUG Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor: 2021-01-25 15:59:40.4092 DEBUG Decrypting secret element using Windows DPAPI.
Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor: 2021-01-25 15:59:40.4424 ERROR An exception occurred while trying to decrypt the element.
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: 2021-01-25 15:59:40.6070 ERROR An exception occurred while processing the key element '<key id="8bc09e78-829b-4211-a4ed-e00d04e9469b" version="1" />'.
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: 2021-01-25 15:59:40.6176 TRACE An exception occurred while processing the key element '<key id="8bc09e78-829b-4211-a4ed-e00d04e9469b" version="1">
  <creationDate>2021-01-25T20:58:35.9194222Z</creationDate>
  <activationDate>2021-01-25T20:58:35.6384352Z</activationDate>
  <expirationDate>2021-04-25T20:58:35.6384352Z</expirationDate>
  <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
    <descriptor>
      <encryption algorithm="AES_256_CBC" />
      <validation algorithm="HMACSHA256" />
      <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection">
        <encryptedKey xmlns="">
          <!-- This key is encrypted with Windows DPAPI. -->
          <value>...del...</value>
        </encryptedKey>
      </encryptedSecret>
    </descriptor>
  </descriptor>
</key>'.
Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver: 2021-01-25 15:59:40.6176 WARN Key {8bc09e78-829b-4211-a4ed-e00d04e9469b} is ineligible to be the default key because its CreateEncryptor method failed.

[GrabYourPitchforks]

Is your app pool configured to run as SYSTEM? And was this intentional? I don't see why aspnet would be trying to load the %SYSTEM32%\config\systemprofile\ directory otherwise.

[vkirienko]

App pools configured under service accounts. WWW service runs under Local System account as usual.

As far as can see systemprofile folder was used all this time (keys were expiring every 3 months) and then suddenly on 12/12/2020 keys are created every day when app pool recycle happens.

I need to check with our server team to see what is so special about these service accounts.