[API Proposal]: Support of secured value decryption configuration options

[HHobeck]

Background and motivation

Dear community,

I came up with the idea to extend the Microsoft.Extensions.Options model to support a secured value decryption of dedicated configuration properties marked with SecuredValueAttribute. Handling with sensitive information in application settings is very difficult and error prone. Because every API developer and third party library authors knows what sensitive information are it is very easy to mark it with deserving protection.

The idea is to encrypt the sensitive information in the application settings with e.g. IDataProtector on design time (via IDE would be awesome) and decrypt it when the application started. This has the advantage that you can check in you configuration without to be afraid that the sensitive information are in the repository.

The idea would be to have a provider based approach and supporting other security provider for AWS, MS Azure and so on.

Thank you for reading.

API Proposal

using System;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace Microsoft.Extensions.Options;

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Property)]
public class SecuredValueAttribute : Attribute
{
}

public static class OptionsWithSecuredValueDecryptionComposition
{
    public static void Register(IServiceCollection serviceCollection)
    {
        if (serviceCollection is null) throw new ArgumentNullException(nameof(serviceCollection));

        serviceCollection.AddSingleton(typeof(IPostConfigureOptions<>), typeof(SecuredValueDecryptionConfigureOptions<>));
    }
}

public sealed class SecuredValueDecryptionConfigureOptions<TOptions> : IPostConfigureOptions<TOptions> where TOptions : class
{
    private readonly IHostEnvironment _hostEnvironment;
    private readonly Lazy<IDataProtector> _dataProtectorLazy;

    public SecuredValueDecryptionConfigureOptions(IHostEnvironment hostEnvironment, IServiceProvider serviceProvider)
    {
        _hostEnvironment = hostEnvironment ?? throw new ArgumentNullException(nameof(hostEnvironment));
        if (serviceProvider is null) throw new ArgumentNullException(nameof(serviceProvider));

        _dataProtectorLazy = new Lazy<IDataProtector>(() =>
            serviceProvider.GetRequiredService<IDataProtectionProvider>().CreateProtector("SecuredValueDecryption")
        );
    }

    public void PostConfigure(string? name, TOptions options)
    {
        if (_hostEnvironment.IsDevelopment()) return;

        // Detect encrypted configuration properties and decrypt them here if the environment is not Development.
    }
}

API Usage

internal class Program
{
    static void Main(string[] arguments)
    {
        IHostBuilder hostBuilder = Host.CreateDefaultBuilder(arguments);
        hostBuilder.ConfigureServices((hostBuilderContext, serviceCollection) =>
        {
            OptionsWithSecuredValueDecryptionComposition.Register(serviceCollection);
            serviceCollection.Configure<ServiceSettings>(
                hostBuilderContext.Configuration.GetSection(typeof(ServiceSettings).FullName!)
            );
        });

        using IHost host = hostBuilder.Build();
        host.Run();
    }
}

[SecuredValue]
public class ServiceSettings
{
    public string UserName { get; set; } = string.Empty;

    [SecuredValue]
    public string Password { get; set; } = string.Empty;
}

Alternative Designs

No response

Risks

This is additive and every developer can decide to use it or not.

Tagging subscribers to this area: @dotnet/area-extensions-hosting
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Dear community,

I came up with the idea to extend the Microsoft.Extensions.Options model to support a secured value decryption of dedicated configuration properties marked with SecuredValueAttribute. Handling with sensitive information in application settings is very difficult and error prone. Because every API developer and third party library authors knows what sensitive information are it is very easy to mark it with deserving protection.

The idea is to encrypt the sensitive information in the application settings with e.g. IDataProtector on design time (via IDE would be awesome) and decrypt it when the application started. This has the advantage that you can check in you configuration without to be afraid that the sensitive information are in the repository.

The idea would be to have a provider based approach and supporting other security provider for AWS, MS Azure and so on.

Thank you for reading.

API Proposal

using System;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace Microsoft.Extensions.Options;

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Property)]
public class SecuredValueAttribute : Attribute
{
}

public static class OptionsWithSecuredValueDecryptionComposition
{
    public static void Register(IServiceCollection serviceCollection)
    {
        if (serviceCollection is null) throw new ArgumentNullException(nameof(serviceCollection));

        serviceCollection.AddSingleton(typeof(IPostConfigureOptions<>), typeof(SecuredValueDecryptionConfigureOptions<>));
    }
}

public sealed class SecuredValueDecryptionConfigureOptions<TOptions> : IPostConfigureOptions<TOptions> where TOptions : class
{
    private readonly IHostEnvironment _hostEnvironment;
    private readonly Lazy<IDataProtector> _dataProtectorLazy;

    public SecuredValueDecryptionConfigureOptions(IHostEnvironment hostEnvironment, IServiceProvider serviceProvider)
    {
        _hostEnvironment = hostEnvironment ?? throw new ArgumentNullException(nameof(hostEnvironment));
        if (serviceProvider is null) throw new ArgumentNullException(nameof(serviceProvider));

        _dataProtectorLazy = new Lazy<IDataProtector>(() =>
            serviceProvider.GetRequiredService<IDataProtectionProvider>().CreateProtector("SecuredValueDecryption")
        );
    }

    public void PostConfigure(string? name, TOptions options)
    {
        // Detect encrypted configuration properties and decrypt them here or get it from
        // AWS or MS Azure if the environment is not Development.
    }
}

API Usage

internal class Program
{
    static void Main(string[] arguments)
    {
        IHostBuilder hostBuilder = Host.CreateDefaultBuilder(arguments);
        hostBuilder.ConfigureServices(
            serviceCollection => OptionsWithSecuredValueDecryptionComposition.Register(serviceCollection)
        );

        using IHost host = hostBuilder.Build();
        host.Run();
    }
}

Alternative Designs

No response

Risks

This is additive and every developer can decide to use it or not.

Author:	HHobeck
Assignees:	-
Labels:	api-suggestion, area-Extensions-Hosting
Milestone:	-

[Clockwork-Muse]

Handling with sensitive information in application settings is very difficult and error prone.

So difficult and error prone that SecureString was removed.
The underlying problem is that protecting your own memory is very difficult, especially once you consider that managed memory systems tend to keep copies just laying around.

The idea is to encrypt the sensitive information in the application settings with e.g. IDataProtector on design time (via IDE would be awesome) and decrypt it when the application started. This has the advantage that you can check in you configuration without to be afraid that the sensitive information are in the repository.

.... if you've encrypted the setting, where are you getting the decryption key from? You can't put that in the repository. You have to put it somewhere secure, like an environment variable or key vault on the deployment target.
And the moment you have to put the decryption key somewhere secure, you may as well put the actual setting in the same secure storage.

Additionally, even if you can put a setting into the source repository, at scale you generally want settings to come from setup/deployment scripts referencing other identities. For example, if you're thinking of things like passwords for databases, rather than storing the database password in the source repository it should be retrieved from the deployed database as part of deployment.

[HHobeck]

Hi Stephen.

Thanks for your thoughts. I would like to reply to this as following:

The underlying problem is that protecting your own memory is very difficult, especially once you consider that managed memory systems tend to keep copies just laying around.

This feature is not about protecting sensitive information in memory it is about how to inject it into the configuration system. As long as you are not concatenating the string you have the sensitive information exactly one time in memory. But this problem you are describing here remains independent of that if you inject the sensitive information via pipeline on deployment time or put it on an environment variable or get it on startup from a security vault on runtime. That means at the end if an attacker has access to your memory then you have other problems.

.... if you've encrypted the setting, where are you getting the decryption key from? You can't put that in the repository. You have to put it somewhere secure, like an environment variable or key vault on the deployment target.
And the moment you have to put the decryption key somewhere secure, you may as well put the actual setting in the same secure storage.

Yes maybe you are right and the solution you are describing gives a maximum value from the security point of view. But you need to consider that not all applications are running in a high security environment. You can think of to have a solution between password in clear text and password injected in the deployment pipeline. In most cases the developer wants to test the application with NonProduction or even with the Production settings locally. That means maybe in 3 of 5 projects there will be other solutions implemented. One solution can be to determine the sensitive information from e.g. AWS Secrets Manager or Azure Key Vault or like I mentioned to just decrypt it on runtime.

To answer your question: The DataProtectionAPI can be used to e.g. use the private key which is located on a dedicated folder where only user have access who are allowed to see the information.

Additionally, even if you can put a setting into the source repository, at scale you generally want settings to come from setup/deployment scripts referencing other identities. For example, if you're thinking of things like passwords for databases, rather than storing the database password in the source repository it should be retrieved from the deployed database as part of deployment.

I agree if you think of a rolling password management then this feature would be not fitting.

[Clockwork-Muse]

As long as you are not concatenating the string you have the sensitive information exactly one time in memory.

This is false.
The memory manager is free to make copies of a string at any time, to enable things like heap compaction.
Depending on how your parse/decrypt is implemented, you'll have partial or whole copies of the raw data in byte arrays.

But you need to consider that not all applications are running in a high security environment.

This isn't high-security stuff. This should be standard security stuff.

In most cases the developer wants to test the application with NonProduction or even with the Production settings locally.

In almost no case should a developer have access to Production secrets, especially anything that would enable read/write access to resources. Ideally, developers should be able to run all unit tests and the majority of base integration tests locally, without a connection to cloud resources. Even when they do need to access remote resources, the frameworks they use should enable them to connect with developer-specific credentials available by the cloud platform. For example, with Azure you can log in to the CLI which gets you a token, and then things like remote database connections use your tokenized credentials instead of what would normally be tokenized credentials for the actual host service.

For the most part, the need for secret variables should be something we actively discourage. The majority of their use is for managing credentials, which causes a number of problems (eg, rolling expired credentials, leaks). Which is why the push is to update frameworks to use token identities, and allow for automatic use by developer tools. Many of the more popular frameworks already work this way (eg, ASP.NET, Spring Boot, etc)

To answer your question: The DataProtectionAPI can be used to e.g. use the private key which is located on a dedicated folder where only user have access who are allowed to see the information.

If you mean using this to manage a private key, no. First, production private keys shouldn't be in the repository simply from a management perspective (never mind the security reasons). Even non-prod keys can be dicey. For local development many frameworks have fallback configs that should be used instead.

[HHobeck]

This is false. The memory manager is free to make copies of a string at any time, to enable things like heap compaction. Depending on how your parse/decrypt is implemented, you'll have partial or whole copies of the raw data in byte arrays.

Okay for sure this is operation system optimisation stuff which I have no insides. I can just say that from the language point of view strings are object types and just the address pointers are stored on the stack and will be copied. Just curious: Can you tell me where the different is of your proposal when injecting the sensitive information on deployment time or get it from a security provider? Is it not stored in memory?

This isn't high-security stuff. This should be standard security stuff.

Okay fair enough to hold the security high. But please consider that not all applications are the same and running in the cloud,

In almost no case should a developer have access to Production secrets, especially anything that would enable read/write access to resources. Ideally, developers should be able to run all unit tests and the majority of base integration tests locally, without a connection to cloud resources. Even when they do need to access remote resources, the frameworks they use should enable them to connect with developer-specific credentials available by the cloud platform. For example, with Azure you can log in to the CLI which gets you a token, and then things like remote database connections use your tokenized credentials instead of what would normally be tokenized credentials for the actual host service.

Of course fair point that the developers should have no access to the production environment. But on the non-production environment they should be able to access ;). You can see my proposal also as additive: You can encrypt just the sensitive information for the NonProduction environment and use another mechanism on Production system or even do both and put only encrypted values in the security provider (in the sense of defense in depth).

For the most part, the need for secret variables should be something we actively discourage. The majority of their use is for managing credentials, which causes a number of problems (eg, rolling expired credentials, leaks). Which is why the push is to update frameworks to use token identities, and allow for automatic use by developer tools. Many of the more popular frameworks already work this way (eg, ASP.NET, Spring Boot, etc)

Can you please explain this in more detail? I'm curios and open for any other solutions. How would you provide the credentials to the application to e.g. connect to the database? You mean not to inject it in the configuration system via environment variable or mounted a secret file right? Does it mean you request the credentials from the security provider each time you are establishing a database connection for instance?

If you mean using this to manage a private key, no. First, production private keys shouldn't be in the repository simply from a management perspective (never mind the security reasons). Even non-prod keys can be dicey. For local development many frameworks have fallback configs that should be used instead.

I didn't say to put it in the repository. You are right this folder should not be under version control. I would like to point out: The interface which will be used here is the DataProtectionAPI from MS which was build for decryption and encryption. You could use a hardware dongle where you are not able to extract the private key if you like.

Tagging subscribers to this area: @dotnet/area-extensions-options
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Dear community,

I came up with the idea to extend the Microsoft.Extensions.Options model to support a secured value decryption of dedicated configuration properties marked with SecuredValueAttribute. Handling with sensitive information in application settings is very difficult and error prone. Because every API developer and third party library authors knows what sensitive information are it is very easy to mark it with deserving protection.

The idea is to encrypt the sensitive information in the application settings with e.g. IDataProtector on design time (via IDE would be awesome) and decrypt it when the application started. This has the advantage that you can check in you configuration without to be afraid that the sensitive information are in the repository.

The idea would be to have a provider based approach and supporting other security provider for AWS, MS Azure and so on.

Thank you for reading.

API Proposal

using System;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace Microsoft.Extensions.Options;

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Property)]
public class SecuredValueAttribute : Attribute
{
}

public static class OptionsWithSecuredValueDecryptionComposition
{
    public static void Register(IServiceCollection serviceCollection)
    {
        if (serviceCollection is null) throw new ArgumentNullException(nameof(serviceCollection));

        serviceCollection.AddSingleton(typeof(IPostConfigureOptions<>), typeof(SecuredValueDecryptionConfigureOptions<>));
    }
}

public sealed class SecuredValueDecryptionConfigureOptions<TOptions> : IPostConfigureOptions<TOptions> where TOptions : class
{
    private readonly IHostEnvironment _hostEnvironment;
    private readonly Lazy<IDataProtector> _dataProtectorLazy;

    public SecuredValueDecryptionConfigureOptions(IHostEnvironment hostEnvironment, IServiceProvider serviceProvider)
    {
        _hostEnvironment = hostEnvironment ?? throw new ArgumentNullException(nameof(hostEnvironment));
        if (serviceProvider is null) throw new ArgumentNullException(nameof(serviceProvider));

        _dataProtectorLazy = new Lazy<IDataProtector>(() =>
            serviceProvider.GetRequiredService<IDataProtectionProvider>().CreateProtector("SecuredValueDecryption")
        );
    }

    public void PostConfigure(string? name, TOptions options)
    {
        if (_hostEnvironment.IsDevelopment()) return;

        // Detect encrypted configuration properties and decrypt them here if the environment is not Development.
    }
}

API Usage

internal class Program
{
    static void Main(string[] arguments)
    {
        IHostBuilder hostBuilder = Host.CreateDefaultBuilder(arguments);
        hostBuilder.ConfigureServices((hostBuilderContext, serviceCollection) =>
        {
            OptionsWithSecuredValueDecryptionComposition.Register(serviceCollection);
            serviceCollection.Configure<ServiceSettings>(
                hostBuilderContext.Configuration.GetSection(typeof(ServiceSettings).FullName!)
            );
        });

        using IHost host = hostBuilder.Build();
        host.Run();
    }
}

[SecuredValue]
public class ServiceSettings
{
    public string UserName { get; set; } = string.Empty;

    [SecuredValue]
    public string Password { get; set; } = string.Empty;
}

Alternative Designs

No response

Risks

This is additive and every developer can decide to use it or not.

Author:	HHobeck
Assignees:	-
Labels:	api-suggestion, untriaged, area-Extensions-Options
Milestone:	-

[Clockwork-Muse]

Okay for sure this is operation system optimication stuff which I have no insides. I can just say that from the language point of view string are object types and just the address pointers are stored on the stack and will be copied. Just curious: Can you tell me where the different is of your proposal when injecting the sensitive information on deployment time or get it from a security provider? Is it not stored in memory?

No, this is C# runtime stuff. It's not the address of the string you have to worry about, the actual memory contents of the string (on the heap) can be copied by the runtime for a variety of reasons.

The data is still stored in memory. The problem is that you generally can't prevent in-process access to your own memory, so isolating it from yourself is difficult or impossible, so doing things like including it as an environment variable doesn't much decrease your security.

But on the non-production environment they should be able to access ;).

Access, yes. Impersonate the deployed service, ideally no.

Can you please explain this in more detail? I'm curios and open for any other solutions. How would you provide the credentials to the application to e.g. connect to the database? You mean not to inject it in the configuration system via environment variable or secret files right? Does it mean you request the credentials from the security provider each time you are establishing a database connection for instance?

For the most part, the various frameworks handle this transparently for you. For developer situations this usually means that after login (eg, az login) a secrets file gets written with time-limited access tokens, to a user-data location, or otherwise modifies the user environment. I'm less sure what happens during actual cloud deployment, but this may still be written as an environment variable.
In either case you don't have to manually manage or retrieve the secret in application code, as the injection is part of the spin-up of the resource.

This tends to also include things like loading secrets straight from key vaults, too.

[HHobeck]

As long as you are not concatenating the string you have the sensitive information exactly one time in memory.

This is false. The memory manager is free to make copies of a string at any time, to enable things like heap compaction. Depending on how your parse/decrypt is implemented, you'll have partial or whole copies of the raw data in byte arrays.

It's just a matter of time until this holds true:

• GC Heap Compaction should clear the source data #10480