multiple JwtBearer authentication schemes continually refresh metadata

[acblksun]

Describe the bug

We require accepting multiple JwtBearer audience/authority pairs for our api.

When we setup multiple JwtBearer authentication for our api, we notice that aspnetcore middleware is continually calling the .well-known/openid-configuration at the minimum refresh interval of 30 seconds instead of the desired AutomaticRefreshInterval of 1 day.

To Reproduce

Steps to reproduce the behavior:

1. Using ASP.NET Core Version 2.2
2. Configure to use multiple Jwt Bearers (see code below)
3. Get a valid authorization token (note - invalid tokens have the same result)
4. Run any endpoint that engages the middleware e.g. [Authorize] and pass the token correctly
5. Observe that the .well-known/openid-configuration for all configured JwtBearer's will be called every 30 seconds (or every RefreshInterval) instead of once a day (or every AutomaticRefreshInterval).

Expected behavior

Expect the default Refresh Interval of 30 seconds, and AutomaticRefreshInterval of 1 day to be sufficient, and the .well-known/openid-configuration to be called at most once per day per scheme.

Note that I did perform a test of a single Jwt Bearer registered as the default, and it did perform as expected only calling at the AutomaticRefreshInterval

Additional context

• using the http interceptor isn't required if you can monitor with Application Insights
• it makes no difference if the audience strings are the same or different

Include the output of dotnet --info

C:\repos\test>dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.401
 Commit:    729b316c13

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17763
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.401\

Host (useful for support):
  Version: 2.2.6
  Commit:  7dac9b1b51

.NET Core SDKs installed:
  1.0.2 [C:\Program Files\dotnet\sdk]
  1.0.3 [C:\Program Files\dotnet\sdk]
  1.0.4 [C:\Program Files\dotnet\sdk]
  1.1.0 [C:\Program Files\dotnet\sdk]
  2.0.2 [C:\Program Files\dotnet\sdk]
  2.0.3 [C:\Program Files\dotnet\sdk]
  2.1.2 [C:\Program Files\dotnet\sdk]
  2.1.4 [C:\Program Files\dotnet\sdk]
  2.1.101 [C:\Program Files\dotnet\sdk]
  2.1.102 [C:\Program Files\dotnet\sdk]
  2.1.103 [C:\Program Files\dotnet\sdk]
  2.1.104 [C:\Program Files\dotnet\sdk]
  2.1.200 [C:\Program Files\dotnet\sdk]
  2.1.201 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.401 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.1.403 [C:\Program Files\dotnet\sdk]
  2.1.500 [C:\Program Files\dotnet\sdk]
  2.1.502 [C:\Program Files\dotnet\sdk]
  2.1.503 [C:\Program Files\dotnet\sdk]
  2.1.504 [C:\Program Files\dotnet\sdk]
  2.1.505 [C:\Program Files\dotnet\sdk]
  2.1.507 [C:\Program Files\dotnet\sdk]
  2.1.700 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.102 [C:\Program Files\dotnet\sdk]
  2.2.300 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Code to reproduce. I used the VS2019 template for a webapi and modified it slightly.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.HttpsPolicy;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

namespace WebApplicationAuthTest
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            // this example uses 2 different auth0 tenants, but this shouldn't matter
            const string audience_one = "https://api1.example.com";
            const string authority_one = "https://test-tenant-a.au.auth0.com/";
            const string audience_two = "https://api2.example.com";
            const string authority_two = "https://test-tenant-b.au.auth0.com/";

            var auth = services
                .AddAuthentication()
#if true
                .AddJwtBearerWithHttpIntercept("Auth-A", audience_one, authority_one)
                .AddJwtBearerWithHttpIntercept("Auth-B", audience_two, authority_two)
#else
                // AddJwtBearerWithHttpIntercept is equivalent to these. All it adds is http client interception for request/response logging.
                .AddJwtBearer("Auth-A", options =>
                {
                    options.Audience = audience_one;
                    options.Authority = authority_one;

                    options.Validate();
                })
                .AddJwtBearer("Auth-B", options =>
                {
                    options.Audience = audience_two;
                    options.Authority = authority_two;

                    options.Validate();
                })
#endif
                ;


            services.AddAuthorization(options =>
            {
                options.DefaultPolicy = new AuthorizationPolicyBuilder()
                    .RequireAuthenticatedUser()
                    .AddAuthenticationSchemes("Auth-A", "Auth-B")
                    .Build();
            });

            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
                app.UseHsts();
            }

            app.UseHttpsRedirection();
            app.UseMvc();
        }
    }


    public static class JwtConfiguration
    {
        public static AuthenticationBuilder AddJwtBearerWithHttpIntercept(this AuthenticationBuilder authenticationBuilder, string schemeIdentifier, string audience, string authority)
        {
            return authenticationBuilder
                 .AddJwtBearer(schemeIdentifier, options =>
                 {
                     options.Audience = audience;
                     options.Authority = authority;


                     // Except for this - we added it ourselves. Technically all we should need to set is this unless we want to modify configurationManager.
                     options.BackchannelHttpHandler = new LoggingHandler(new System.Net.Http.HttpClientHandler());

#if false

                     // this is as-is defaults from https://github.com/aspnet/AspNetCore/blob/v2.2.6/src/Security/Authentication/JwtBearer/src/JwtBearerPostConfigureOptions.cs
                     if (string.IsNullOrEmpty(options.MetadataAddress) && !string.IsNullOrEmpty(options.Authority))
                     {
                         options.MetadataAddress = options.Authority;
                         if (!options.MetadataAddress.EndsWith("/", StringComparison.Ordinal))
                         {
                             options.MetadataAddress += "/";
                         }

                         options.MetadataAddress += ".well-known/openid-configuration";
                     }

                     var httpClient = new System.Net.Http.HttpClient(options.BackchannelHttpHandler ?? new System.Net.Http.HttpClientHandler());
                     httpClient.Timeout = options.BackchannelTimeout;
                     httpClient.MaxResponseContentBufferSize = 1024 * 1024 * 10; // 10 MB

                     var configurationManager = new Microsoft.IdentityModel.Protocols.ConfigurationManager<Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfiguration>(
                         options.MetadataAddress, new Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever(),
                         new Microsoft.IdentityModel.Protocols.HttpDocumentRetriever(httpClient) { RequireHttps = options.RequireHttpsMetadata });
                     options.ConfigurationManager = configurationManager;


                     // Except here - we manually increase the RefreshInterval to prove it's the RefreshInterval
                     configurationManager.RefreshInterval = TimeSpan.FromMinutes(5);
                     // configurationManager.AutomaticRefreshInterval = 
#endif


                     options.Validate();
                     Console.WriteLine($"Added {schemeIdentifier} for authority {authority} and audience {audience}");

                 });
        }
    }

    public class LoggingHandler : System.Net.Http.DelegatingHandler
    {
        public LoggingHandler(System.Net.Http.HttpMessageHandler innerHandler)
            : base(innerHandler)
        {
        }

        protected override async Task<System.Net.Http.HttpResponseMessage> SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)
        {
            Console.WriteLine("Request:");
            Console.WriteLine(request.ToString());
            if (request.Content != null)
            {
                Console.WriteLine(await request.Content.ReadAsStringAsync());
            }
            Console.WriteLine();

            System.Net.Http.HttpResponseMessage response = await base.SendAsync(request, cancellationToken);

            Console.WriteLine("Response:");
            Console.WriteLine(response.ToString());
            if (response.Content != null)
            {
                Console.WriteLine(await response.Content.ReadAsStringAsync());
            }
            Console.WriteLine();

            return response;
        }
    }
}

[Tratcher]

Does it hit this code path?
https://github.com/aspnet/AspNetCore/blob/71c5c66b211c7ab9d44751e39cef27c525124bb7/src/Security/Authentication/JwtBearer/src/JwtBearerHandler.cs#L112-L121
A) Token arrives for Auth-B.
B) Authenticate runs for Auth-A, fails with SecurityTokenSignatureKeyNotFoundException (?), requests a rate limited refresh of metadata
C) Authenticate runs for Auth-B and succeeds.

If this is what's happening then you should only see refreshes for A, not B. Disabling RefreshOnIssuerKeyNotFound would prevent it.

[HudsonAkridge]

We're having the exact same problem where RefreshOnIssuerKeyNotFound is firing on either Bearer Scheme failing. One of the two will always fail for us, while the other succeeds.

[Tratcher]

@HudsonAkridge it should only ever fire for the first provider. Can you confirm?

[eliaslopezgt]

@Tratcher If I add 4 JWTBearerHandlers and only one can validate the token, it reloads the configuration on the other 3. Hope that helps. #14397

[blowdart]

Does it happen every time? Every so often isn't unsurprising as it will go through the handlers in order and refresh to see if there's a new key.

[eliaslopezgt]

Maybe this section from the issue I created helps.

Expected behavior

This behavior is acceptable as long as there's only one JWTBearerHandler.
Having multiple JWTBearerHandlers results in unwanted traffic to the configuration endpoints.
This can be currently mitigated by:

1. Setting the RefreshInterval to something different than its default (30 seconds)
2. Setting RefreshOnIssuerKeyNotFound to false in all the JWTBearerHandlers

My proposed solution is to create an additional OnSignatureValidationFailed event that can be triggered before Options.ConfigurationManager.RequestRefresh(); in JwtBearerHandler.cs
https://github.com/aspnet/AspNetCore/blob/master/src/Security/Authentication/JwtBearer/src/JwtBearerHandler.cs

This with the purpose of allowing the caller to intercept the RequestRefresh and in this case, inject logic that can compare the invalid token audience and authority, with the audience and authority of the handler, if they are the same, then trigger the refresh otherwise it means we are trying to validate a token that is going to fail the validation no matter if we refresh.

The only change to the JWTBearerHandler is the addition of the OnSignatureValidationFailed event. I have a PR ready but wanted to discuss if this could bring any value or not.

[Tratcher]

@blowdart this is functioning as intended, and the mitigations listed above are effective:

1. Setting the RefreshInterval to something different than its default (30 seconds)
2. Setting RefreshOnIssuerKeyNotFound to false in all the JWTBearerHandlers

One way to redesign this would be to build one JwtAuthHandler that could contain multiple configurations. It would loop through all of them before triggering this kind of failure. Similar logic already exists when there are multiple token validators configured.

https://github.com/aspnet/AspNetCore/blob/752d99ca531f587fae92da63c1120f95c453e72a/src/Security/Authentication/JwtBearer/src/JwtBearerHandler.cs#L103-L105

Recommend backlog.

[eliaslopezgt]

@blowdart can I help with it?