Host: StartupTimeout not enforced for hosted services and can break application state

[mus65]

Description

The host implementation doesn't seem to check StartupTimeout itself, but relies on the service implementations checking the cancellationToken.

We had a situation where this lead to a broken application state. In the reproduction example, the ABackgroundService is cancelled because of the startup timeout, but the application starts successfully anyway. So now the application is running without the background service.

Even without the ABackgroundService, I wouldn't expect the application to start successfully here, because the StartupTimeout was 5s, but the ASlowStartService took 10s to start up.

I realize that the try/catch in ABackgroundService should not be needed. We did this as a workaround for #98935. Either way I could see other applications doing this unintentionally.

Reproduction Steps

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

HostApplicationBuilder builder = Host.CreateApplicationBuilder();
builder.Services.AddHostedService<ABackgroundService>();
builder.Services.AddHostedService<ASlowStartService>();
builder.Logging.AddConsole();

builder.Services.Configure<HostOptions>(o =>
{
  o.StartupTimeout = TimeSpan.FromSeconds(5);
});

var host = builder.Build();
await host.RunAsync();

internal class ABackgroundService(ILogger<ABackgroundService> logger) : BackgroundService
{
  protected override async Task ExecuteAsync(CancellationToken stoppingToken)
  {
    try
    {
      await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
    }
    catch (OperationCanceledException)
    {
      logger.LogInformation("cancelled");
    }
  }
}

internal class ASlowStartService : IHostedService
{
  public async Task StartAsync(CancellationToken cancellationToken)
  {
    await Task.Delay(TimeSpan.FromSeconds(10), CancellationToken.None);
  }

  public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

Expected behavior

Host fails to start.

Actual behavior

Host starts successfully, but BackgroundService is aborted.

info: ABackgroundService[0]
      cancelled
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.

Regression?

No response

Known Workarounds

We worked around this adding cancellationToken.ThrowIfCancellationRequested() in StartedAsync.

Configuration

.NET 8.0.7
Windows x64

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-extensions-hosting
See info in area-owners.md if you want to be subscribed.

[steveharter]

It is a cooperative model based on CancellationToken; see StartAsync and StopAsync where the CancellationToken needs to be flowed.

Is the intent of the sample:

  public async Task StartAsync(CancellationToken cancellationToken)
  {
    await Task.Delay(TimeSpan.FromSeconds(10), CancellationToken.None);
  }

to not flow the cancellationToken? i.e. if changed to:

  public async Task StartAsync(CancellationToken cancellationToken)
  {
    await Task.Delay(TimeSpan.FromSeconds(10), cancellationToken);
  }

then the result is an exception in the ASlowStartServiceStartAsync():

info: ABackgroundService[0]
      cancelled
fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at ASlowStartService.StartAsync(CancellationToken cancellationToken) in C:\Users\sharter\source\repos\ConsoleApp270\ConsoleApp270\Program.cs:line 37

[mus65]

This was intentional for the sample. In our actual problem case, we were calling a third party sync API that doesn't accept a cancellationToken.

Couldn't the host call cancellationToken.ThrowIfCancellationRequested() after each stage?

[steveharter]

Couldn't the host call cancellationToken.ThrowIfCancellationRequested() after each stage?

It could, and I wonder why it wasn't done that way to begin with. If we change it now, it will be a breaking change, so we'd need to consider whether we need an opt-in or opt-out. Also, there's some details to be worked out:

• The options to Start\StopConcurrently=false option won't be covered unless they cooperate with the cancellationToken.
• For stopping, currently the code doesn't even check the cancellationTokenup up-front like it does with Start. Not sure why.
• Today we have a guarantee to call each service for stopping and then throw an AggregateException at the end meaning we'll likely a task cancelled exception for each service after they run (grouped into the aggregate exception) unless we want to ignore\handle that specifically.

Moving to v10 for consideration; feedback here welcomed.

[mus65]

It could, and I wonder why it wasn't done that way to begin with. If we change it now, it will be a breaking change, so we'd need to consider whether we need an opt-in or opt-out.

Imho you shouldn't have to opt into this. And while this would be worth a breaking change notice, I'm not sure an opt-out is even necessary. Why would you opt out of this? If this change causes a startup failure, the configured timeout was too low to begin with and could just be increased or disabled.