Health Checking / Heartbeat for BackgroundService, IHostedService

[milen-denev]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

Introduction:
I am not quite sure where should I post this feature request, but because the nuget packages that I am trying to use to achieve this results lead to this repo, I am posting here my proposition.

What my configuration is:
I have a Host app, running as WindowsService on a server, with several services added using .AddHostedService<>()
Also I have services.Configure -> BackgroundServiceExceptionBehavior = BackgroundServiceExceptionBehavior.Ignore
What I am trying to achieve:
Sometimes my services may crash due to dynamic changes, and misconfiguration and mostly due to the fact that I am not using try catch patterns. In case a background service crashes I get an event in the Event Viewer, but this has to be manually checked, or reported by users that something isn't running at all or use try finally to make the service report itself.
In case if there is permanent issue that has to be resolved manually, it may be better to leave the BackgroundService "crashed" / inactive and after the issue is fixed, having the whole service restarted in order to be updated by the "patching" service.

Describe the solution you'd like

1. The solution that I would love to see is to be able from other service or from other place (ex. in the Main() method) to check the status of the BackgroundService, ex. if it's running, starting, stopping, inactive.
2. It may be great to be able to have another method within the service itself, that runs in parallel to the main thread / Task, and allows to be used like heartbeat that reports somewhere that the service is alive, and still use the Task.Delay() to configure how frequenlty to give a heartbeat.
3. A control over the services like Something.Stop() / Start() / Pause() may be useful (ex. updating some app that the service depends on and pausing until it's done).
4. Some straightforward way to collect "diagnostic" or other data from the main thread, like how much time took for the latest query to execute, like back-and-forth thingie between the hearbeat thread and main the thread.

Additional context

Currently I digged as much as I can and I didn't find absolutely anything that could help me with monitoring, controlling or at least getting the state of a BackgroundService (without using Azure Monitoring). Azure Monitoring and Application Insights it's a great service , but it's not free (1.94 EUR for log injection per GB) and if I want to actully use this data for something outside the Azure portal, I have to use the LogAnalytics API and kusto query language. If there is any other way except this one, I would be very happy to hear it.

[davidfowl]

What exactly are you looking for? UI? An API? It’s unclear from this description…

[milen-denev]

For sure I am looking for an API to be used. I don't think that an UI fits in this case anyways, I cannot imagine a windows service having a UI for checking it's health, the way AddHealthChecksUI does for ASP.NET Core websites.

But if someone comes up with an idea , let's say going to "localhost:5932" and opening up a webpage thats looking like the AddHealthChecksUI , it might be great feature.

Although the core functionallity, of being able to check the status of a BackgroundService and it's state is much more importatnt.

[davidfowl]

Can you write this health check today? Enumerate all hosted services, case to BackgroundService an check the Task (https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.hosting.backgroundservice.executetask?view=dotnet-plat-ext-7.0#microsoft-extensions-hosting-backgroundservice-executetask) to see if it’s faulted?

I’m still not 100% sure what you’re asking for API wise. Are these background services that you wrote? Why can’t they handle errors or report health status ?

[milen-denev]

Ok so, here is what I did. I am not quite sure if it's a good "design choise" or it's the best way to do it:

I wrote an new class which inherits the BackgroundService

public abstract class ExtendedBackgroundService : BackgroundService
{
    public TimeSpan HeartbeatDelay { get; set; } = TimeSpan.FromSeconds(1);
    public DateTime LastFired { get => lastFired; set => lastFired = DateTime.Now; }
    public ServiceStatus Status { get => status; set => status = ServiceStatus.Running; }
    private DateTime lastFired { get; set; }
    private ServiceStatus status { get; set; }

    private Task HealthCheckTask { get; set; }

    public override async Task StartAsync(CancellationToken cancellationToken)
    {
        HealthCheckTask = await Task.Factory.StartNew(async () => await HeartbeatAsync(cancellationToken), cancellationToken, TaskCreationOptions.LongRunning, TaskScheduler.Current);
        await base.StartAsync(cancellationToken);
    }

    public virtual async Task HeartbeatAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            lastFired = DateTime.Now;
            status = ServiceStatus.Running;
            Console.WriteLine("Running");
            await Task.Delay(HeartbeatDelay, stoppingToken);
        }
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        return Task.CompletedTask;
    }

    public override Task StopAsync(CancellationToken cancellationToken)
    {
        Status = ServiceStatus.Stopped;
        HealthCheckTask.Dispose();
        return base.StopAsync(cancellationToken);
    }
}

public enum ServiceStatus
{
    Started,
    Running,
    Stopped
}

And there is a sample of the "MyService" which inherits the ExtendedBackgroundService:

public class MyService : ExtendedBackgroundService
{
    private readonly MyContext myContext;
    private readonly SomeManager someManager;

    public ProductPointsService(MyContext myContext, SomeManager someManager)
    {
        this.myContext = myContext;
        this.someManager = someManager;
        HeartbeatDelay = TimeSpan.FromSeconds(5);
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await someManager.DoingSomething();
            await Task.Delay(TimeSpan.FromHours(24), stoppingToken);
        }
    }
}

And finally the a HealthCheckService:

public class HealthCheckService : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;

    public HealthCheckService(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            using (var scope = _serviceProvider.CreateScope())
            {
                var host = scope.ServiceProvider.GetRequiredService<IHost>();
               //I say IHostedService because if I try to GetServices<ExtendedBackgroundService> it simply doesn't work
                var hostedServices = host.Services.GetServices<IHostedService>();

                foreach (var service in hostedServices)
                {
                    if(service.GetType().Name == "MyService")
                    {
                        var _service = service as ExtendedBackgroundService;
                    }
                }
            }

            await Task.Delay(1000, stoppingToken);
        }
    }
}

This is what you mean by "Enumerate all hosted services, case to BackgroundService an check the Task" ?

[Dulatr]

This issue is similar to what I'm also looking for but I'm not using windows services.

Came across this open issue and interpreted @davidfowl's statement as such:

public sealed class WorkerHealthCheck : IHealthCheck
{
	private readonly IServiceProvider _services;

	public WorkerHealthCheck(IServiceProvider services)
	{
		_services = services;
	}

	public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = new CancellationToken())
	{

		var workers = _services
			.GetServices<IHostedService>()
			.Where(service => service is MyWorker)
			.ToList();

		if (workers.Any())
		{
		        // could also just check status of all workers of type or whatever 
			var worker = workers.First() as MyWorker;
			var status = worker?.ExecuteTask?.Status;
			if (status is not null)
			{
				return Task.FromResult(status == TaskStatus.Faulted
					? HealthCheckResult.Unhealthy("MyWorker is not running", worker?.ExecuteTask?.Exception) : HealthCheckResult.Healthy("MyWorker is running"));
			}
		}

		return Task.FromResult(HealthCheckResult.Healthy("No worker was started"));
	}
}

Which could be extended to a more involved status check on the service itself. However I am running into an issue where the status of the execute task for my worker is always awaiting activation. The worker itself does basically this:

public sealed class MyWorker : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // setup
        
        while (!stoppingToken.IsCancellationRequested)
        {
            // await and handle service bus messages
        }
    }
}

@milen-denev Did you end up getting your solution to work? I know this is a bit old at this point but issue was still open.

I feel like I'm misunderstanding something pretty basic, but I'm not sure why when I check the status of the execution task it is always awaiting activation. Any help would be appreciated.

[milen-denev]

@Dulatr I think you have to implement the WorkerHealthCheck logic within another ExecuteAsync function of another service. All I was trying to point out is that there is nothing (that I am aware of) does exactly that, like health checks, stopping starting, pausing, restarting a BackgroundService. I hope to have soon something simple to use that does that.