Threading issues in Entity Framework Core 3.0 health check

Following the exciting release of .NET Core 3.0, I was expecting that our services would handle the upgrade from 2.2 to 3.0 with ease. After the upgrade, however, the services sometimes responded with "Unhealthy" and a 503 status code!

The connection was not closed. The connection's current state is open.

The issue was triggered by running two health check queries against the SQL database in the same scoped instance. A scoped instance is an instance that is created once every client request. That only means one thing... Threading issues! Let's take a closer look:

services.AddHealthChecks()
    .AddDbContextCheck<AppDbContext>()
    .AddCheck<CustomDatabaseHealthCheck>(nameof(CustomDatabaseHealthCheck));

One health check is the built-in health check from .NET Core, DbContextHealthCheck. The Microsoft documentation tells us that:

The DbContextHealthCheck calls EF Core's CanConnectAsync method. You can customize what operation is run when checking health using AddDbContextCheck method overloads.

The other health check is a custom health check that is written for the specific application.

Why does this happen?

This GitHub issue was reported after the release of 3.0 and refers to changes in the CheckHealthAsync method. This problem occurs after the upgrade from .NET Core 2.2 to 3.0. When looking at the changes between the two versions, it seems that both versions of the API use the same scoped life-cycle for all health checks. So it's strange that the 2.2 version works in the same scope and the 3.0 doesn't.

Let's look at the custom health check:

public class CustomDatabaseHealthCheck : IHealthCheck
{
    public AppDbContext DB { get; }

    public DatabaseHealthCheck(AppDbContext db)
    {
        DB = db;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            await DB.Table.FirstOrDefaultAsync(cancellationToken);
            return HealthCheckResult.Healthy();
        }
        catch (Exception)
        {
            return HealthCheckResult.Unhealthy();
        }
    }
}

The DbContext gets injected through the constructor and shares the same instance as the other health checks. A quick Google search on the issue suggests changing the life cycle of the DbContext from scoped to transient, but that is a rather drastic change for this case.

Solving the issue

We need to create a separate scope for the DbContext and write the custom health logic inside that scope.

public class CustomDatabaseHealthCheck : IHealthCheck
{
    public IServiceScopeFactory ScopeFactory { get; }

    public DatabaseHealthCheck(IServiceScopeFactory scopeFactory)
    {
        ScopeFactory = scopeFactory;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        using (var scope = ScopeFactory.CreateScope())
        {
            try
            {
                var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                await db.Table.FirstOrDefaultAsync(cancellationToken);
                return HealthCheckResult.Healthy();
            }
            catch (Exception)
            {
                return HealthCheckResult.Unhealthy();
            }
        }
    }
}

Here we create a new scope and retrieves the DbContext from the service provider. Inside it, the context is fully isolated and won't interfere with other threads. The application is now healthy!

One could argue that the custom health check should be some sort of smoke test rather than a health check. A solution could be to remove the custom health check altogether, but this approach should work if you need this as a health check after all.