[Bug]: SSL Connection Error in CosmosDbContainerTest

[wilsjame]

Testcontainers version

3.7.0

Using the latest Testcontainers version?

Yes

Host OS

macOS Sonoma Version 14.2.1 (23C71)

Host arch

i386

.NET version

8.0

Docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:02 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    24.0.7
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.0-desktop.2
    Path:     /Users/james/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3-desktop.2
    Path:     /Users/james/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/james/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/james/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  0.1
    Path:     /Users/james/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.10
    Path:     /Users/james/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/james/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/james/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/james/.docker/cli-plugins/docker-scout

Server:
 Containers: 24
  Running: 2
  Paused: 0
  Stopped: 22
 Images: 41
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.5.11-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 23.48GiB
 Name: docker-desktop
 ID: 7ed1da88-0431-4488-aa71-2ad49cd8db43
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

What happened?

This line throws an error: SSL connection could not be established.

testcontainers-dotnet/tests/Testcontainers.CosmosDb.Tests/CosmosDbContainerTest.cs

Line 31 in 1c80fe8

var accountProperties = await cosmosClient.ReadAccountAsync()

Expected no error and test passes.

Relevant log output

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.

System.Net.Http.HttpRequestException
The SSL connection could not be established, see inner exception.
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.Azure.Cosmos.CosmosHttpClientCore.ExecuteHttpHelperAsync(HttpRequestMessage requestMessage, ResourceType resourceType, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.CosmosHttpClientCore.SendHttpHelperAsync(Func`1 createRequestMessageAsync, ResourceType resourceType, HttpTimeoutPolicy timeoutPolicy, IClientSideRequestStatistics clientSideRequestStatistics, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)
   at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetAccountPropertiesHelper.GetAndUpdateAccountPropertiesAsync(Uri endpoint)
   at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetAccountPropertiesHelper.GetAccountPropertiesAsync()
   at Microsoft.Azure.Cosmos.GatewayAccountReader.InitializeReaderAsync()
   at Microsoft.Azure.Cosmos.CosmosAccountServiceConfiguration.InitializeAsync()
   at Microsoft.Azure.Cosmos.DocumentClient.InitializeGatewayConfigurationReaderAsync()
   at Microsoft.Azure.Cosmos.DocumentClient.GetInitializationTaskAsync(IStoreClientFactory storeClientFactory)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.GetAsync(TKey key, Func`2 singleValueInitFunc, Func`2 forceRefresh)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.GetAsync(TKey key, Func`2 singleValueInitFunc, Func`2 forceRefresh)
   at Microsoft.Azure.Cosmos.DocumentClient.EnsureValidClientAsync(ITrace trace)
   at Microsoft.Azure.Cosmos.DocumentClient.GetDatabaseAccountPrivateAsync(Uri serviceEndpoint, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.ClientContextCore.RunWithDiagnosticsHelperAsync[TResult](String containerName, String databaseName, OperationType operationType, ITrace trace, Func`2 task, Func`2 openTelemetry, String operationName, RequestOptions requestOptions)
   at Microsoft.Azure.Cosmos.ClientContextCore.<>c__DisplayClass31_0`1.<<OperationHelperWithRootTraceWithSynchronizationContextAsync>b__0>d.MoveNext()

Additional information

#833 introduced these changes.

[WakaToa]

Having the same issue. Last run on Friday succeeded. Todays run on exact same commit fails.

[HofmeisterAn]

I had a quick look too and can confirm and reproduce the issue. It appears that the Cosmos DB client is not even calling/using the provided HTTP client (that sets the custom validation callback), and the exceptions happen pretty early. Since the module has not changed, and as @WakaToa mentioned, his configuration ran until Friday, I suspect that the image has changed. AFAIK, there is no version we can pin. Probably the best idea is to create an upstream issue.

[WakaToa]

@HofmeisterAn Can you check what happens if you use the following WaitStrategy?

private class CosmosWaitStrategy : IWaitUntil
{
    public async Task<bool> UntilAsync(IContainer container)
    {
        var uri = new Uri(
            $"https://{container.Hostname}:{container.GetMappedPublicPort(CosmosDbBuilder.CosmosDbPort)}/_explorer/emulator.pem");

        try
        {
            using var httpClient = new HttpClient(new HttpClientHandler()
            {
                ServerCertificateCustomValidationCallback = (_, _, _, _) => true
            });

            var result = await httpClient.GetAsync(uri);

            return result.StatusCode is >= HttpStatusCode.OK and < HttpStatusCode.Ambiguous;
        }
        catch (Exception e)
        {
            return false;
        }
        finally
        {
            await Task.Delay(1000);
        }
    }
}

Usage:

this.container = new CosmosDbBuilder()
            .WithPrivileged(true)
            .WithExposedPort(CosmosDbBuilder.CosmosDbPort)
            .WithCleanUp(true)
            .WithWaitStrategy(Wait.ForUnixContainer()
                .AddCustomWaitStrategy(new CosmosWaitStrategy()))
            .Build();

await this.container.StartAsync();

It seems that the Cosmos Emulator is not ready yet when the tests start. The WaitStrategy will wait until it is fully initialized and the certificate is exposed. Its working for me.

[HofmeisterAn]

Oh, interesting (odd coincidence). Indeed, it works on my machine too. I did something similar yesterday but did not wait long enough. It is super slow. Is it slow on your machine too? I thought it was faster. However, I have not used it much in the past. Are you interested in contributing a fix? I made some changes and used the preconfigured HTTP client. WDYT?

private sealed class WaitUntil : IWaitUntil
{
    public async Task<bool> UntilAsync(IContainer container)
    {
        // CosmosDB's preconfigured HTTP client will redirect the request to the container. 
        const string requestUri = "https://localhost/_explorer/emulator.pem";

        var httpClient = ((CosmosDbContainer)container).HttpClient;

        try
        {
            using var httpResponse = await httpClient.GetAsync(requestUri)
                .ConfigureAwait(false);

            return httpResponse.IsSuccessStatusCode;
        }
        catch (Exception)
        {
            return false;
        }
        finally
        {
            httpClient.Dispose();
        }
    }
}

[HofmeisterAn]

The wait strategy has been updated (#1109).

[mmisztal1980]

I've used the CosmosDbContainerWaitStrategy locally to debug the process, and apparently the internal HTTP client keeps returning null for me.

public sealed class CosmosDbContainerWaitStrategy : IWaitUntil
{
    private const string ReadyProbeUrl = "https://localhost/_explorer/index.html";

    public async Task<bool> UntilAsync(IContainer container)
    {
        try
        {
            using var httpClient = ((CosmosDbContainer)container).HttpClient;

            var result = await httpClient.GetAsync(ReadyProbeUrl)
                .ConfigureAwait(false); // returns null and stays in loop

            return result.IsSuccessStatusCode;
        }
        catch (Exception)
        {
            return false;
        }
    }
}

Inside the emulator, running the mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview image, I'm getting:

DocDBTrace Information: 0 : SetupConfigurations {"GatewayListenAddress":"0.0.0.0","PostgresHostName":"127.0.0.1","PostgresPort":9712,"PostgresDatabase":"postgres","GatewayPublicEndpoint":"127.0.0.1","UserName":"cosmosdev","ApplicationInsightsConnectionString":"InstrumentationKey=ff1dbd63-a66b-43cd-9863-812458fa4ba6","ApplicationInsightsMeterName":"CosmosPGMetrics"}
DocDBTrace Information: 0 : Npgsql Non-Transaction settings -c idle_in_transaction_session_timeout=60000 -c statement_timeout=60000 
DocDBTrace Information: 0 : Starting up Kestrel runtime
Hosting environment: Production
Content root path: /publish/
Now listening on: http://0.0.0.0:8081
DocDBTrace Warning: 0 : Non-HTTPS request detected.
DocDBTrace Error: 0 : Failed request processing System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46
DocDBTrace Error: 0 : Failed request processing System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46
994 assets
11976 modules
webpack 5.88.2 compiled successfully in 30094 ms
DocDBTrace Error: 0 : Failed request processing System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46
DocDBTrace Error: 0 : Failed request processing System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46
DocDBTrace Error: 0 : Failed request processing System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767
   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117

The /_explorer/index.html keeps returning:

{"code":"InternalServerError","message":"System.NullReferenceException: Object reference not set to an instance of an object.\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46"}

My container initialization code is:

Container = new CosmosDbBuilder()
            .WithImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview")
            //.WithEnvironment("ENABLE_EXPLORER", "true")
            .WithEnvironment("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "3")
            .WithEnvironment("AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE", "true")
            //.WithEnvironment("AZURE_COSMOS_EMULATOR_NO_SSL", "false")
            //.WithEnvironment("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", "127.0.0.1")
            .WithExposedPort(8081)
            .WithPortBinding(8081, true)
            .WithWaitStrategy(Wait
                .ForUnixContainer()
                .AddCustomWaitStrategy(new CosmosDbContainerWaitStrategy())
            )
            .Build();

Has anyone run into a similar issue?

[HofmeisterAn]

@mmisztal1980 This discussion explains what you need to override to use the new image. There is also a PR that prepares the update.

[mmisztal1980]

Thank you, I'll have a look at it

edit
confirmed - works