[Bug]: Bitbucket pipelines fail after upgrading from 4.0.0 to 4.1.0

[shainegordon]

Testcontainers version

4.1.0

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

x86

.NET version

9.0.102

Docker version

unknown

Docker info

unknown

What happened?

Today I updated to 4.1.0

Locally everything is fine, however, in bitbucket pipelines, TestContainer errors out.

I have a 4.0.0 rollback hotfix branch, and it works correctly.

The resource reaper is disabled in BitBucket

TestcontainersSettings.ResourceReaperEnabled = string.IsNullOrEmpty(Environment.GetEnvironmentVariable("BITBUCKET_BUILD_NUMBER"));

This is what our bitbucket-pipelines step looks like

  dotnetTest: &dotnetTest
    name: .NET Tests
    image: mcr.microsoft.com/dotnet/sdk:9.0
    caches:
      - dotnetcore
      - docker
    services:
      - docker
    script:
      - dotnet test --configuration Release

Relevant log output

pipeline logs:

  Error Message:
   OneTimeSetUp: Docker.DotNet.DockerApiException : Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: Invalid request"}
  Stack Trace:
     at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /_/src/Docker.DotNet/DockerClient.cs:line 492
   at Docker.DotNet.DockerClient.MakeRequestAsync[T](IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /_/src/Docker.DotNet/DockerClient.cs:line 257
   at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 64
   at DotNet.Testcontainers.Clients.DockerContainerOperations.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/DockerContainerOperations.cs:line 213
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 317
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 416
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 280

docker logs (for failed 4.1.0):

time="2025-01-17T12:11:42.079051162Z" level=info msg="API listen on [::]:2375"
time="2025-01-17T12:12:16Z" level=info msg="Pipelines plugin request authorization." allowed=false method=HEAD plugin=pipelines uri=/_ping
time="2025-01-17T12:12:16.898125640Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "
time="2025-01-17T12:12:16Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2025-01-17T12:12:16Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.41/images/load?quiet=1"
time="2025-01-17T12:13:08Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2025-01-17T12:13:09Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/info
time="2025-01-17T12:13:09Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/version
time="2025-01-17T12:13:09Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/images/postgres:15.3/json"
time="2025-01-17T12:13:09Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/containers/create
time="2025-01-17T12:13:09.305364386Z" level=error msg="AuthZRequest for POST /containers/create returned error: authorization denied by plugin pipelines: Invalid request"

docker logs (for success 4.0.0):

time="2025-01-17T12:53:01.210589617Z" level=info msg="API listen on [::]:2375"
time="2025-01-17T12:53:37Z" level=info msg="Pipelines plugin request authorization." allowed=false method=HEAD plugin=pipelines uri=/_ping
time="2025-01-17T12:53:37.088073667Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "
time="2025-01-17T12:53:37Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2025-01-17T12:53:37Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.41/images/load?quiet=1"
time="2025-01-17T12:54:39Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2025-01-17T12:54:39Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/info
time="2025-01-17T12:54:39Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/version
time="2025-01-17T12:54:40Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/images/postgres:15.3/json"
time="2025-01-17T12:54:40Z" level=info msg="Container create request." ArgsEscaped=false AttachStderr=false AttachStdin=false AttachStdout=false ExposedPorts="map[5432/tcp:{}]" Healthcheck="<nil>" Labels="map[org.testcontainers:true org.testcontainers.lang:dotnet org.testcontainers.resource-reaper-session:45040909-f480-4600-a849-e46f59de5546 org.testcontainers.session-id:45040909-f480-4600-a849-e46f59de5546 org.testcontainers.version:4.0.0+1a78654e9205d3ff01b807967a5408feec921622]" MacAddress= NetworkDisabled=false OnBuild="[]" OpenStdin=false StdinOnce=false StopSignal= StopTimeout="<nil>" Tty=false plugin=pipelines
time="2025-01-17T12:54:40Z" level=info msg="Container create request." AutoRemove=false BlkioDeviceReadBps="[]" BlkioDeviceReadIOps="[]" BlkioDeviceWriteBps="[]" BlkioDeviceWriteIOps="[]" BlkioWeight=0 BlkioWeightDevice="[]" CPUCount=0 CPUPercent=0 CPUPeriod=0 CPUQuota=0 CPURealtimePeriod=0 CPURealtimeRuntime=0 CPUShares=0 CapAdd="[]" CapDrop="[]" Cgroup= CgroupParent= ConsoleSize="[0 0]" ContainerIDFile= CpusetCpus= CpusetMems= DNS="[]" DNSOptions="[]" DNSSearch="[]" DeviceCgroupRules="[]" Devices="[]" ExtraHosts="[]" GroupAdd="[]" IOMaximumBandwidth=0 IOMaximumIOps=0 Init="<nil>" IpcMode= Isolations= KernelMemory=0 Links="[]" LogConfig="{ map[]}" MaskedPaths="[]" Memory=0 MemoryReservation=0 MemorySwap=0 MemorySwappiness="<nil>" Mounts="[]" NanoCPUs=0 NetworkMode=default OomKillDisable="<nil>" OomScoreAdj=0 PidMode= PidsLimit="<nil>" PortBindings="map[5432/tcp:[{0.0.0.0 }]]" Privileged=false PublishAllPorts=false ReadOnlyPaths="[]" RestartPolicy="{ 0}" Runtime= SecurityOpt="[]" ShmSize=0 StorageOpt="map[]" Sysctls="map[]" Ulimits="[]" UsernsMode= VolumeDriver= VolumesFrom="[]" plugin=pipelines
time="2025-01-17T12:54:40Z" level=info msg="Container create request." EndpointsConfig="map[]" plugin=pipelines
time="2025-01-17T12:54:40Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/containers/create

Additional information

No response

[HofmeisterAn]

I think the issue is related to the Docker.DotNet update. After a quick search, it seems there are several different issues that can cause the access authorization plugin to fail. It might be a good idea to create a reproducer using Docker.DotNet and use it to debug the pipeline. I suspect the issue is either due to the Docker Engine API version update or, more likely, the migration from Json.NET to System.Text.Json. It probably makes more sense to create the issue in the forked version of Docker.DotNet.

Unfortunately, I do not have access to a Bitbucket pipeline and cannot look into it. However, here is how you can create a container instance using the plain Docker.DotNet API. I hope this helps you or someone identify the root cause.

const string image = "postgres:15.3";
// TODO: Set the Docker host address according to your Bitbucket Pipelines environment.
using var dockerClientConfiguration = new DockerClientConfiguration(new Uri("npipe://./pipe/docker_engine"));
using var dockerClient = dockerClientConfiguration.CreateClient();
await dockerClient.Images.CreateImageAsync(new ImagesCreateParameters { FromImage = image }, new AuthConfig(), new Progress<JSONMessage>());
await dockerClient.Containers.CreateContainerAsync(new CreateContainerParameters { Image = image });

Edit: If you're okay with it, I will move the issue to the Docker.DotNet repository.

[shainegordon]

Thanks for the feedback @HofmeisterAn and work on this awesome library

Happy for you to move the issue to Docker.DotNet 👍

For now we have no issue or pressing need to upgrade

[HofmeisterAn]

I understand; Bitbucket Pipeline users will need a fix to benefit from further bug fixes, improvements, and features. Just something to keep in mind for the future.

[HofmeisterAn]

Someone reached out to me on Slack and mentioned they were in contact with Atlassian support. The support team indicated that the issue is related to incompatible Docker versions. Bitbucket is using an older version (25.0.3), whereas we recently upgraded to version 27.3.1. Docker 25.0 supports a maximum API version of 1.44, while Docker 27.3 supports API version 1.47.

The Docker client allows specifying an API version. If someone can verify that downgrading the API version resolves the issue, we can enhance Testcontainers to support the DOCKER_API_VERSION environment variable.

var dockerClient = new DockerClientConfiguration(new Uri("<host>")).CreateClient(new Version(1, 44));
// TODO: Pull an image and create a container instance.

[AbbTek]

@HofmeisterAn

Updating to 4.1.0... This is the docker log from Bitbucket

Updating to 4.1.0.txt

I tried what you suggested with DockerClientConfiguration

But I got these errors (I tried 1.44 and 1.41 same errors)

time="2025-02-03T06:59:16Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2025-02-03T06:59:16Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.41/images/create?fromImage=338783334886.dkr.ecr.ap-southeast-2.amazonaws.com%2Fintegration-test-database&tag=2025.01.31"
time="2025-02-03T07:00:48Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri="/v1.41/containers/create?name=integration-db"
time="2025-02-03T07:00:48.876462917Z" level=error msg="AuthZRequest for POST /v1.41/containers/create?name=integration-db returned error: authorization denied by plugin pipelines: Invalid request"

Hope this can help, thanks for your package is really cool

[HofmeisterAn]

I just took a quick look at the logs. Are you trying to pull the image from a private registry? If so, you need to include the credentials. If the image already exists and is cached, you can skip pulling it or simply use a publicly available image like testcontainers/helloworld.