[Bug]: `TimeoutException` when sending many concurrent API requests to Podman

[vlflorian]

Testcontainers version

4.4.0

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

AMD64

.NET version

8.0.401

Docker version

$ podman version
Client:       Podman Engine
Version:      5.4.2
API Version:  5.4.2
Go Version:   go1.24.2
Git Commit:   be85287fcf4590961614ee37be65eeb315e5d9ff
Built:        Wed Apr  2 18:33:14 2025
OS/Arch:      windows/amd64

Server:       Podman Engine
Version:      5.4.2
API Version:  5.4.2
Go Version:   go1.23.7
Git Commit:   be85287fcf4590961614ee37be65eeb315e5d9ff
Built:        Wed Apr  2 02:00:00 2025
OS/Arch:      linux/amd64

Docker info

$ podman info
Client:
  APIVersion: 5.4.2
  Built: 1743611594
  BuiltTime: Wed Apr  2 18:33:14 2025
  GitCommit: be85287fcf4590961614ee37be65eeb315e5d9ff
  GoVersion: go1.24.2
  Os: windows
  OsArch: windows/amd64
  Version: 5.4.2
host:
  arch: amd64
  buildahVersion: 1.39.4
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.13-1.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 99.98
    systemPercent: 0.01
    userPercent: 0.01
  cpus: 24
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "41"
  eventLogger: journald
  freeLocks: 2043
  hostname: FRA-D-246
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.167.4-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 14985383936
  memTotal: 16608931840
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250415.g2340bbf-1.fc41.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: unix:///run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 102h 35m 36.00s (Approximately 4.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 1825034240
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.4.2
  BuildOrigin: Fedora Project
  Built: 1743552000
  BuiltTime: Wed Apr  2 02:00:00 2025
  GitCommit: be85287fcf4590961614ee37be65eeb315e5d9ff
  GoVersion: go1.23.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

What happened?

Hi!

I am not entirely sure if this is a bug or related to something else, but I wanted to follow up on my question here (where someone reported the same problem that we are encountering) so as to not derail the conversation.

Context: removed Docker Desktop, installed Podman, with Docker extension. We use TestContainers in our tests for spinning up Redis, Postgres, MinIO. The problem occurs with any of these containers. Both me and some colleagues are having the same issue.

Issue: TimeOutException, immediately after launching test with dotnet test. Stacktrace below.

Thank you for any help or insight anyone could give us!

Relevant log output

Failed Xyz.Tests.SomeTest [1 ms]
  Error Message:
   System.TimeoutException : The operation has timed out.
  Stack Trace:
     at System.IO.Pipes.NamedPipeClientStream.ConnectInternal(Int32 timeout, CancellationToken cancellationToken, Int32 startTime)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Docker.DotNet.DockerClient.<>c__DisplayClass6_0.<<-ctor>b__0>d.MoveNext() in /_/src/Docker.DotNet/DockerClient.cs:line 80
--- End of stack trace from previous location ---
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 164
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Microsoft.Net.Http.Client/ManagedHandler.cs:line 80
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken) in /_/src/Docker.DotNet/DockerClient.cs:line 423
   at Docker.DotNet.DockerClient.MakeRequestAsync[T](IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /_/src/Docker.DotNet/DockerClient.cs:line 254
   at Docker.DotNet.ContainerOperations.StartContainerAsync(String id, ContainerStartParameters parameters, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 207        
   at DotNet.Testcontainers.Clients.TestcontainersClient.StartAsync(String id, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 129
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 505
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 301
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, ILogger logger, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 224
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, ILogger logger, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 248
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartDefaultAsync(IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, ILogger logger, Boolean isWindowsEngineEnabled, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 135
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 319
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 454
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 298
   at Xyz.Inf.TestUtils.DbFixtures.PostgresFixture.InitializeAsync()

Additional information

No response

[HofmeisterAn]

System.IO.Pipes.NamedPipeClientStream.ConnectInternal(Int32 timeout, CancellationToken cancellationToken, Int32 startTime)

It looks like Testcontainers can't connect to the named pipe (Podman Engine). What's the Podman endpoint in Settings > Resources? Could you also share the output of podman context ls? Are you able to run containers from the CLI?

[vlflorian]

Hi @HofmeisterAn,

I greatly appreciate the swift response and help!

The Podman endpoints seems to be: npipe://\.\pipe\podman-machine-default

$  podman context ls
Name                         URI                                                          Identity                                                               Default     ReadWrite
podman-machine-default       ssh://user@127.0.0.1:53942/run/user/1000/podman/podman.sock  C:\Users\vl\.local\share\containers\podman\machine\machine  false       true
podman-machine-default-root  ssh://root@127.0.0.1:53942/run/podman/podman.sock            C:\Users\vl\.local\share\containers\podman\machine\machine  true        true

Running containers from the cli works fine.

[HofmeisterAn]

Are you sure everything was uninstalled correctly? No leftovers? Are there any environment variables or settings left in ~/.testcontainers.properties? Could you please share the Testcontainers logs?

The configuration you shared looks good and is similar to what I have on my test machine.

[vlflorian]

I reinstalled podman and podman desktop to be sure, but to no effect.
Weirdly enough, the tests work fine when running in Rider. Apparently it had a setting that set test parallelism to 1. When set to 4, I get the same TimeOutException as the one that occurs when running with dotnet test. (Both ways to test were running fine with Docker) I am clueless :-)

• There is one env var that we have set: DOCKER_HOST=npipe://./pipe/podman-machine-default, could this be a problem?

• ~/.testcontainers.properties contents:

docker.auth.config={...}
docker.host=npipe://./pipe/podman-machine-default

• I do not know how to view the TestContainer logs and can't seem to find how to, sorry.

[HofmeisterAn]

The configuration settings DOCKER_HOST and docker.host shouldn't be necessary. Although they seem correct, could you please try removing them? Testcontainers should automatically detect the container runtime environment.

• I do not know how to view the TestContainer logs and can't seem to find how to, sorry.

Try running the tests in debug in your IDE; the logs should appear in the console output of your IDE.

[vlflorian]

Hi Andre,
Same issue, I'm afraid. parallel 1 succeeds, 4 fails (or when running with dotnet test)

Only logs I could find:

info: MinIOFixture[0]
      Connected to Docker:
        Host: npipe://./pipe/podman-machine-default
        Server Version: 5.4.2
        Kernel Version: 5.15.167.4-microsoft-standard-WSL2
        API Version: 1.41
        Operating System: fedora
        Total Memory: 15.47 GB
info: PostgresFixture[0]
      Connected to Docker:
        Host: npipe://./pipe/podman-machine-default
        Server Version: 5.4.2
        Kernel Version: 5.15.167.4-microsoft-standard-WSL2
        API Version: 1.41
        Operating System: fedora
        Total Memory: 15.47 GB
info: RedisFixture[0]
      Connected to Docker:
        Host: npipe://./pipe/podman-machine-default
        Server Version: 5.4.2
        Kernel Version: 5.15.167.4-microsoft-standard-WSL2
        API Version: 1.41
        Operating System: fedora
        Total Memory: 15.47 GB
info: MinIOFixture[0]
      Docker container e73ca213adc5 created
info: MinIOFixture[0]
      Start Docker container e73ca213adc5
info: MinIOFixture[0]
      Wait for Docker container e73ca213adc5 to complete readiness checks
info: MinIOFixture[0]
      Docker container e73ca213adc5 ready
info: MinIOFixture[0]
      Docker container a62085b538a5 created
info: RedisFixture[0]
      Docker container f6b607951eee created
info: PostgresFixture[0]
      Docker container d6a250d2c72a created
info: RedisFixture[0]
      Start Docker container f6b607951eee
info: MinIOFixture[0]
      Start Docker container a62085b538a5
info: PostgresFixture[0]
      Start Docker container d6a250d2c72a
info: RedisFixture[0]
      Wait for Docker container f6b607951eee to complete readiness checks
info: MinIOFixture[0]
      Wait for Docker container a62085b538a5 to complete readiness checks
info: PostgresFixture[0]
      Wait for Docker container d6a250d2c72a to complete readiness checks
info: PostgresFixture[0]
      Execute "pg_isready --host localhost --dbname testdb --username testuser" at Docker container d6a250d2c72a
info: PostgresFixture[0]
      Docker container d6a250d2c72a ready

[HofmeisterAn]

The output looks pretty good. I'm just wondering why you're seeing the line Connected to Docker multiple times. Are you using different logger instances? That line should only be logged once per connection and logger instance (maybe you can test how it behaves when reusing the same logger instance).

I have no idea why you're running into this. It kind of looks like there are too many simultaneous requests, but I can't imagine that's the issue. You're using xUnit.net, right?

[vlflorian]

We are using 3 TestContainers at once 🙈 we are using XUnit, yes!
Something akin to too many simultaneous requests sounds like it could be the problem, since the errors are mostly disappearing when the tests are run with parallelism disabled.

[HofmeisterAn]

We are using 3 TestContainers at once

This shouldn't be a problem at all (unless you're hitting some resource limits). I've never run into an issue like that with Docker. Maybe try reaching out to the Podman team for some guidance. I'm still curious, though, why you're seeing the container runtime info more than once.