[Bug]: Reusing of containers is broken due to original container can be considered running while it was stopped by means of Testcontainers

[mabrarov]

Testcontainers version

v0.37.0

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

amd64

Go version

1.24.2

Docker version

Client: Docker Engine - Community
 Version:           28.1.1
 API version:       1.49
 Go version:        go1.23.8
 Git commit:        4eba377
 Built:             Fri Apr 18 09:54:03 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.1.1
  API version:      1.49 (minimum version 1.24)
  Go version:       go1.23.8
  Git commit:       01f442b
  Built:            Fri Apr 18 09:52:20 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community
 Version:    28.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.35.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 85
 Server Version: 28.1.1
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-503.40.1.el9_5.x86_64
 Operating System: Rocky Linux 9.5 (Blue Onyx)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.34GiB
 Name: dev.localdomain.local
 ID: cf2a8f9e-9b2e-4fe2-9b7f-d3f1770b34f1
 Docker Root Dir: /containers/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  192.168.0.0/16
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

Refer to item 2 in #3165 (comment) for description of this issue, please.

Note that this repository pull request builds (GitHub actions) fail sometimes because of this issue. Examples:

1. feat: make port mapping checks timeout configurable #3176 (comment)
2. feat: make port mapping checks timeout configurable #3176 (comment).

There is a chance that the root cause of this issue is located in Docker Engine (Daemon) not performing proper synchronization or caching status of container.

Relevant log output

$ go test -count 1 -v -run TestGenericContainer_stop_start_withReuse
=== RUN   TestGenericContainer_stop_start_withReuse
2025/05/31 03:25:38 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 28.1.1
  API Version: 1.48
  Operating System: Rocky Linux 9.5 (Blue Onyx)
  Total Memory: 15704 MB
  Testcontainers for Go Version: v0.37.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 3b6979337694b78adf59fe38d33bea2b86bf51b571ae8ee9f4d6fd5e734175d9
  Test ProcessID: c1fca114-45bd-4f76-a163-0f584c4aef27
2025/05/31 03:25:38 🐳 Creating container for image nginx:alpine
2025/05/31 03:25:38 🐳 Creating container for image testcontainers/ryuk:0.11.0
2025/05/31 03:25:38 ✅ Container created: 8e85a41e50db
2025/05/31 03:25:38 🐳 Starting container: 8e85a41e50db
2025/05/31 03:25:38 ✅ Container started: 8e85a41e50db
2025/05/31 03:25:38 ⏳ Waiting for container id 8e85a41e50db image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms skipInternalCheck:false}
2025/05/31 03:25:38 🔔 Container is ready: 8e85a41e50db
2025/05/31 03:25:38 ✅ Container created: c9b1a45072e6
2025/05/31 03:25:38 🐳 Starting container: c9b1a45072e6
2025/05/31 03:25:39 ✅ Container started: c9b1a45072e6
2025/05/31 03:25:39 🔔 Container is ready: c9b1a45072e6
2025/05/31 03:25:39 🐳 Stopping container: c9b1a45072e6
2025/05/31 03:25:49 ✅ Container stopped: c9b1a45072e6
2025/05/31 03:25:49 ✅ Container started: c9b1a45072e6
2025/05/31 03:25:49 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:49 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:49 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:49 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:49 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:50 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:50 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:51 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:51 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:52 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:52 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:53 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:53 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:53 All requested ports were not exposed: port 8080/tcp is not mapped yet
2025/05/31 03:25:54 container logs (all exposed ports, [8080/tcp], were not mapped in 5s: port 8080/tcp is not mapped yet):
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2025/05/31 00:25:39 [notice] 1#1: using the "epoll" event method
2025/05/31 00:25:39 [notice] 1#1: nginx/1.27.5
2025/05/31 00:25:39 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.2.0)
2025/05/31 00:25:39 [notice] 1#1: OS: Linux 5.14.0-503.40.1.el9_5.x86_64
2025/05/31 00:25:39 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 4000000:4000000
2025/05/31 00:25:39 [notice] 1#1: start worker processes
2025/05/31 00:25:39 [notice] 1#1: start worker process 30
2025/05/31 00:25:39 [notice] 1#1: start worker process 31
2025/05/31 00:25:39 [notice] 1#1: start worker process 32
2025/05/31 00:25:39 [notice] 1#1: start worker process 33
2025/05/31 00:25:39 [notice] 1#1: start worker process 34
2025/05/31 00:25:39 [notice] 1#1: start worker process 35
2025/05/31 00:25:39 [notice] 1#1: start worker process 36
2025/05/31 00:25:39 [notice] 1#1: start worker process 37

    reuse_test.go:39:
                Error Trace:    /ws/github/testcontainers/testcontainers-go/reuse_test.go:39
                Error:          Received unexpected error:
                                create container: all exposed ports, [8080/tcp], were not mapped in 5s: port 8080/tcp is not mapped yet
                Test:           TestGenericContainer_stop_start_withReuse
2025/05/31 03:25:54 🐳 Stopping container: c9b1a45072e6
2025/05/31 03:25:54 ✅ Container stopped: c9b1a45072e6
2025/05/31 03:25:54 🐳 Terminating container: c9b1a45072e6
2025/05/31 03:25:54 🚫 Container terminated: c9b1a45072e6
--- FAIL: TestGenericContainer_stop_start_withReuse (15.38s)
FAIL
exit status 1
FAIL    github.com/testcontainers/testcontainers-go     15.404s

Additional information

In the failed test, Testcontainers for Go uses (in order of invocation, only suspected invocations are listed):

1. /containers/{id}/stop Docker Engine API (is referenced below as "container stop API") in testcontainers_test.TestGenericContainer_stop_start_withReuse → testcontainers.Container.Stop → testcontainers.DockerContainer.Stop → client.ContainerAPIClient.ContainerStop → client.Client.ContainerStop.
2. /containers/json Docker Engine API (is referenced below as "container list API") to find container by name in testcontainers.GenericContainer → testcontainers.DockerProvider.ReuseOrCreateContainer → testcontainers.DockerProvider.findContainerByName → client.Client.ContainerList.
   This API call sometimes returns status of container as running (refer to c.State at debugger screenshot from feat(kafka,redpanda): support for waiting for mapped ports without external checks #3165 (comment)) even though previous API call (to stop container) was completed successfully (i.e. status of container cannot be running).

The only issue of Docker Engine "container list API" I found is moby/moby#44512, but it is not connected with "container stop API".

[mabrarov]

Here is the way this issue (seems to be Docker Engine issue) can be reproduced without Testcontainers for Go, but using Docker Engine API only (requires Bash, curl and jq):

container_id="$(docker run -d --name 'my-nginx' 'nginx:alpine')" && \
sleep 10s && \
docker_unix_socket='/var/run/docker.sock' && \
curl_unix_socket_host='http://localhost' && \
containers_list_url="${curl_unix_socket_host}/containers/json?all=1&filters=$(jq -rn --arg x '{"name":{"^my-nginx$":true}}' '$x|@uri')" && \
curl -v --unix-socket "${docker_unix_socket}" -X POST "${curl_unix_socket_host}/containers/${container_id}/stop" \
  --next -v --unix-socket "${docker_unix_socket}" "${containers_list_url}" && \
docker ps -a && \
docker rm -fv "${container_id}"

Output of failed test on my environment (Ubuntu 24.04, Docker Engine 28.2.2):

*   Trying /var/run/docker.sock:0...
* Connected to localhost (/run/docker.sock) port 80
> POST /containers/2544009d337e764ecebf532c076c7265839b16a4efefdbd3af4703e6eb44af69/stop HTTP/1.1
> Host: localhost
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 204 No Content
< Api-Version: 1.50
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/28.2.2 (linux)
< Date: Mon, 02 Jun 2025 15:15:02 GMT
<
* Connection #0 to host localhost left intact
* Found bundle for host: 0x5eb9796d6b60 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection with host localhost
> GET /containers/json?all=1&filters=%7B%22name%22%3A%7B%22%5Emy-nginx%24%22%3Atrue%7D%7D HTTP/1.1
> Host: localhost
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Api-Version: 1.50
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/28.2.2 (linux)
< Date: Mon, 02 Jun 2025 15:15:02 GMT
< Content-Length: 954
<
[{"Id":"2544009d337e764ecebf532c076c7265839b16a4efefdbd3af4703e6eb44af69","Names":["/my-nginx"],"Image":"nginx:alpine","ImageID":"sha256:6769dc3a703c719c1d2756bda113659be28ae16cf0da58dd5fd823d6b9a050ea","Command":"/docker-entrypoint.sh nginx -g 'daemon off;'","Created":1748877291,"Ports":[{"PrivatePort":80,"Type":"tcp"}],"Labels":{"maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"},"State":"running","Status":"Up 10 seconds","HostConfig":{"NetworkMode":"bridge"},"NetworkSettings":{"Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"be:d4:d1:6d:70:a3","DriverOpts":null,"GwPriority":0,"NetworkID":"92101625e849d3de4233a47d66c694654803bdc91a01216335a89b2edb0b7bde","EndpointID":"aa1b095fa28c74b24d285af3b0ced413c3645cf1bf70ccf1089bb1697fb0706d","Gateway":"172.17.0.1","IPAddress":"172.17.0.2","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}},"Mounts":[]}]
* Connection #0 to host localhost left intact
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS                              PORTS     NAMES
2544009d337e   nginx:alpine   "/docker-entrypoint.…"   11 seconds ago   Exited (0) Less than a second ago             my-nginx
2544009d337e764ecebf532c076c7265839b16a4efefdbd3af4703e6eb44af69

Note that Status reported by the 2nd curl request is Up 10 seconds (and State is running) while STATUS reported by "docker ps" command (which runs after curl) is Exited (0) Less than a second ago.

If I give Docker Engine a little bit more time b/w "container stop API" call and "container list API" call by modifying test this way (using 2 curl commands instead of 2 URLs in a single curl command):

container_id="$(docker run -d --name 'my-nginx' 'nginx:alpine')" && \
sleep 10s && \
docker_unix_socket='/var/run/docker.sock' && \
curl_unix_socket_host='http://localhost' && \
containers_list_url="${curl_unix_socket_host}/containers/json?all=1&filters=$(jq -rn --arg x '{"name":{"^my-nginx$":true}}' '$x|@uri')" && \
curl -v -X POST --unix-socket "${docker_unix_socket}" "${curl_unix_socket_host}/containers/${container_id}/stop" && \
curl -v --unix-socket "${docker_unix_socket}" "${containers_list_url}" && \
docker ps -a && \
docker rm -fv "${container_id}"

then test succeeds (no issue happens, status reported by "container list API" matches status reported by "docker ps" command):

*   Trying /var/run/docker.sock:0...
* Connected to localhost (/run/docker.sock) port 80
> POST /containers/53f0b9d24f4ec047c6312d0cf88296907f4e6e70823ca0f866b6e0c1c0303f40/stop HTTP/1.1
> Host: localhost
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 204 No Content
< Api-Version: 1.50
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/28.2.2 (linux)
< Date: Mon, 02 Jun 2025 15:31:36 GMT
<
* Connection #0 to host localhost left intact
*   Trying /var/run/docker.sock:0...
* Connected to localhost (/run/docker.sock) port 80
> GET /containers/json?all=1&filters=%7B%22name%22%3A%7B%22%5Emy-nginx%24%22%3Atrue%7D%7D HTTP/1.1
> Host: localhost
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Api-Version: 1.50
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/28.2.2 (linux)
< Date: Mon, 02 Jun 2025 15:31:36 GMT
< Content-Length: 840
<
[{"Id":"53f0b9d24f4ec047c6312d0cf88296907f4e6e70823ca0f866b6e0c1c0303f40","Names":["/my-nginx"],"Image":"nginx:alpine","ImageID":"sha256:6769dc3a703c719c1d2756bda113659be28ae16cf0da58dd5fd823d6b9a050ea","Command":"/docker-entrypoint.sh nginx -g 'daemon off;'","Created":1748878286,"Ports":[],"Labels":{"maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"},"State":"exited","Status":"Exited (0) Less than a second ago","HostConfig":{"NetworkMode":"bridge"},"NetworkSettings":{"Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"","DriverOpts":null,"GwPriority":0,"NetworkID":"92101625e849d3de4233a47d66c694654803bdc91a01216335a89b2edb0b7bde","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}},"Mounts":[]}]
* Connection #0 to host localhost left intact
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS                              PORTS     NAMES
53f0b9d24f4e   nginx:alpine   "/docker-entrypoint.…"   10 seconds ago   Exited (0) Less than a second ago             my-nginx
53f0b9d24f4ec047c6312d0cf88296907f4e6e70823ca0f866b6e0c1c0303f40

[mabrarov]

Opened Docker Engine issue at moby/moby#50133

[mabrarov]

Opened #3197 to workaround this issue.