[BUG] `up` incorrectly recreates containers without changes + recreated containers have wrong files inside

[rivit98]

Description

Hi, I have the following problem:

I have a repository where each service has its own separate directory containing a Dockerfile (same for each service) and a file named flag.txt (different for each service, the contents follow the service name). I noticed that after executing docker compose -p test up --build twice, some containers are recreated, but there was no configuration change. What's more - the flag.txt inside the recreated containers is completely different than expected (see the video).

Repository is available here: https://github.com/rivit98/docker-compose-experiments

bug.mp4

Steps To Reproduce

1. git clone https://github.com/rivit98/docker-compose-experiments
2. cd docker-compose-experiments
3. docker compose -p test up --build
4. stop created containers with ctrl+c
5. docker compose -p test up --build - observe that some containers were recreated even if there were no changes in configuration/files/etc. If there were no recreated containers go to step 4.
6. find * -maxdepth 0 -type d | xargs -I{} bash -c "echo -n 'service: {} '; docker compose -p test exec -T {} cat /flag.txt; echo" - observe that containers that were recreated have wrong flag.txt file.

Compose Version

Docker Compose version v2.29.1

Docker Environment

Client: Docker Engine - Community
 Version:    27.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 14
  Running: 14
  Paused: 0
  Stopped: 0
 Images: 19
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-39-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 961.6MiB
 Name: localhost
 ID: 204f59ae-1ecc-436e-b7ab-d883414c3c73
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[idsulik]

Hi! Couldn't reproduce it on main branch, on v2.29.1 tag and using version v2.28.1.
always ok: 9, not_ok: 0

[rivit98]

Hi @idsulik.
I set up a new VM to redo the testing, and it turns out I can reproduce the issue on a clean Ubuntu 24.04 with the same Docker and Docker Compose versions as stated in the report.

[jhrotko]

@rivit98 could you try to reproduce with the latest version?

[rivit98]

@jhrotko
yes, I can reproduce it on fresh VM with the following docker and docker-compose versions:

root@localhost:~/docker-compose-experiments# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.1 LTS"

root@localhost:~/docker-compose-experiments# docker compose version
Docker Compose version v2.29.7

root@localhost:~/docker-compose-experiments# docker info
Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 9
  Running: 9
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-45-generic
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.922GiB
 Name: localhost
 ID: c042fe84-c641-4d8d-887d-c6f00d7aa127
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[EkkoG]

Another demo reproduce this bug

root@vultr:~# docker compose version
Docker Compose version v2.29.7
root@vultr:~# docker version
Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:11 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
root@vultr:~# cat docker-compose.yml 
services:
  sdk:
    #image: openwrt/sdk:x86_64
    image: openwrt/sdk:x86-64-openwrt-23.05
    command: cat feeds.conf.default
      
root@vultr:~# docker compose up
[+] Running 1/1
 ✔ Container root-sdk-1  Created                                                                                                                                                                  17.1s 
Attaching to sdk-1
sdk-1  | src-git-full base https://git.openwrt.org/openwrt/openwrt.git;openwrt-23.05
sdk-1  | src-git packages https://git.openwrt.org/feed/packages.git;openwrt-23.05
sdk-1  | src-git luci https://git.openwrt.org/project/luci.git;openwrt-23.05
sdk-1  | src-git routing https://git.openwrt.org/feed/routing.git;openwrt-23.05
sdk-1  | src-git telephony https://git.openwrt.org/feed/telephony.git;openwrt-23.05
sdk-1 exited with code 0
root@vultr:~# vim docker-compose.yml 
root@vultr:~# cat docker-compose.yml 
services:
  sdk:
    image: openwrt/sdk:x86_64
      #image: openwrt/sdk:x86-64-openwrt-23.05
    command: cat feeds.conf.default
      
root@vultr:~# docker compose up
[+] Running 1/0
 ✔ Container root-sdk-1  Recreated                                                                                                                                                                 0.1s 
Attaching to sdk-1
sdk-1  | src-git-full base https://git.openwrt.org/openwrt/openwrt.git;openwrt-23.05
sdk-1  | src-git packages https://git.openwrt.org/feed/packages.git;openwrt-23.05
sdk-1  | src-git luci https://git.openwrt.org/project/luci.git;openwrt-23.05
sdk-1  | src-git routing https://git.openwrt.org/feed/routing.git;openwrt-23.05
sdk-1  | src-git telephony https://git.openwrt.org/feed/telephony.git;openwrt-23.05
sdk-1 exited with code 0
root@vultr:~# docker compose down
[+] Running 2/2
 ✔ Container root-sdk-1  Removed                                                                                                                                                                   0.0s 
 ✔ Network root_default  Removed                                                                                                                                                                   0.2s 
root@vultr:~# docker compose up
[+] Running 2/2
 ✔ Network root_default  Created                                                                                                                                                                   0.1s 
 ✔ Container root-sdk-1  Created                                                                                                                                                                  19.0s 
Attaching to sdk-1
sdk-1  | src-git-full base https://git.openwrt.org/openwrt/openwrt.git
sdk-1  | src-git packages https://git.openwrt.org/feed/packages.git
sdk-1  | src-git luci https://git.openwrt.org/project/luci.git
sdk-1  | src-git routing https://git.openwrt.org/feed/routing.git
sdk-1  | src-git telephony https://git.openwrt.org/feed/telephony.git
sdk-1  | #src-git video https://github.com/openwrt/video.git
sdk-1  | #src-git targets https://github.com/openwrt/targets.git
sdk-1  | #src-git oldpackages http://git.openwrt.org/packages.git
sdk-1  | #src-link custom /usr/src/openwrt/custom-feed
sdk-1 exited with code 0