Description
I've spent a chunk of time trying to chase down an issue we're seeing on some of our AWS ECS hosts where healthchecks were spuriously failing and causing containers to exit. I noticed from looking at the task state change events that somehow we had containers binding to different ports between IPv4 and IPv6 and all of our issues were caused by a mismatch here.
Rolling back to Docker 20.10.5 fixed this by not publishing IPv6 bindings for our containers at all (essentially undoing the work done in https://github.com/moby/libnetwork/pull/2608/files) but while we probably can avoid using IPv6 bindings either in Docker or in AWS' ECS Agent (According to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html#task-networking-vpc-dual-stack IPv6 shouldn't be happening unless opted into anyway so I think this is a surprise caused by Docker 20.10.6 now showing IPv6 bindings in the API when it didn't previously) but that doesn't escape the fact that there's a bug somewhere in libnetwork (I think, I'm a bit hazy on where responsibilities lie now with the splitting of the projects) that is causing Docker to publish the wrong ports on IPv6.
Running the following command:
xargs -P 0 -I {} docker run --detach --rm --name test-{} -P nginx<<<$(seq 1 100)attempts to launch 100 NGINX containers in parallel which makes my laptop completely fall over and most of the container starts to fail. However when I run this I reliably get port mismatches. Sorting the docker ps output by the IPv4 address shows this well:
| CONTAINER ID | IMAGE | COMMAND | PORTS | NAMES | |
|---|---|---|---|---|---|
| bb9c4984b395 | nginx | /docker-entrypoint.… | 0.0.0.0:49164->80/tcp, | :::49164->80/tcp | test-2 |
| 571041910b22 | nginx | /docker-entrypoint.… | 0.0.0.0:49165->80/tcp, | :::49165->80/tcp | test-6 |
| 6d6379bf998f | nginx | /docker-entrypoint.… | 0.0.0.0:49166->80/tcp, | :::49166->80/tcp | test-4 |
| c6fa6a39c31d | nginx | /docker-entrypoint.… | 0.0.0.0:49167->80/tcp, | :::49167->80/tcp | test-15 |
| 82e49c9bbb0f | nginx | /docker-entrypoint.… | 0.0.0.0:49168->80/tcp, | :::49168->80/tcp | test-11 |
| b903859bdb9c | nginx | /docker-entrypoint.… | 0.0.0.0:49169->80/tcp, | :::49169->80/tcp | test-8 |
| c568a230b5b9 | nginx | /docker-entrypoint.… | 0.0.0.0:49170->80/tcp, | :::49170->80/tcp | test-12 |
| 78d13f523722 | nginx | /docker-entrypoint.… | 0.0.0.0:49171->80/tcp, | :::49171->80/tcp | test-3 |
| aeef1fa41148 | nginx | /docker-entrypoint.… | 0.0.0.0:49172->80/tcp, | :::49172->80/tcp | test-10 |
| 2b9cfea3221c | nginx | /docker-entrypoint.… | 0.0.0.0:49173->80/tcp, | :::49173->80/tcp | test-16 |
| 1174325c7394 | nginx | /docker-entrypoint.… | 0.0.0.0:49174->80/tcp, | :::49174->80/tcp | test-5 |
| 7258d99985b6 | nginx | /docker-entrypoint.… | 0.0.0.0:49175->80/tcp, | :::49175->80/tcp | test-14 |
| 470391662adb | nginx | /docker-entrypoint.… | 0.0.0.0:49176->80/tcp, | :::49176->80/tcp | test-9 |
| 3aa55932f786 | nginx | /docker-entrypoint.… | 0.0.0.0:49177->80/tcp, | :::49177->80/tcp | test-17 |
| fdad706caea0 | nginx | /docker-entrypoint.… | 0.0.0.0:49178->80/tcp, | :::49178->80/tcp | test-13 |
| ac893be13721 | nginx | /docker-entrypoint.… | 0.0.0.0:49179->80/tcp, | :::49179->80/tcp | test-1 |
| d16c3bc3db80 | nginx | /docker-entrypoint.… | 0.0.0.0:49180->80/tcp, | :::49180->80/tcp | test-20 |
| c159beb98784 | nginx | /docker-entrypoint.… | 0.0.0.0:49181->80/tcp, | :::49181->80/tcp | test-19 |
| 0d449991367b | nginx | /docker-entrypoint.… | 0.0.0.0:49182->80/tcp, | :::49182->80/tcp | test-7 |
| 6a99f572659a | nginx | /docker-entrypoint.… | 0.0.0.0:49183->80/tcp, | :::49183->80/tcp | test-18 |
| 6cdc6c1aee92 | nginx | /docker-entrypoint.… | 0.0.0.0:49184->80/tcp, | :::49184->80/tcp | test-21 |
| 9ac258de0f00 | nginx | /docker-entrypoint.… | 0.0.0.0:49185->80/tcp, | :::49185->80/tcp | test-22 |
| a753a81defcc | nginx | /docker-entrypoint.… | 0.0.0.0:49186->80/tcp, | :::49186->80/tcp | test-25 |
| 2f8543fbc1ae | nginx | /docker-entrypoint.… | 0.0.0.0:49187->80/tcp, | :::49187->80/tcp | test-27 |
| bb1a37f274e0 | nginx | /docker-entrypoint.… | 0.0.0.0:49188->80/tcp, | :::49188->80/tcp | test-30 |
| 787d4286597d | nginx | /docker-entrypoint.… | 0.0.0.0:49189->80/tcp, | :::49189->80/tcp | test-24 |
| cfca7dbe2815 | nginx | /docker-entrypoint.… | 0.0.0.0:49191->80/tcp, | :::49191->80/tcp | test-31 |
| 93e001248f3a | nginx | /docker-entrypoint.… | 0.0.0.0:49192->80/tcp, | :::49192->80/tcp | test-28 |
| 723055c393d6 | nginx | /docker-entrypoint.… | 0.0.0.0:49193->80/tcp, | :::49193->80/tcp | test-29 |
| 336a0e93ca3b | nginx | /docker-entrypoint.… | 0.0.0.0:49194->80/tcp, | :::49194->80/tcp | test-26 |
| 6f404ff119b9 | nginx | /docker-entrypoint.… | 0.0.0.0:49195->80/tcp, | :::49195->80/tcp | test-33 |
| 61efecef0103 | nginx | /docker-entrypoint.… | 0.0.0.0:49196->80/tcp, | :::49196->80/tcp | test-35 |
| 1a1633e3563c | nginx | /docker-entrypoint.… | 0.0.0.0:49197->80/tcp, | :::49197->80/tcp | test-32 |
| f00b4f4b3e43 | nginx | /docker-entrypoint.… | 0.0.0.0:49198->80/tcp, | :::49198->80/tcp | test-39 |
| b703b502cb0d | nginx | /docker-entrypoint.… | 0.0.0.0:49199->80/tcp, | :::49199->80/tcp | test-65 |
| 95cd8136cb78 | nginx | /docker-entrypoint.… | 0.0.0.0:49200->80/tcp, | :::49200->80/tcp | test-66 |
| 025aa7cd9e81 | nginx | /docker-entrypoint.… | 0.0.0.0:49201->80/tcp, | :::49201->80/tcp | test-37 |
| 8067402d8030 | nginx | /docker-entrypoint.… | 0.0.0.0:49202->80/tcp, | :::49202->80/tcp | test-41 |
| 1.79769313486232E+308 | nginx | /docker-entrypoint.… | 0.0.0.0:49203->80/tcp, | :::49203->80/tcp | test-40 |
| 8dc601214115 | nginx | /docker-entrypoint.… | 0.0.0.0:49204->80/tcp, | :::49204->80/tcp | test-42 |
| b32b03f7d1e8 | nginx | /docker-entrypoint.… | 0.0.0.0:49205->80/tcp, | :::49205->80/tcp | test-49 |
| e5733efe3ac5 | nginx | /docker-entrypoint.… | 0.0.0.0:49206->80/tcp, | :::49206->80/tcp | test-44 |
| f433ea6d4085 | nginx | /docker-entrypoint.… | 0.0.0.0:49208->80/tcp, | :::49207->80/tcp | test-46 |
| 364e5ecbd186 | nginx | /docker-entrypoint.… | 0.0.0.0:49209->80/tcp, | :::49208->80/tcp | test-43 |
| 2276f9cada2e | nginx | /docker-entrypoint.… | 0.0.0.0:49210->80/tcp, | :::49209->80/tcp | test-36 |
| 9ee5b8acc20b | nginx | /docker-entrypoint.… | 0.0.0.0:49211->80/tcp, | :::49210->80/tcp | test-38 |
| 1cec14c1e094 | nginx | /docker-entrypoint.… | 0.0.0.0:49212->80/tcp, | :::49211->80/tcp | test-50 |
| aa3d7b553cd3 | nginx | /docker-entrypoint.… | 0.0.0.0:49215->80/tcp, | :::49214->80/tcp | test-55 |
| 4593e5efdc49 | nginx | /docker-entrypoint.… | 0.0.0.0:49216->80/tcp, | :::49215->80/tcp | test-56 |
| 5f37baa5f433 | nginx | /docker-entrypoint.… | 0.0.0.0:49217->80/tcp, | :::49216->80/tcp | test-67 |
| 5ade3066ffef | nginx | /docker-entrypoint.… | 0.0.0.0:49218->80/tcp, | :::49217->80/tcp | test-53 |
| 4610422e9af1 | nginx | /docker-entrypoint.… | 0.0.0.0:49220->80/tcp, | :::49219->80/tcp | test-99 |
| 0a2f89796ce0 | nginx | /docker-entrypoint.… | 0.0.0.0:49222->80/tcp, | :::49221->80/tcp | test-51 |
Container f433ea6d4085 is mapped to host port 49208 on IPv4 but mapped to host port 49207 for IPv6. This isn't an issue by itself as the host is essentially listening on two ports for the same container port but if you then look at the next container 364e5ecbd186 which is mapped to host port 49209 for IPv4 and 49208 for IPv6. So requests to this container on IPv6 will instead hit the other container. In this case with the containers all running the same image that's fine but where you have different containers then the request will fail as it will be mapped to the wrong container. The same repeats down the list so that all the next containers are then off by one throughout.