Skip to content

Commit da72e2b

Browse files
adrianrioboclaude
andcommitted
fix(gitlab): fix nested-container DNS by propagating netns and mounting containers.conf
Root cause (confirmed live on s390x runner): podman build RUN steps run inside a nested container created by inner Podman inside the docker executor container. Inner Podman (using Netavark) creates a new bridge for these build-step containers, but the privileged-but-bridge-networked outer container has no working iptables/ nftables NAT, so packets from the 10.88.0.0/16 bridge subnet can never reach the DNS servers. The inner containers' /etc/resolv.conf had the correct nameservers written but DNS queries timed out because they could not be routed. The previous chmod 644 and dns_servers fixes correctly configured the outer executor container but did not propagate into the nested build-step containers. Fixes applied: 1. Add netns = "host" to /etc/containers/containers.conf. When the file is bind-mounted into the executor container (see #3), inner Podman reads this and creates all build-step containers sharing the outer container's network namespace instead of a new bridge. They inherit the working resolv.conf with no NAT required. 2. Guarantee /etc/containers/containers.conf exists before runner registration even when DNS detection finds no upstream servers, so the bind mount in #3 always has a real file to attach to. 3. Add --docker-volumes to the runner registration so that every executor container gets the host's containers.conf bind-mounted at /etc/containers/containers.conf:ro. This makes the nested inner Podman pick up both dns_servers and netns = "host" automatically, without any change to the CI job Dockerfile or podman build invocation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent b9f6434 commit da72e2b

1 file changed

Lines changed: 51 additions & 5 deletions

File tree

pkg/integrations/gitlab/snippet-linux.sh

Lines changed: 51 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@ if [ -n "$_dns_servers" ]; then
4343
done
4444
sudo mkdir -p /etc/containers
4545
if [ ! -f /etc/containers/containers.conf ]; then
46-
printf '[containers]\ndns_servers = [%s]\n' "$_toml_list" \
47-
| sudo tee /etc/containers/containers.conf > /dev/null
46+
printf '[containers]\ndns_servers = [%s]\ndns_options = ["timeout:2", "attempts:5", "single-request"]\n' \
47+
"$_toml_list" | sudo tee /etc/containers/containers.conf > /dev/null
4848
elif grep -q '^\[containers\]' /etc/containers/containers.conf; then
4949
# Scope the dns_servers check to the [containers] section only
5050
if awk '/^\[containers\]/{f=1;next} /^\[/{f=0} f && /^dns_servers/{found=1} END{exit !found}' \
@@ -60,10 +60,28 @@ if [ -n "$_dns_servers" ]; then
6060
sudo sed -i "/^\[containers\]/a dns_servers = [${_toml_list}]" \
6161
/etc/containers/containers.conf
6262
fi
63+
# Add or update dns_options within [containers]
64+
if grep -q '^dns_options' /etc/containers/containers.conf; then
65+
sudo sed -i 's|^dns_options.*|dns_options = ["timeout:2", "attempts:5", "single-request"]|' \
66+
/etc/containers/containers.conf
67+
else
68+
sudo sed -i '/^\[containers\]/a dns_options = ["timeout:2", "attempts:5", "single-request"]' \
69+
/etc/containers/containers.conf
70+
fi
6371
else
64-
printf '\n[containers]\ndns_servers = [%s]\n' "$_toml_list" \
65-
| sudo tee -a /etc/containers/containers.conf > /dev/null
72+
printf '\n[containers]\ndns_servers = [%s]\ndns_options = ["timeout:2", "attempts:5", "single-request"]\n' \
73+
"$_toml_list" | sudo tee -a /etc/containers/containers.conf > /dev/null
6674
fi
75+
# Ensure the file is world-readable so rootless Podman can also load it
76+
sudo chmod 644 /etc/containers/containers.conf
77+
fi
78+
79+
# Guarantee the file exists even when DNS detection found nothing, so that the
80+
# volume mount added to the runner below always has a real file to bind.
81+
sudo mkdir -p /etc/containers
82+
if [ ! -f /etc/containers/containers.conf ]; then
83+
printf '[containers]\n' | sudo tee /etc/containers/containers.conf > /dev/null
84+
sudo chmod 644 /etc/containers/containers.conf
6785
fi
6886

6987
{{- if .LogToJournald}}
@@ -90,8 +108,35 @@ else
90108
printf '\n[containers]\nlog_driver = "journald"\n' \
91109
| sudo tee -a /etc/containers/containers.conf > /dev/null
92110
fi
111+
sudo chmod 644 /etc/containers/containers.conf
93112
{{- end}}
94113

114+
# Set netns = "host" so inner Podman (running inside the CI executor container)
115+
# uses the outer container's network namespace for nested build-step containers.
116+
# Without this, Netavark creates a bridge for inner containers but that bridge
117+
# has no working NAT inside a privileged non-host-network container, causing DNS
118+
# and internet access to fail in every podman build RUN step.
119+
if [ ! -f /etc/containers/containers.conf ]; then
120+
printf '[containers]\nnetns = "host"\n' \
121+
| sudo tee /etc/containers/containers.conf > /dev/null
122+
elif grep -q '^\[containers\]' /etc/containers/containers.conf; then
123+
if awk '/^\[containers\]/{f=1;next} /^\[/{f=0} f && /^netns/{found=1} END{exit !found}' \
124+
/etc/containers/containers.conf; then
125+
awk '/^\[containers\]/{s=1} /^\[/ && !/^\[containers\]/{s=0}
126+
s && /^netns/{$0="netns = \"host\""} 1' \
127+
/etc/containers/containers.conf \
128+
| sudo tee /etc/containers/containers.conf.tmp > /dev/null \
129+
&& sudo mv /etc/containers/containers.conf.tmp /etc/containers/containers.conf
130+
else
131+
sudo sed -i '/^\[containers\]/a netns = "host"' \
132+
/etc/containers/containers.conf
133+
fi
134+
else
135+
printf '\n[containers]\nnetns = "host"\n' \
136+
| sudo tee -a /etc/containers/containers.conf > /dev/null
137+
fi
138+
sudo chmod 644 /etc/containers/containers.conf
139+
95140
# Register runner using docker executor backed by Podman
96141
# --docker-privileged is required for Podman: containers need CAP_SYS_ADMIN to mount /proc
97142
sudo gitlab-runner register \
@@ -102,7 +147,8 @@ sudo gitlab-runner register \
102147
--executor "docker" \
103148
--docker-image "fedora:latest" \
104149
--docker-host "unix:///run/podman/podman.sock" \
105-
--docker-privileged
150+
--docker-privileged \
151+
--docker-volumes "/etc/containers/containers.conf:/etc/containers/containers.conf:ro"
106152

107153
{{- if not .Unsecure}}
108154
# Create a dedicated system user for running CI jobs

0 commit comments

Comments
 (0)