[Bug] - Yum not respecting proxy_username/proxy_password in dnf.conf

[pandel]

Describe the bug
It is not possible to build a Docker image based on amazonlinux:2023 behind an authentication proxy. Yum (curl) always returns a HTTP 407 error, regardless of its configuration.

The proxy address itself is reachable inside the container (tested) and also seems to be read by yum, because I can provoke a connection error message when I misconfigure the proxy URL, but the authentication mechanism doesn't seem to work. Sadly, I cannot avoid the proxy authentication in any case.

To Reproduce
To reproduce the error you need to connect to the Internet via an authentication proxy and add the following to the Dockerfile before the first yum install command is being used:

RUN echo "proxy=http://<proxyserverurl>:<proxyport>" >> /etc/dnf/dnf.conf &&\
    echo "proxy_username=<proxyuser>" >> /etc/dnf/dnf.conf &&\
    echo "proxy_password=<proxypass>" >> /etc/dnf/dnf.conf

Expected behavior
According to this AWS re:Post documentation yum install should connect to the Internet using the proxy now.

Reported error

#8 1.421 error: Curl error (56): Failure when receiving data from the peer for https://cdn.amazonlinux.com/al2023/core/mirrors/2023.6.20250203/x86_64/mirror.list [CONNECT tunnel failed, response 407] (https://cdn.amazonlinux.com/al2023/core/mirrors/2023.6.20250203/x86_64/mirror.list).
#8 1.422 Amazon Linux 2023 repository                    0.0  B/s |   0  B     00:00
#8 1.422 Errors during downloading metadata for repository 'amazonlinux':
#8 1.422   - Curl error (56): Failure when receiving data from the peer for https://cdn.amazonlinux.com/al2023/core/mirrors/2023.6.20250203/x86_64/mirror.list [CONNECT tunnel failed, response 407]
#8 1.423 Error: Failed to download metadata for repo 'amazonlinux': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://cdn.amazonlinux.com/al2023/core/mirrors/2023.6.20250203/x86_64/mirror.list [CONNECT tunnel failed, response 407]

In my case, I do have to use this proxy inside my Dockerfile, as I don't have any possibility to directly connect to the Internet.

What can be done?

Regards,
Holger

[zcobol]

@pandel this looks like a Docker issue. Try adding the proxy as an argument to the docker build command instead.

[pandel]

@zcobol Please not this kind of "the OP's not knowing what he's doing" assumption-based kind of hints! I do know about build args and ENV vars, Docker daemon overrides regarding proxy settings or modifications to the daemon init files regarding proxy settings for its environment, blablablabla... this is not what this here is about!

I build Docker images for a lot of years now behind authentication proxies! No Debian, Ubuntu, Alpine or whatever not Yum-based Linux is making any problems here. This is the first time I sadly have to use (Amazonlinux) a Yum-based distro and this happens ootb and a long Google search this afternoon revealed that Yum has had enough proxy problems throughout the past.

Additionally, there are some simple curl commands inside the Dockerfile I am referring to that do work nicely with the proxy as long as Yum does not come into play!

So, it is definitely a Yum problem, which might have to do with it being executed inside of a Docker environment, but then it is still a Yum problem.

[stewartsmith]

Interesting....

So, AL2023 uses dnf rather than yum (there's a symlink from yum to dnf for compatibility), which uses a completely different software stack than yum does.

If possible to do so, can you check to see if the same problem is present in current Fedora?

[pandel]

Yes it is. Here is what I did...

How to provide proxy server information
In all of my tests, I provided the proxy server information in the following ways

• as docker build --build-arg https_proxy=... --build-arg http_proxy=... ...
• as part of the Dockerfile as follows:

RUN echo "proxy=http://<proxyserverurl>:<proxyport>" >> /etc/dnf/dnf.conf &&\
    echo "proxy_username=<proxyuser>" >> /etc/dnf/dnf.conf &&\
    echo "proxy_password=<proxypass>" >> /etc/dnf/dnf.conf

Docker base images used
Following images have been used in my tests

• Fedora 37 (as I read, that Amazonlinux 2023 is partly based on it)
• Fedora 42

Example Dockerfile used

FROM fedora:42

ARG S6_VERSION="v2.2.0.3"

RUN echo "proxy=http://proxy.server.local:8080" >> /etc/dnf/dnf.conf &&\
    echo "proxy_username=proxyuser" >> /etc/dnf/dnf.conf &&\
    echo "proxy_password=proxypass" >> /etc/dnf/dnf.conf

RUN curl --fail -L https://github.com/just-containers/s6-overlay/releases/download/${S6_VERSION}/s6-overlay-amd64.tar.gz \
    -o /tmp/s6-overlay-amd64.tar.gz && \
    ls -l /tmp && \
    tar xzf /tmp/s6-overlay-amd64.tar.gz -C / --exclude="./bin" && \
    tar xzf /tmp/s6-overlay-amd64.tar.gz -C /usr ./bin && \
    rm  /tmp/s6-overlay-amd64.tar.gz

RUN yum install curl-minimal xz gnupg tar gzip openssl findutils procps -y &&\
    yum clean all

ENTRYPOINT [ "/init" ]

Test results
Additionally I tested both versions without any proxy information at all, which led to a hanging build process <- pretty clear if there's no way to reach the Internet at all.

• Fedora 37:

#6 [3/4] RUN curl --fail -L https://github.com/just-containers/s6-overlay/releases/download/v2.2.0.3/s6-overlay-amd64.tar.gz     -o /tmp/s6-overlay-amd64.tar.gz &&     ls -l /tmp &&     tar xzf /tmp/s6-overlay-amd64.tar.gz -C / --exclude="./bin" &&     tar xzf /tmp/s6-overlay-amd64.tar.gz -C /usr ./bin &&     rm  /tmp/s6-overlay-amd64.tar.gz
#6 0.320   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#6 0.320                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1808k  100 1808k    0     0  1858k      0 --:--:-- --:--:-- --:--:-- 1858k
#6 1.299 total 1812
#6 1.299 -rw-r--r-- 1 root root 1851478 Feb 20 00:34 s6-overlay-amd64.tar.gz
#6 DONE 1.5s

#7 [4/4] RUN yum install curl-minimal xz gnupg tar gzip openssl findutils procps -y &&    yum clean all
#7 1.549 Fedora 37 - x86_64                              0.0  B/s |   0  B     00:01
#7 1.549 Errors during downloading metadata for repository 'fedora':
#7 1.549   - Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=fedora-37&arch=x86_64 [Received HTTP code 407 from proxy after CONNECT]
#7 1.559 Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=fedora-37&arch=x86_64 [Received HTTP code 407 from proxy after CONNECT]

• Fedora 42:

#6 [3/4] RUN curl --fail -L https://github.com/just-containers/s6-overlay/releases/download/v2.2.0.3/s6-overlay-amd64.tar.gz     -o /tmp/s6-overlay-amd64.tar.gz &&     ls -l /tmp &&     tar xzf /tmp/s6-overlay-amd64.tar.gz -C / --exclude="./bin" &&     tar xzf /tmp/s6-overlay-amd64.tar.gz -C /usr ./bin &&     rm  /tmp/s6-overlay-amd64.tar.gz
#6 0.322   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#6 0.322                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1808k  100 1808k    0     0  2045k      0 --:--:-- --:--:-- --:--:-- 3293k
#6 1.212 total 1812
#6 1.212 -rw-r--r-- 1 root root 1851478 Feb 20 00:36 s6-overlay-amd64.tar.gz
#6 DONE 1.4s

#7 [4/4] RUN yum install curl-minimal xz gnupg tar gzip openssl findutils procps -y &&    yum clean all
#7 0.336 Updating and loading repositories:
#7 0.926  Fedora - Rawhide - Developmental packa ???% |   0.0   B/s |   0.0   B |  00m01s
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.
#7 0.926 >>> Librepo error: Cannot prepare internal mirrorlist: Curl error (66): Failed tFailed to download metadata (metalink: "https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64") for repository "rawhide"
#7 0.926  Librepo error: Cannot prepare internal mirrorlist: Curl error (66): Failed to initialise SSL crypto engine for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 [Failed to find SSL backend for endpoint]

Conclude
As you see, both tests show that curlon its own works flawlessly given the proxy address via --build-arg, but both versions show, that yum or dnf or what the heck this distro uses is not able to use proxy information at all, regardless of being specified via dnf.conf or environment variables.

But it seems, the error is different between the Fedora releases:

• in F37 you can see that yum tries to connect via proxy, but can't authenticate and returns a 407
• in F42 somehow yumis using its curl backend totally wrong

[elsaco]

@pandel according to man dnf.conf if no proxy is defined in dnf.conf or set to _none_ then the environment variables for curl are being used instead:

Note: The curl environment variables (such as http_proxy) are effective if this  option  is  unset  (or
'_none_' is set in the repository configuration). See the curl man page for details.

You mention that curl works in your Dockefile, so please try docker build w/out setting yum proxies. It might work, didn't try it!

[pandel]

@elsaco I already did before, but this led to yum not working with the same error. I just used dnf.conf because it was mentioned in the official AmazonLinux FAQ and hoped it would fix that.

[zcobol]

There's a quick test you can do and find out if dnf works with your proxy settings. Start a Docker container, not 2023-minimal since it doesn't have a package manager, add your proxy settings and run a yum/dnf command. Both yum & dnf are just links to dnf-3 in AL2023.

[pandel]

@zcobol You did not read a single bit of my long test result post, right? I did as you suggested and more already...

[zcobol]

@pandel Sorry you didn't like my suggestion. IMO, it's easier to troubleshoot with a running container then watching it fail at container buildup. If you have access to the proxy inspect the messages there also. A 407 response from the proxy is clear that the issue is with the settings.

[stewartsmith]

Yes it is. Here is what I did...

Thanks for testing it on Fedora!

For background on versioning, our rpm/dnf stack in AL2023 is currently close to the CentOS 9 Stream versions, which all started off around what Fedora 34 had and have then evolved a bit over time (different paces for different components). With Fedora switching to DNF5 more recently, this involves larger changes in dnf and libdnf, but the backend of talking to repositories (librepo) remains common.

Although with different build options, currently: CentOS 9 Stream is on 1.14.5 as is AL2023, upstream is at 1.19.0 (as is all current Fedora versions).

One change that caught my eye quickly looking through the interim librepo releases is in 1.18.0 related to URL encoding of username and password, with new options that came in as part of the rpm-software-management/librepo#316 pull request, with fixes in dnf in rpm-software-management/dnf#2036, and rpm-software-management/libdnf#1669 (rpm-software-management/libdnf#1668 is the fix) for libdnf. With rpm-software-management/libdnf@5449211 being merged into libdnf 0.73.3. There's also this dnf issue which is awfully similar, but should be fixed by the PRs mentioned here.

If we look at Fedora 37's libdnf, it's 0.72.0, before these fixes, although Fedora 41 should have the fixes as 0.73.3 has been there since August 2024.

So... that could explain the F37 and AL2023 issue (assuming there's URL encoding fun going on, you do not have to mention anything about a real password of course), but it doesn't explain the F41 behavior unless your container image is really old.

Next idea... Amazon Linux 2023 by default ships with a minimal build of curl and libcurl to minimize space and how many other libraries are dragged into the default images. This is something part of the Fedora curl spec file as well, we just choose different defaults for how we ship our images. One possible interesting thing is the disabling of ntlm auth for (curl|libcurl)-minimal. By any chance does your proxy use NTLM auth?

[zcobol]

For what is worth AL2023 dnf does work over a proxy. Sample access log:

1740078812.357    140 192.168.50.8 TCP_TUNNEL/200 6407 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/18.172.170.7 -
1740078812.431     70 192.168.50.8 TCP_TUNNEL/200 10039 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/18.172.170.7 -
1740078813.371    933 192.168.50.8 TCP_TUNNEL/200 4825582 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/18.172.170.7 -
1740078813.371    934 192.168.50.8 TCP_TUNNEL/200 27732713 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/18.172.170.7 -
1740078813.371    933 192.168.50.8 TCP_TUNNEL/200 544825 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/18.172.170.7 -
1740078972.428    149 192.168.50.8 TCP_TUNNEL/200 6407 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/3.163.24.99 -
1740078972.478     48 192.168.50.8 TCP_TUNNEL/200 10040 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/3.163.24.99 -
1740078973.372    888 192.168.50.8 TCP_TUNNEL/200 4825511 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/3.163.24.99 -
1740078973.372    887 192.168.50.8 TCP_TUNNEL/200 27729568 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/3.163.24.99 -
1740078973.372    887 192.168.50.8 TCP_TUNNEL/200 544445 CONNECT cdn.amazonlinux.com:443 - HIER_DIRECT/3.163.24.99 -

squid proxy server was used for testing.

[pandel]

@zcobol
Sorry for being rude, but I tried to be as detailed as possible in my answer and your suggestion just seemed to me as you did not read it completely before answering. And this is something I totally dislike, sorry for that...

@stewartsmith
Many thanks for your decent research. As I am not familiar with the whole RPM/Yum/Dnf based stuff it was really interesting to read.

I made some progress today investigating the whole thing:

As I wasn't able to build the image based on amazonlinux:2023 behind my proxy, I built it on another machine and transferred the image over to my server, so that the image is absolutely fresh. Then I ran a test container (simple bash command as entrypoint) with this image and tested different things inside again:

• I know now, that dnfbehaves the same if the proxy is set via dnf.conf or https_proxy environment variables, the result is always the same
• a simple curl command followed by a random image URL downloads the image perfectly fine
• a dnf install git results in the known error

-> So, it is irrelevant if the commands are being run during build phase or in a running container (@zcobol)

@stewartsmith
Your hint regarding NTLM might get us in the right direction! Under normal circumstances, the proxy first wants NTLM (Kerberos), which is the standard for all our normal Windows workplace PCs, but it is configured to fallback to basic auth, if a client does not support it, which is the case with most of our other servers and special network components, like printers and such. What I don't understand here is why a single curl command inside the container works, but if it is being used by dnf it does not! Does dnf somehow "strip down" the used libraries when it is using curl from inside its own routines?

@zcobol
Sadly, I don't have access to anything regarding the proxy server itself, because it is hosted in our data center which we are connected to. I can only try to contact a technician and hope for the best... and that's what we come to next :D

Here comes the main progress of today:
I was lucky to get in contact with a data center technician (one of the guys who manage the proxy), who agreed during my tests to temporarily disable authentication for this machine only and a single test run (!) and suddenly, without having to specify proxyuser and proxypassword, dnf worked totally fine and I was able to build the image behind my proxy!

So, it must have something to do with how authentication against this proxy seems to be handled by the distro and yes, NTLM to basic auth fallback might be the main reason here.

[danie-dejager]

@pandel Is a complex password with various different characters? Sometimes, with some software, I have to escape password characters. That version of dnf is still based on Python 3. Could you try a longer password that contains limited characters like aword-aword-aword1 and see if that makes a difference?

[pandel]

Sorry for the late answer... No, it has nothing to do with the password. If you would have read what I already wrote, then you would have read that it is working under normal conditions, with other containers on other OSs, but simply not with yum/dnf. And no, I will not change the password for a test that isn't worth it...