Nomad with HTTP Proxy

[udu68767687iuiu]

Nomad version

Output from nomad version
Nomad v1.4.2 (039d70e)

Operating system and Environment details

Rocky 9.4 on Bare metal server
Nomad client hosted on this server.
Server has a public interface and a private interface.

Issue

I have setup a squid proxy through which I proxy all my HTTP/HTTPS traffic from my containers to the internet.
My containers use the proxy without a problem using an environment variable HTTPS_PROXY=myserverip:port.
Right now I also want to proxify the HTTP/HTTPS coming from the server host to the internet.
So I deploy the env variables on the system using files /etc/environment and /etc/profile.
It works as intended, when trying to curl an external http/https website it goes through squid proxy except for the nomad process that doesn't use the proxy but uses the public interface.
During my tests I blocked HTTP/HTTPS traffic on the public interface, so when I tried to launch a new nomad job, the job failed because it could not pull the docker container image from the remote registry.
That is how I saw that the traffic was going through the public interface, to further confirm it I did a tcpdump on the public interface and I caught the traffic going to the remote registry.
With this I am sure that nomad ignores the HTTPS_PROXY variable.
Is there a specific configuration that I am unaware of ?

Reproduction steps

So I used the HTTP_PROXY and HTTPS_PROXY variables inside the files /etc/environment and /etc/profile to deploy my proxy variables and make them persist on the system.
/etc/profile

https_proxy=http://serverip:19000
http_proxy=http://serverip:19000

/etc/environment

export https_proxy=http://serverip:19000
export http_proxy=http://serverip:19000

After that I source these files to update the system configuration, and I also restart the nomad service.

source /etc/profile
source /etc/environment
systemctl restart nomad

After that I try to launch a nomad job with a docker image not in the local images.

Expected Result

The docker image is pulled by nomad and the job is running.

Actual Result

The docker image is not pulled from the remote registry.
When the docker pull is done in the CLI it goes through squid as intended.

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[jrasell]

Hi @udu68767687iuiu and thanks for raising this issue.

I don't believe systemd will be spawning the Nomad process with the environment populated from your current user shell. To set environment variables for the process, systemd provides a number of environment configuration options exposed by the unit file which include:

• EnvironmentFile that allows environment variables to be read from a text file.
• Environment that specifies inline variables to expose to the process.

I'll keep this issue open for a couple of days if you have any followup questions.

[udu68767687iuiu]

Hello jrasell, thank you for your answer. I forgot to specify that I tried both these methods with the systemd service. But I reached the same results as described in my previous post using the same tests. I don't know if there is another workaround to enforce nomad to use proxy.

[jrasell]

Hi @udu68767687iuiu; I wonder if it is a case of configuring Docker to use a proxy? The Nomad driver is using the Docker client for its work and I don't see anything specific in the code in this area.

[udu68767687iuiu]

Hi @jrasell ,
Well my case is even weirder because I already set my docker daemon.json with the following configuration :

{
  "data-root": "dockerfolder",
  "proxies":
  {
      "http-proxy":"http://proxyaddress:10000",
      "https-proxy":"http://proxyaddress:10000",
      "no-proxy": "10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,127.0.0.1,localhost,.localdomain"
  }
}

When I try a docker run from the CLI it works, but when I try to run a job from nomad that needs to pull an image from a registry then it fails.
I will try to reproduce my setup again from scratch and keep you informed of the status.