Let's Encrypt in Docker - docker.env: no such file or directory

[nemanjam]

When I try to enable letsencrypt it can't find docker.env file although it exists, I checked manually. Here is the error log.

ubuntu@arm1:~$ dokku letsencrypt:enable nextjs-app
=====> Enabling letsencrypt for nextjs-app
-----> Enabling ACME proxy for nextjs-app...
       ok: run: nginx: (pid 18034) 3421s
-----> Getting letsencrypt certificate for nextjs-app via HTTP-01
        - Domain 'nextjs-app.dokku.arm1.localhost3002.live'
docker: open /home/ubuntu/traefik-proxy/apps/dokku/dokku-data/home/dokku/nextjs-app/letsencrypt/certs/ac00fb3b1783f8750bfd5ca350e514d4918ca459/docker.env: no such file or directory.
See 'docker run --help'.
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for nextjs-app...
       ok: run: nginx: (pid 18034) 3421s
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

It's an empty file but it exists, here it is from the container:

ubuntu@arm1:~/traefik-proxy$ docker exec -it dokku bash
root@2c3660d832dd:/tmp# cat /home/dokku/nextjs-app/letsencrypt/certs/ac00fb3b1783f8750bfd5ca350e514d4918ca459/docker.env
root@2c3660d832dd:/tmp# ls -la /home/dokku/nextjs-app/letsencrypt/certs/ac00fb3b1783f8750bfd5ca350e514d4918ca459/docker.env
-rwxr-xr-x 1 dokku dokku 0 Feb 25 17:20 /home/dokku/nextjs-app/letsencrypt/certs/ac00fb3b1783f8750bfd5ca350e514d4918ca459/docker.env

And here it is from the host:

ubuntu@arm1:~/traefik-proxy$ ls -la /home/ubuntu/traefik-proxy/apps/dokku/dokku-data/home/dokku/nextjs-app/letsencrypt/certs/ac00fb3b1783f8750bfd5ca350e514d4918ca459/docker.env
-rwxr-xr-x 1 200 200 0 Feb 25 17:20 /home/ubuntu/traefik-proxy/apps/dokku/dokku-data/home/dokku/nextjs-app/letsencrypt/certs/ac00fb3b1783f8750bfd5ca350e514d4918ca459/docker.env

sudo vi /home/dokku/nextjs-app/letsencrypt/certs/ac00fb3b1783f8750bfd5ca350e514d4918ca459/docker.env

I use this docker-compose.yml:

https://github.com/nemanjam/traefik-proxy/blob/main/apps/dokku/docker-compose.yml

services:
  dokku:
    container_name: dokku
    # image: dokku/dokku:0.30.1
    build:
      context: .
      # install pack in Dockerfile
      dockerfile: Dockerfile
    ports:
      - '3022:22'
    environment:
      - DOKKU_HOSTNAME=dokku.${SERVER_HOSTNAME}
      - DOKKU_HOST_ROOT=${PWD}/dokku-data/home/dokku
    volumes:
      - ${PWD}/dokku-data:/mnt/dokku
      - ${PWD}/plugin-list:/mnt/dokku/plugin-list
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - proxy

dokku report nextjs-app: https://gist.github.com/nemanjam/1e66aa8683ea3535fe1a1ea1848f1dab

dokku ps:inspect nextjs-app: https://gist.github.com/nemanjam/629767d02b5493b8eeb42ee9171d8f55

~/traefik-proxy/apps/dokku$ tree -da .: https://gist.github.com/nemanjam/6437f96ed522fb812e45ea5231ebe05a

[nemanjam]

I've tried exposing Dokku container directly without Traefik but I get exact same error, its not the reason.

    ports:
      - '3022:22'
      - '443:443'
      - '80:80'

[feng-zh]

Get the same issue when running dokku inside of container.

By looking into the code for build docker run command:

dokku-letsencrypt/internal-functions

Lines 42 to 48 in 5dc9bf5

	docker run --rm \
	--env-file "$host_config_dir/docker.env" \
	--user $DOKKU_UID:$DOKKU_GID \
	-v "$host_config_dir:/certs" \
	-v "$DOKKU_LIB_ROOT/data/letsencrypt/$APP:/webroot" \
	"${PLUGIN_IMAGE}:${PLUGIN_IMAGE_VERSION}" \
	"${config[@]}" run | sed "s/^/ /"

The --env-file parameter is using host folder, not the folder inside of dokku container.

Because of docker run requires loading this env file to construct docker API call, before sending request to docker daemon, it get "no such file or directory" error. --env-file argument is not like volume mount, should use dokku side file path.

My current workaround is to exec into dokku container, and create symbo-link folder to let dokku container docker client can read file with same location in host level. This may recreate again after dokku container recreated (like upgrading).

[paschaldev]

@feng-zh Care to share code for symlink inside the dokku container. Currently stuck with problem

[feng-zh]

@feng-zh Care to share code for symlink inside the dokku container. Currently stuck with problem

@paschaldev this will depend on what host folder you are mapping to container dokku folder "/mnt/dokku".

Here is the example assume you are mapping host folder "/path/to/host/dokku-data" into dokku container "/mnt/dokku" folder, then you need;

# inside of dokku container
mkdir -p /path/to/host
cd /path/to/host
ln -s /mnt/dokku dokku-data

# verify the dokku folder is found by using host folder path inside of dokku container
ls /path/to/host/dokku-data/home/dokku

[akvadrako]

This workaround helped me get further, but then another issues crops up:

acme: error presenting token: could not create required directories in webroot for HTTP challenge: mkdir /webroot/.well-known: permission denied

Seems like dokku-letsencrypt is not really tested with docker. The issue seems to be that letsencrypt is trying to mount the container's data directory, not the host data directory.

The workaround is to make a symlink on the host like this (assuming /opt/dokku is the /mnt/dokku source):

ln -s /opt/dokku/var/lib/dokku /var/lib/dokku

[atomicptr]

Setting the symlink I get

=====> Enabling letsencrypt for APP
-----> Enabling ACME proxy for APP...
       ok: run: nginx: (pid 70) 630555s
-----> Getting letsencrypt certificate for APP via HTTP-01
        - Domain 'DOMAIN'
2024/09/17 14:00:01 No key found for account EMAIL. Generating a P256 key.
2024/09/17 14:00:01 Could not check/create directory for account EMAIL: mkdir /certs/accounts: permission denied
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for APP...
       ok: run: nginx: (pid 70) 630556s
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

After that I made these directories writable from host:

(Dont actually use 777 here, I'm just debugging)

chmod 777 /var/lib/dokku/home/dokku/APP/letsencrypt/certs/HASH/
chmod -R 777 /var/lib/dokku/data/letsencrypt/APP/

Which solved the "cant create directory" stuff although I still can't get this to work

=====> Enabling letsencrypt for APP
-----> Enabling ACME proxy for APP...
       ok: run: nginx: (pid 70) 702731s
-----> Getting letsencrypt certificate for APP via HTTP-01
        - Domain 'DOMAIN'
       DOCKERS:  docker run --rm --env-file /var/lib/dokku/home/dokku/APP/letsencrypt/certs/HASH/docker.env --user 200:200 -v /var/lib/dokku/home/dokku/APP/letsencrypt/certs/HASH:/certs -v /var/lib/dokku/data/letsencrypt/APP:/webroot goacme/lego:v4.9.1 --http --http.webroot /webroot --pem --accept-tos --cert.timeout 2592000 --path /certs --server https://acme-v02.api.letsencrypt.org/directory --email EMAIL --domains DOMAIN run
2024/09/18 10:02:57 [INFO] [DOMAIN] acme: Obtaining bundled SAN certificate
2024/09/18 10:02:58 [INFO] [DOMAIN] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/ID
2024/09/18 10:02:58 [INFO] [DOMAIN] acme: Could not find solver for: tls-alpn-01
2024/09/18 10:02:58 [INFO] [DOMAIN] acme: use http-01 solver
2024/09/18 10:02:58 [INFO] [DOMAIN] acme: Trying to solve HTTP-01
2024/09/18 10:03:03 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/ID
2024/09/18 10:03:03 Could not obtain certificates:
	error: one or more domains had a problem:
[DOMAIN] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: IP: Invalid response from http://DOMAIN/.well-known/acme-challenge/CHALLENGE_KEY: 404
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for APP...
       ok: run: nginx: (pid 70) 702738s
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

The file also gets correctly created (and removed) just not accessible for some reason

[krokhale]

This isn't working for me either, and unfortunately it's hard to say if the other dokku plugins might have similar issues running inside a docker container. My use case is running on a mac mini and I have decided to switch to using a multipass instance with dokku and it works great without any symlink hacks. Multipass uses quemu and docker uses something more efficient but power usage wise it's similar while using dokku.