acmetool v0.2.1 failing getting certificate (ACMEv2)

[backstube]

Good afternoon,
after upgrading to the most recent verion acmetool won't get any certificates. Previous version was v.0.0.67 among others, which have been serving for a long time just fine. After upgrading, a quickstart has been done. This is the base configuration:

#> acmetool status
Settings:
  ACME_STATE_DIR: /var/db/acme
  ACME_HOOKS_DIR: /usr/local/libexec/acme/hooks
  Default directory URL: https://acme-v02.api.letsencrypt.org/directory
  Preferred key type: rsa-4096
  Additional webroots:
    /var/run/acme/acme-challenge

Available accounts:
  Account(acme-v02.api.letsencrypt.org%2fdirectory/01234567890abcdefghijklmnopqrst)
    thumbprint: c_mnopqrstuvwxyzmnopqrstuvwxyz

Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0)
  error: Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): no certificate satisfies this target

ACMEv2 is set everywhere, as challenge completion method webroot has been set (as it was the choice over the last couple years). For webroot, nginx is configured according to Hugo's user guide by including a server location block to have no redundancies across several domains and their corresponding (virtual) servers. This method has been working over dozens of renewals over the years, and it proves working by dropping an arbitrary file in there (/var/run/acme/acme-challenge/) and subsequentally fetching it (wget, browser). No problem.

The problem is, acmetool would not fetch any certificate:

#> acmetool --version
go version go1.15.6 freebsd/amd64 gc cgo=true
acmetool-v0.2.1-freebsd-ports
#> acmetool --xlog.syslog --xlog.facility=local7 --xlog.severity=debug reconcile
20210202150310 [DEBUG] acmetool.storageops: Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): best certificate satisfying is <nil>, err=Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): no certificate satisfies this target
20210202150310 [DEBUG] acmetool.storageops: Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): requesting certificate
20210202150349 [DEBUG] fdb: enforce permissions: keys/1234567890abcdefghij/privkey 0/0 0/0
20210202150349 [DEBUG] acmetool.storageops: Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): ordering certificate
20210202150350 [DEBUG] acmetool.reshttp: acquire port "[::]:80" "abcdefghijklmn--my--challenge--abcdefghijklmn"
20210202150350 [DEBUG] acmetool.reshttp: listening on [::]:80
20210202150350 [DEBUG] acmetool.reshttp: acquire port ":80" "abcdefghijklmn--my--challenge--abcdefghijklmn"
20210202150350 [DEBUG] acmetool.reshttp: failed to listen on :80: listen tcp :80: bind: address already in use
20210202150350 [DEBUG] acmetool.reshttp: acquire port "[::1]:402" "abcdefghijklmn--my--challenge--abcdefghijklmn"
20210202150350 [DEBUG] acmetool.reshttp: listening on [::1]:402
20210202150350 [DEBUG] acmetool.reshttp: acquire port "127.0.0.1:402" "abcdefghijklmn--my--challenge--abcdefghijklmn"
20210202150350 [DEBUG] acmetool.reshttp: listening on 127.0.0.1:402
20210202150350 [DEBUG] acmetool.reshttp: acquire port "[::1]:4402" "abcdefghijklmn--my--challenge--abcdefghijklmn"
20210202150350 [DEBUG] acmetool.reshttp: listening on [::1]:4402
20210202150350 [DEBUG] acmetool.reshttp: acquire port "127.0.0.1:4402" "abcdefghijklmn--my--challenge--abcdefghijklmn"
20210202150350 [DEBUG] acmetool.reshttp: listening on 127.0.0.1:4402
20210202150350 [DEBUG] acme.responder: writing 1 webroot challenge files
20210202150350 [DEBUG] acme.responder: writing webroot file /var/run/acme/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn
20210202150350 [DEBUG] acme.hooks: calling hook script: /usr/local/libexec/acme/hooks/haproxy
20210202150350 [DEBUG] acme.hooks: calling hook script: /usr/local/libexec/acme/hooks/reload
20210202150350 [DEBUG] acme.responder: http-01 self test for "example.com"
20210202150355 [INFO] acme.responder: http-01 self test failed: example.com: Get "http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20210202150355 [DEBUG] acme.responder: removing webroot file /var/run/acme/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn
20210202150355 [DEBUG] acme.hooks: calling hook script: /usr/local/libexec/acme/hooks/haproxy
20210202150355 [DEBUG] acme.hooks: calling hook script: /usr/local/libexec/acme/hooks/reload
20210202150355 [DEBUG] acmetool.solver: challenge start failed: Get "http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20210202150355 [DEBUG] acme.hooks: calling hook script: /usr/local/libexec/acme/hooks/haproxy
20210202150355 [DEBUG] acme.hooks: calling hook script: /usr/local/libexec/acme/hooks/reload
20210202150355 [DEBUG] acmetool.solver: challenge start failed: could not install DNS challenge, no hooks succeeded
20210202150355 [DEBUG] acmetool.solver: challenge instantiation failed: challenge type not supported
20210202150356 [ERROR] acmetool.storageops: Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): failed to request certificate: the following errors occurred:
exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz-v3/0123456789" [due to inner error: the following errors occurred:
Get "http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn": context deadline exceeded (Client.Timeout exceeded while awaiting headers);
could not install DNS challenge, no hooks succeeded;
challenge type not supported]
20210202150356 [DEBUG] acmetool.storageops: done processing targets, reconciliation complete, 1 errors occurred
20210202150356 [ERROR] acmetool.storageops: error while processing targets: the following errors occurred:
error satisfying Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): the following errors occurred:
exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz-v3/0123456789" [due to inner error: the following errors occurred:
Get "http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn": context deadline exceeded (Client.Timeout exceeded while awaiting headers);
could not install DNS challenge, no hooks succeeded;
challenge type not supported]
20210202150356 [ERROR] acmetool.storageops: failed to reconcile: the following errors occurred:
error satisfying Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): the following errors occurred:
exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz-v3/0123456789" [due to inner error: the following errors occurred:
Get "http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn": context deadline exceeded (Client.Timeout exceeded while awaiting headers);
could not install DNS challenge, no hooks succeeded;
challenge type not supported]
20210202150405 [DEBUG] acmetool.storageops: disjoint hostname mapping: "example.com" -> Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0)
20210202150405 [DEBUG] acmetool.storageops: could not find certificate satisfying Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): no certificate satisfies this target
20210202150405 [CRITICAL] acmetool: fatal: reconcile: the following errors occurred:
error satisfying Target(example.com;https://acme-v02.api.letsencrypt.org/directory;0): the following errors occurred:
exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz-v3/0123456789" [due to inner error: the following errors occurred:
Get "http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn": context deadline exceeded (Client.Timeout exceeded while awaiting headers);
could not install DNS challenge, no hooks succeeded;
challenge type not supported]
#>

While running a reconcile, after generating a key pair for the hostname at hand, acmetool puts a challenge webroot file in that said directory. I simply can see it using ls -l. To me, this somewhat mismatches with log lines like could not install DNS challenge, no hooks succeeded;.

I have also altered the challenge completion to proxy method for test, including nginx server configuration modifications, but to no success. Lastly I shut the web server down, releasing ports 80 and 443 to acmetool to do its thing. Still no success.

Can anyone explain what's going on here? What's wrong? Any help is highly appreciated.

[mnalis]

local firewall / resolving issue? this is the problem: 20210202150355 [INFO] acme.responder: http-01 self test failed: example.com: Get "http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

self-test means the acmetool client tried to retrieve this url, and faild due to timeout. You should try to manually run wget http://example.com/.well-known/acme-challenge/abcdefghijklmn--my--challenge--abcdefghijklmn from same machine, and it will probably fail with similar timeout. Then you proceed to debug (does ping example.com resolve to correct IP, can you telnet example.com 80 etc.)