Embedded ACME proxy server?

[BioEvo]

This allows other ACME clients (like Caddy under privite networks) to use another Caddy as an ACME endpoint, and proxy/convert ACME challenge to upstream like Let's Encrypt.

[pinkeen]

I love the idea as it would allow to avoid having to provision client nodes' trust store with the private CA certificates. This is very convenient as this setup (and later maintenance) can become really complex, especially in big heterogenous environments, embedded devices, etc.

I'd like to add some random input (rambling) to this request too.

Certificate cache

Cache all certificates returned on behalf of the ACME upstream to prevent failures that would be caused by rate limits of the upstream ACME authority.

Possible use-case scenario from real-life experience: Stateless and/or ephemeral cloud nodes which need to request a new certificate on each start, often for the same domains.

Custom authentication challenge (zero-configuration?)

Authenticate the requester by verifying the trust relationship inferred from existing configuration - e.g. requester is an upstream with known client certificate.

I don't have any good idea for verifying the authorization to receive specific certificate though. I wonder if step-ca's provisioning providers could be used for this... Possibly, if the requester is already an upstream and the requested domain matches any of the the ones that it already handles traffic from then there is no challenge needed at all.

---

Setup I have that would benefit from this feature:

• I've got a lot of web services running in my home network (behind a double NAT) which I am exposing publicly.
• I've set up two caddy instances:
  • Public - serves as an https-terminating public reverse-proxy
  • Private - upstream¹ of the public caddy which also serves as an https-terminating aggregator for the private network services²
• When on the home network I want to be able to access the local services directly using the same hostnames in order to avoid unnecessary roundtrip and be able to do that even if the private network has got intermittent connection problems (prevent the ingress being a SPOF for local access).

¹ The ingress is connected to the upstreams via Nebula Overlay Mesh Network.
² There might be multiple private instances of course, but in my case at most two or three (per virtualization host I guess) as it's usually much less work to set up and manage (monitor) a single caddy instance that to configure each service separately to use acme for its certs. Especially that these services are usually running on very different hardware/virt and software platforms.

PS On an unrelated topic - I've just come up with a crazy but vague idea. Nebula is golang too, also uses PKI authentication and has a built-in DNS server (which I actually use as a sub-zone of my primary domain's zone). It would be super-cool to marry the two together. Not sure how though - maybe one could partially automate or provide services for the other so common-usage scenarios could be simplified using one piece of software. Huh, this would be kind of an HTTP routing mesh with out-of-the-box private overlay network.

---

👆 I've just realized how utterly pointless my setup is. 😞 I can just use nginx on the public node as it can pass-through encrypted traffic while still retaining the ability to route it based on SNI Host (ngx_stream_ssl_preread_module. I've even done this before... Though it's not possible to do any custom handling of the encrypted traffic on the ingress server though so I cannot use a singe authorization portal for all of the upstreams (planned to use caddy for this).

[mholt]

@pinkeen Very interesting, thanks for writing that up.

PS On an unrelated topic - I've just come up with a crazy but vague idea. Nebula is golang too, also uses PKI authentication and has a built-in DNS server (which I actually use as a sub-zone of my primary domain's zone). It would be super-cool to marry the two together.

If it can be used as a library, it can probably become a Caddy module without too much trouble! https://caddyserver.com/docs/extending-caddy

I can just use nginx on the public node as it can pass-through encrypted traffic while still retaining the ability to route it based on SNI Host (ngx_stream_ssl_preread_module

Caddy can do this too with Project Conncept that I'm working on (it already works, actually), a layer 4 proxy: https://github.com/mholt/conncept

[francislavoie]

Caddy can do this too with Project Conncept that I'm working on (it already works, actually), a layer 4 proxy: mholt/conncept

For whoever reads this, this is now publicly available: https://github.com/mholt/caddy-l4

[m-martin-78]

Hi, I love the idea of an embedded ACME proxy server.

For now, I'm trying to use this project: https://github.com/noahkw/acmetk

However, I ran into an error with Caddy as a client of this ACME proxy:

2023-08-18 16:42:47,954 - aiohttp.access - INFO - 10.0.2.100 [18/Aug/2023:14:42:47 +0000] "POST /new-order HTTP/1.1" 400 389 "-" "Caddy/ CertMagic acmez (linux; amd64)"
[... stacktrace snipped ...]
josepy.errors.DeserializationError: Deserialization error: Could not decode 'status' (''): Deserialization error: Status not recognized

It looks like the Caddy ACME client, when ordering a new certificate, does not set the "status" property to a valid value (see https://datatracker.ietf.org/doc/html/rfc8555#section-7.1.3)

What do you think?

(related ticket on the original project: noahkw/acmetk#64)

[mholt]

@m-martin-78 I would be surprised if it's a bug in the ACME client (ACMEz) due to it being used successfully without bugs in production for years now. But maybe? 🤷‍♂️ It doesn't really make sense though, as the client doesn't set the status when requesting a certificate. The server's job is to set the status, and the client uses that to control its behavior.

Unfortunately the error message doesn't really give us much to go on. If you can make it minimally reproducible (without an ACME proxy) then please file an issue at https://github.com/mholt/acmez with instructions to minimally reproduce the issue.

(And let's keep this issue on-topic.) Thanks!