pebble-challtestsrv: small binary for mock DNS & ACME challenges. (#181)

Boulder has a nice handy [`challtestsrv` package and
command](https://github.com/letsencrypt/boulder/tree/9e39680e3f78c410e2d780a7badfe200a31698eb/test/challtestsrv)
used for integration tests. Its small and useful enough that the library portion
has been promoted to a first-class repo:
https://github.com/letsencrypt/challtestsrv 

The stand-alone binary with an HTTP management interface can come live in the
Pebble repo where more folks can use it without pulling in all of Boulder. I've
heard from a few ACME client developers that this would be useful to them. It is
possible we could achieve the same thing by leaving the binary in the Boulder
repo using the updated code that doesn't import other things from Boulder.
Moving it out of the repo will help us commit to working on abstractions that
make tests cleaner. This also makes it quick and easy to have a full Pebble
environment with mock DNS without needing to install tools from other repos.

The dependency on the letsencrypt/challtestsrv package does require pulling in
a dep. on `github.com/miekg/dns` (and vendoring it) but I think its a fair
tradeoff.

The provided Dockerfile is now split into two dockerfiles (see `docker/`
directory): one for `pebble` and one for `pebble-challtestsrv`. They are both
updated to use Go 1.11, to build with the vendored modules instead of fetching
them at build time, and to use the latest Alpine base image. 

A new `docker-compose.yml` example is included that starts up
a `pebble-challtestsrv` container and a `pebble` container that uses the former
as its DNS server. The README is updated to explain the usage briefly.

Author: cpu

.dockerignore

@@ -1,4 +1,3 @@
 .git
 pebble.exe
 pebble
-vendor/

README.md

@@ -82,7 +82,38 @@ pebble -config ./test/config/pebble-config.json
 Afterwards you can access the Pebble server's ACME directory
 at `https://localhost:14000/dir`.
 
-### Docker Image
+### Docker
+
+Pebble includes a [docker-compose](https://docs.docker.com/compose/) file that
+will create a `pebble` instance that uses a `pebble-challtestsrv` instance for
+DNS resolution.
+
+To build and start the containers run:
+
+```
+docker-compose up
+```
+
+Afterwards you can access the ACME API from your host machine at
+`https://localhost:14000/dir` and the `pebble-challtestsrv`'s management
+interface at `http://locahost:8055`.
+
+To get started you may want to update the `pebble-challtestsrv` mock DNS data
+with a new default IPv4 address to use to respond to `A` queries from `pebble`:
+
+```
+curl --request POST --data '{"ip":"172.20.0.1"}' http://localhost:8055/set-default-ipv4
+```
+
+See the [pebble-challtestsrv
+README](https://github.com/letsencrypt/pebble/blob/master/cmd/pebble-challtestsrv/README.md)
+for more information.
+
+#### Prebuilt Docker Images
+
+If you would prefer not to use the provided `docker-compose.yml`, or to build
+container images yourself, you can also use the [published
+images](https://hub.docker.com/r/letsencrypt/pebble/).
 
 With a docker-compose file:
 
@@ -92,7 +123,7 @@ version: '3'
 services:
  pebble:
   image: letsencrypt/pebble
-  command: pebble -config ./test/my-pebble-config.json
+  command: pebble -config /test/my-pebble-config.json
   ports:
     - 14000:14000
   environment:
@@ -135,14 +166,21 @@ By default Pebble uses the system DNS resolver, this may mean that caching cause
 problems with DNS-01 validation. It may also mean that no DNSSEC validation is
 performed.
 You should configure your system's recursive DNS resolver according to your
-needs or use the `-dnsserver` flag to define an address to a DNS server. 
+needs or use the `-dnsserver` flag to define an address to a DNS server.
 
 ```
 pebble -dnsserver 10.10.10.10:5053
 pebble -dnsserver 8.8.8.8:53
 pebble -dnsserver :5053
 ```
 
+You may find it useful to set `pebble`'s `-dnsserver` to the address you used as
+the `-dns01` argument when starting up a `pebble-challtestsrv` instance. This
+will let you easily mock DNS data for Pebble. See the included
+`docker-compose.yml` and the [pebble-challtestsrv
+README](https://github.com/letsencrypt/pebble/blob/master/cmd/pebble-challtestsrv/README.md)
+for more information.
+
 ### Testing at full speed
 
 By default Pebble will sleep a random number of seconds (from 0 to 15) between

cmd/config.go cmd/command.gocmd/config.go renamed to cmd/command.go

@@ -4,7 +4,10 @@ import (
 	"encoding/json"
 	"fmt"
 	"io/ioutil"
+	"log"
 	"os"
+	"os/signal"
+	"syscall"
 )
 
 // ReadConfigFile takes a file path as an argument and attempts to
@@ -31,3 +34,28 @@ func FailOnError(err error, msg string) {
 		os.Exit(1)
 	}
 }
+
+var signalToName = map[os.Signal]string{
+	syscall.SIGTERM: "SIGTERM",
+	syscall.SIGINT:  "SIGINT",
+	syscall.SIGHUP:  "SIGHUP",
+}
+
+// CatchSignals catches SIGTERM, SIGINT, SIGHUP and executes a callback
+// method before exiting
+func CatchSignals(callback func()) {
+	sigChan := make(chan os.Signal, 1)
+	signal.Notify(sigChan, syscall.SIGTERM)
+	signal.Notify(sigChan, syscall.SIGINT)
+	signal.Notify(sigChan, syscall.SIGHUP)
+
+	sig := <-sigChan
+	log.Printf("Caught %s", signalToName[sig])
+
+	if callback != nil {
+		callback()
+	}
+
+	log.Printf("Exiting")
+	os.Exit(0)
+}

cmd/pebble-challtestsrv/README.md

@@ -0,0 +1,139 @@
+# Pebble Challenge Test Server
+
+**Important note: The `pebble-challtestsrv` command is for TEST USAGE ONLY. It
+is trivially insecure, offering no authentication. Only use
+`pebble-challtestsrv` in a controlled test environment.**
+
+The standalone `pebble-challtestsrv` binary lets you run HTTP-01, HTTPS HTTP-01,
+DNS-01, and TLS-ALPN-01 challenge servers that external programs can add/remove
+challenge responses to using a HTTP management API.
+
+For example this is used by the Boulder integration tests to easily add/remove
+TXT records for DNS-01 challenges for the `chisel.py` ACME client, and to test
+redirect behaviour for HTTP-01 challenge validation.
+
+### Usage
+
+```
+Usage of pebble-challtestsrv:
+  -dns01 string
+    Comma separated bind addresses/ports for DNS-01 challenges and fake DNS data. Set empty to disable. (default ":8053")
+  -http01 string
+    Comma separated bind addresses/ports for HTTP-01 challenges. Set empty to disable. (default ":5002")
+  -https01 string
+    Comma separated bind addresses/ports for HTTPS HTTP-01 challenges. Set empty to disable. (default ":5003")
+  -management string
+    Bind address/port for management HTTP interface (default ":8055")
+  -tlsalpn01 string
+    Comma separated bind addresses/ports for TLS-ALPN-01 and HTTPS HTTP-01 challenges. Set empty to disable. (default ":5001")
+```
+
+To disable a challenge type, set the bind address to `""`. E.g.:
+
+* To run HTTP-01 only: `pebble-challtestsrv -dns01 "" -tlsalpn01 ""`
+* To run DNS-01 only: `challtestsrv -http01 "" -tlsalpn01 ""`
+* To run TLS-ALPN-01 only: `challtestsrv -http01 "" -dns01 ""`
+
+### Management Interface
+
+_Note: These examples assume the default `-management` interface address, `:8056`._
+
+#### Mock DNS
+
+##### Default A/AAAA Responses
+
+To set the default IPv4 address used for responses to `A` queries that do not
+match explicit mocks run:
+
+    curl -X POST -d '{"ip":"10.10.10.2"}' http://localhost:8056/set-default-ipv4
+
+Similarly to set the default IPv6 address used for responses to `AAAA` queries
+that do not match explicit mocks run:
+
+    curl -X POST -d '{"ip":"::1"}' http://localhost:8056/set-default-ipv6
+
+To clear the default IPv4 or IPv6 address POST the same endpoints with an empty
+(`""`) IP.
+
+##### Mocked A/AAAA Responses
+
+To add IPv4 addresses to be returned for `A` queries for
+`test-host.letsencrypt.org` run:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org", "addresses":["12.12.12.12", "13.13.13.13"]}' http://localhost:8056/add-a
+
+The mocked `A` responses can be removed by running:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org"}' http://localhost:8056/clear-a
+
+To add IPv6 addresses to be returned for `AAAA` queries for
+`test-host.letsencrypt.org` run:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org", "addresses":["2001:4860:4860::8888", "2001:4860:4860::8844"]}' http://localhost:8056/add-aaaa
+
+The mocked `AAAA` responses can be removed by running:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org"}' http://localhost:8056/clear-aaaa
+
+##### Mocked CAA Responses
+
+To add a mocked CAA policy for `test-host.letsencrypt.org` that allows issuance
+by `letsencrypt.org` run:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org", "policies":[{"tag":"issue","value":"letsencrypt.org"}]}' http://localhost:8055/add-caa
+
+To remove the mocked CAA policy for `test-host.letsencrypt.org` run:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org"}' http://localhost:8055/clear-caa
+
+#### HTTP-01
+
+To add an HTTP-01 challenge response for the token `"aaaa"` with the content `"bbbb"` run:
+
+    curl -X POST -d '{"token":"aaaa", "content":"bbbb"}' http://localhost:8056/add-http01
+
+Afterwards the challenge response will be available over HTTP at
+`http://localhost:5002/.well-known/acme-challenge/aaaa`, and HTTPS at
+`https://localhost:5002/.well-known/acme-challenge/aaaa`.
+
+The HTTP-01 challenge response for the `"aaaa"` token can be deleted by running:
+
+    curl -X POST -d '{"token":"aaaa"}' http://localhost:8056/del-http01
+
+##### Redirects
+
+To add a redirect from `/.well-known/acme-challenge/whatever` to
+`https://localhost:5003/ok` run:
+
+    curl -X POST -d '{"path":"/.well-known/whatever", "targetURL": "https://localhost:5003/ok"}' http://localhost:8056/add-redirect
+
+Afterwards HTTP requests to `http://localhost:5002/.well-known/whatever/` will
+be redirected to `https://localhost:5003/ok`. HTTPS requests that match the
+path will not be served a redirect to prevent loops when redirecting the same
+path from HTTP to HTTPS.
+
+To remove the redirect run:
+
+    curl -X POST -d '{"path":"/.well-known/whatever"}' http://localhost:8056/del-redirect
+
+#### DNS-01
+
+To add a DNS-01 challenge response for `_acme-challenge.test-host.letsencrypt.org` with
+the value `"foo"` run:
+
+    curl -X POST -d '{"host":"_acme-challenge.test-host.letsencrypt.org", "value": "foo"}' http://localhost:8056/add-txt
+
+To remove the mocked DNS-01 challenge response run:
+
+    curl -X POST -d '{"host":"_acme-challenge.test-host.letsencrypt.org"}' http://localhost:8056/clear-txt
+
+#### TLS-ALPN-01
+
+To add a TLS-ALPN-01 challenge response certificate for the host
+`test-host.letsencrypt.org` with the key authorization `"foo"` run:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org", "content":"foo"}' http://localhost:8056/add-tlsalpn01
+
+To remove the mocked TLS-ALPN-01 challenge response run:
+
+    curl -X POST -d '{"host":"test-host.letsencrypt.org"}' http://localhost:8056/clear-tlsalpn01

cmd/pebble-challtestsrv/dnsone.go

@@ -0,0 +1,65 @@
+package main
+
+import "net/http"
+
+// addDNS01 handles an HTTP POST request to add a new DNS-01 challenge TXT
+// record for a given host/value.
+//
+// The POST body is expected to have two non-empty parameters:
+// "host" - the hostname to add the mock TXT response under.
+// "value" - the key authorization value to return in the TXT response.
+//
+// A successful POST will write http.StatusOK to the client.
+func (srv *managementServer) addDNS01(w http.ResponseWriter, r *http.Request) {
+	// Unmarshal the request body JSON as a request object
+	var request struct {
+		Host  string
+		Value string
+	}
+	if err := mustParsePOST(&request, r); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	// If the request has an empty host or value it's a bad request
+	if request.Host == "" || request.Value == "" {
+		w.WriteHeader(http.StatusBadRequest)
+		return
+	}
+
+	// Add the DNS-01 challenge response TXT to the challenge server
+	srv.challSrv.AddDNSOneChallenge(request.Host, request.Value)
+	srv.log.Printf("Added DNS-01 TXT challenge for Host %q - Value %q\n",
+		request.Host, request.Value)
+	w.WriteHeader(http.StatusOK)
+}
+
+// delDNS01 handles an HTTP POST request to delete an existing DNS-01 challenge
+// TXT record for a given host.
+//
+// The POST body is expected to have one non-empty parameter:
+// "host" - the hostname to remove the mock TXT response for.
+//
+// A successful POST will write http.StatusOK to the client.
+func (srv *managementServer) delDNS01(w http.ResponseWriter, r *http.Request) {
+	// Unmarshal the request body JSON as a request object
+	var request struct {
+		Host string
+	}
+	if err := mustParsePOST(&request, r); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	// If the request has an empty host value it's a bad request
+	if request.Host == "" {
+		w.WriteHeader(http.StatusBadRequest)
+		return
+	}
+
+	// Delete the DNS-01 challenge response TXT for the given host from the
+	// challenge server
+	srv.challSrv.DeleteDNSOneChallenge(request.Host)
+	srv.log.Printf("Removed DNS-01 TXT challenge for Host %q\n", request.Host)
+	w.WriteHeader(http.StatusOK)
+}

cmd/pebble-challtestsrv/http.go

@@ -0,0 +1,24 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"io/ioutil"
+	"net/http"
+)
+
+// mustParsePOST will attempt to read a JSON POST body from the provided request
+// and unmarshal it into the provided ob. If an error occurs at any point it
+// will be returned.
+func mustParsePOST(ob interface{}, request *http.Request) error {
+	jsonBody, err := ioutil.ReadAll(request.Body)
+	if err != nil {
+		return err
+	}
+
+	if string(jsonBody) == "" {
+		return errors.New("Expected JSON POST body, was empty")
+	}
+
+	return json.Unmarshal(jsonBody, ob)
+}