http: panic serving: runtime error: invalid memory address or nil pointer dereference

[ricosega]

Subject of the issue

After deploying the server in kubernetes with custom CA, couldn't make it work because of the error below.

Your environment

Kubernetes 1.20.0
step-certificates: 0.15.15

Steps to reproduce

Deploy the step-certificates helm-chart with inject block with your root CA certs and keys.

Expected behaviour

Getting an specific error about db not configured.

Actual behaviour

{"duration":"95.261µs","duration-ns":95261,"fields.time":"2021-06-23T20:08:14Z","level":"info","method":"GET","msg":"","name":"ca","path":"/acme/acme/directory","protocol":"HTTP/1.1","referer":"","remote-address":"192.168.143.76","request-id":"c39p9bh1gp8uen3ofb3g","response":"{\"newNonce\":\"https://tls.glo.helena/acme/acme/new-nonce\",\"newAccount\":\"https://tls.glo.helena/acme/acme/new-account\",\"newOrder\":\"https://tls.glo.helena/acme/acme/new-order\",\"revokeCert\":\"https://tls.glo.helena/acme/acme/revoke-cert\",\"keyChange\":\"https://tls.glo.helena/acme/acme/key-change\"}","size":292,"status":200,"time":"2021-06-23T20:08:14Z","user-agent":"cert-manager/v1.4.0 (clean) golang.org/x/crypto/acme","user-id":""}
2021/06/23 20:08:14 /usr/local/go/src/net/http/server.go:3137: http: panic serving 192.168.143.76:42764: runtime error: invalid memory address or nil pointer dereference
goroutine 26294 [running]:
net/http.(*conn).serve.func1(0xc0001f85a0)
	/usr/local/go/src/net/http/server.go:1824 +0x153
panic(0x10a1380, 0x1aae900)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/smallstep/certificates/acme/api.(*Handler).addNonce.func1(0x7f9246bb4f98, 0xc000716f20, 0xc00070e900)
	/src/acme/api/middleware.go:65 +0x62
github.com/smallstep/certificates/acme/api.(*Handler).lookupProvisioner.func1(0x7f9246bb4f98, 0xc000716f20, 0xc00070e800)
	/src/acme/api/middleware.go:308 +0x2a2
github.com/smallstep/certificates/acme/api.(*Handler).baseURLFromRequest.func1(0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/src/acme/api/middleware.go:58 +0x17c
net/http.HandlerFunc.ServeHTTP(0xc00029f660, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00046f320, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:425 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc00029f620, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00046f320, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:70 +0x50c
github.com/go-chi/chi.(*Mux).Mount.func1(0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:292 +0x122
net/http.HandlerFunc.ServeHTTP(0xc000427e60, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00046f200, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:425 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc00029f0f0, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00046f200, 0x7f9246bb4f98, 0xc000716f20, 0xc00070e600)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:82 +0x2d1
github.com/smallstep/certificates/logging.(*LoggerHandler).ServeHTTP(0xc000300bd0, 0x14140a0, 0xc0004e69a0, 0xc00070e600)
	/src/logging/handler.go:49 +0xe2
github.com/smallstep/certificates/logging.RequestID.func1.1(0x14140a0, 0xc0004e69a0, 0xc00070e500)
	/src/logging/context.go:38 +0x20c
net/http.HandlerFunc.ServeHTTP(0xc000300c00, 0x14140a0, 0xc0004e69a0, 0xc00070e500)
	/usr/local/go/src/net/http/server.go:2069 +0x44
net/http.serverHandler.ServeHTTP(0xc0001c4540, 0x14140a0, 0xc0004e69a0, 0xc00070e500)
	/usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc0001f85a0, 0x14184f8, 0xc0003084c0)
	/usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3013 +0x39b
2021/06/23 20:08:14 /usr/local/go/src/net/http/server.go:3137: http: panic serving 192.168.143.76:42766: runtime error: invalid memory address or nil pointer dereference
goroutine 26260 [running]:
net/http.(*conn).serve.func1(0xc0005a8280)
	/usr/local/go/src/net/http/server.go:1824 +0x153
panic(0x10a1380, 0x1aae900)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/smallstep/certificates/acme/api.(*Handler).addNonce.func1(0x7f9246bb4f98, 0xc000096c00, 0xc0005ce500)
	/src/acme/api/middleware.go:65 +0x62
github.com/smallstep/certificates/acme/api.(*Handler).lookupProvisioner.func1(0x7f9246bb4f98, 0xc000096c00, 0xc0005ce400)
	/src/acme/api/middleware.go:308 +0x2a2
github.com/smallstep/certificates/acme/api.(*Handler).baseURLFromRequest.func1(0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/src/acme/api/middleware.go:58 +0x17c
net/http.HandlerFunc.ServeHTTP(0xc00029f660, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00046f320, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:425 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc00029f620, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00046f320, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:70 +0x50c
github.com/go-chi/chi.(*Mux).Mount.func1(0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:292 +0x122
net/http.HandlerFunc.ServeHTTP(0xc000427e60, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00046f200, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:425 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc00029f0f0, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce300)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00046f200, 0x7f9246bb4f98, 0xc000096c00, 0xc0005ce200)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:82 +0x2d1
github.com/smallstep/certificates/logging.(*LoggerHandler).ServeHTTP(0xc000300bd0, 0x14140a0, 0xc000586700, 0xc0005ce200)
	/src/logging/handler.go:49 +0xe2
github.com/smallstep/certificates/logging.RequestID.func1.1(0x14140a0, 0xc000586700, 0xc0005ce100)
	/src/logging/context.go:38 +0x20c
net/http.HandlerFunc.ServeHTTP(0xc000300c00, 0x14140a0, 0xc000586700, 0xc0005ce100)
	/usr/local/go/src/net/http/server.go:2069 +0x44
net/http.serverHandler.ServeHTTP(0xc0001c4540, 0x14140a0, 0xc000586700, 0xc0005ce100)
	/usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc0005a8280, 0x14184f8, 0xc0000307c0)
	/usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3013 +0x39b
2021/06/23 20:08:14 /usr/local/go/src/net/http/server.go:3137: http: panic serving 192.168.143.76:42768: runtime error: invalid memory address or nil pointer dereference
goroutine 26305 [running]:
net/http.(*conn).serve.func1(0xc0002db900)
	/usr/local/go/src/net/http/server.go:1824 +0x153
panic(0x10a1380, 0x1aae900)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/smallstep/certificates/acme/api.(*Handler).addNonce.func1(0x7f9246bb4f98, 0xc000096da0, 0xc0005ce900)
	/src/acme/api/middleware.go:65 +0x62
github.com/smallstep/certificates/acme/api.(*Handler).lookupProvisioner.func1(0x7f9246bb4f98, 0xc000096da0, 0xc0005ce800)
	/src/acme/api/middleware.go:308 +0x2a2
github.com/smallstep/certificates/acme/api.(*Handler).baseURLFromRequest.func1(0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/src/acme/api/middleware.go:58 +0x17c
net/http.HandlerFunc.ServeHTTP(0xc00029f660, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00046f320, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:425 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc00029f620, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00046f320, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:70 +0x50c
github.com/go-chi/chi.(*Mux).Mount.func1(0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:292 +0x122
net/http.HandlerFunc.ServeHTTP(0xc000427e60, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00046f200, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:425 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc00029f0f0, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce700)
	/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00046f200, 0x7f9246bb4f98, 0xc000096da0, 0xc0005ce600)
	/go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:82 +0x2d1
github.com/smallstep/certificates/logging.(*LoggerHandler).ServeHTTP(0xc000300bd0, 0x14140a0, 0xc00081c000, 0xc0005ce600)
	/src/logging/handler.go:49 +0xe2
github.com/smallstep/certificates/logging.RequestID.func1.1(0x14140a0, 0xc00081c000, 0xc000814000)
	/src/logging/context.go:38 +0x20c
net/http.HandlerFunc.ServeHTTP(0xc000300c00, 0x14140a0, 0xc00081c000, 0xc000814000)
	/usr/local/go/src/net/http/server.go:2069 +0x44
net/http.serverHandler.ServeHTTP(0xc0001c4540, 0x14140a0, 0xc00081c000, 0xc000814000)
	/usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc0002db900, 0x14184f8, 0xc000810000)
	/usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3013 +0x39b

Additional context

Already know what is the problem, in the helm chart is missing the db config, I created a PR to the helm chart adding this.

[maraino]

The problem is caused because the reference ca.json injected does not include a db configuration, the following change adds a db in the values.yaml
b2740f2#diff-259c32a25be7d822500dd568e592c69ec3f888645d2a4b1923cc85e953b114d1

[maraino]

@dopey should step-ca fail on start if an acme provisioner is configured and the db is not?

[ricosega]

I created a PR to the helm chart adding this.

That solves the issue but I think there should be a checking of the config...

[maraino]

I've just saw the PR, you might need to rebase, because it might conflict with my changes. I've also asked @estenrye about the change on the secrets.

[dopey]

@dopey should step-ca fail on start if an acme provisioner is configured and the db is not?

@maraino I'm not sure :/ You may still want to use the other provisioners and you won't run into a 5xx until you make a request to the ACME API. The least surprising thing would be to fail up front, just will be unfortunate if that breaks existing setups.

[dopey]

Going to close this issue since it appears @maraino already committed a fix. Please comment or reopen if the issue has not been resolved.