pkiCert does not properly save certificates across multiple templates

[remilapeyre]

Nomad version

v1.9.3

Issue

The information at

nomad/website/content/docs/job-specification/template.mdx

Lines 672 to 710 in 2eb2b6c

	#### As individual files
	For templates, all dependencies are mapped into a single list. This means that
	multiple templates watching the same path return the same data.
	```hcl
	template {
	data = <<EOH
	{{ with pkiCert "pki/issue/foo" "common_name=foo.service.consul" "ip_sans=127.0.0.1" }}
	{{- .Cert -}}
	{{ end }}
	EOH
	destination = "${NOMAD_SECRETS_DIR}/certificate.crt"
	change_mode = "restart"
	}
	template {
	data = <<EOH
	{{ with pkiCert "pki/issue/foo" "common_name=foo.service.consul" "ip_sans=127.0.0.1" }}
	{{- .CA -}}
	{{ end }}
	EOH
	destination = "${NOMAD_SECRETS_DIR}/ca.crt"
	change_mode = "restart"
	}
	template {
	data = <<EOH
	{{ with pkiCert "pki/issue/foo" "common_name=foo.service.consul" "ip_sans=127.0.0.1" }}
	{{- .Key -}}
	{{ end }}
	EOH
	destination = "${NOMAD_SECRETS_DIR}/private_key.key"
	change_mode = "restart"
	}
	```
	These are three different input templates, but when run under the Nomad job,
	they are compressed into a single call, sharing the resulting data.

is wrong, pkiCert does not save the certificates and all three templates make a call to Vault, which means that the certificate and the private key will not match.

This part of the documentation as been changed in 1eb1dbf#diff-2e97a76f1d2de1fec74a5033b6be6625127e85de0c991ce328535bd384659807L654. Previously the example used the secret function which does save its result across invocations so the text was correct.

The change to this part of the documentation should be rollbacked for now and caching support could be added to pkiCert in a second time.

[pkazmierczak]

Hey @remilapeyre, thanks for reporting the issue. @aimeeu, can you have a look at this?

[pkazmierczak]

ok @remilapeyre, I looked into this and indeed 3 calls are being made and the certificates do not match. At this point I still can't say if this is a mistake in the documentation or if it's a bug in our template engine. I'll put this on our board, thanks again for reporting.

[remilapeyre]

Hi @pkazmierczak, I checked this a bit more and now understand better when and when not things are getting cached.

Dependencies in the consul-template engine are cached based on Dependency.String() and secret goes through the trouble of hashing its input instead of using the file name like pkiCert does.

I'm pretty sure it has always been that way and I don't think that caching a call across multiple templates was considered at the time.

pkiCert was introduced to avoid fetching a new cert on the agent startup, because Vault may be down or it's actually expensive to generate or store certificates in Vault's backend.

The documentation actually explains this behavior at https://github.com/hashicorp/consul-template/blob/main/docs/templating-language.md?plain=1#L720-L766 even thought it's super awkward to have to use writeToFile instead of being able to just use multiple template blocks.

In the context of Nomad I don't think it makes sense to recommend using pkiCert because it uses the local file as the cache, but all allocations will start with an empty local directory anyway so the cache will be empty and a new cert will need to be fetched in all cases.

It looks to me like pkiCert only makes sense when using consul-template / vault agent where the files stored on disk can survive agent restarts / server reboots and where the user might actually benefit from this cache support.

I think we should update the documentation to recommend using secret and document the thwarts of pkiCert better.

[mohammedpatla]

We actually ran into this error, trying to load certs from our vault into Nomad Docker jobs. Is there a workaround or fix for this?