Workload identity - Vault token renewal runs every 30 seconds

[nicoche]

Hey!

Nomad version

Nomad 1.9.6 (also seemed to happen on 1.10)

Issue

Hey!

We use use Vault secrets in our job spec templates. We recently upgraded to Nomad 1.9 and Workload Identity.

We have an issue where it seems that nomad-clients are trying too often to create Vault tokens. From what I read of the documentation, nomad-client is supposed to create a token when first booting an alloc, with the TTL specified by the Identity. Then, it would renew the token when we are halfways through the TTL.

From what I read of the code, an initial TTL of 30s is specified here https://github.com/hashicorp/nomad/blob/main/client/allocrunner/taskrunner/vault_hook.go#L292 but then, I couldn't understand where the TTL specified by the workload identity is enforced. It seems to me that the token is renewed every 30s/2=15s.

Do you think this is a bug? If no, do you know what could be happening?

[tgross]

Hi @nicoche! The TTL in the identity block is the TTL for the Workload Identity. What you're looking for is the TTL of the Vault token, which is set by the Vault auth configuration in the token_ttl field. Because that's set by default to 0, Nomad continues to use the initial 30s value. Renewing tokens in Vault in fairly cheap compared to creating new tokens, so this is a tradeoff to allow the Vault tokens to age-out quickly once the allocation exits.

[nicoche]

Hey! Thanks for your answer 🙂

If I understood correctly, nomad-client calls Vault's /auth/jwt/role/:name endpoint and always sets token_ttl to 30s; this is not configurable.

I also noticed that there is a lock here. Is there a single client for the whole nomad-agent? If yes, since renewing makes a network call (in our case over the Internet) and we have a lock, there's a limited number of calls we can make within 30s.
I think that explains an issue we had: we had a lot of 403 logs when trying to renew tokens. I think it's because nomad-client was not able to renew all needed tokens within 30s, so some of them expired.

I have a few questions:

• What's an identity TTL? I'm not sure I get what it is
• What's the benefit(s) of allowing Vault tokens to age-out quickly after an allocation exits?
• Can we make the 30s ttl configurable? Happy to provide a PR here, I would probably just need help with the naming of the configuration variable
• Is that lock necessary?

[tgross]

If I understood correctly, nomad-client calls Vault's /auth/jwt/role/:name endpoint and always sets token_ttl to 30s; this is not configurable.

No, sorry if I was unclear. Nomad calls Vault's /auth/jwt/login API. That /auth/jwt/role/:name endpoint is where you would call to configure Vault's JWT auth method, as described in the Nomad docs here.

I also noticed that there is a lock here. Is there a single client for the whole nomad-agent? If yes, since renewing makes a network call (in our case over the Internet) and we have a lock, there's a limited number of calls we can make within 30s.
...
Is that lock necessary?

Yes. This is also an unfortunate tradeoff -- every Vault API client we construct will open a new TCP connection to Vault. And you can't reuse the Vault client across Vault namespaces without setting it for the whole client (I'm not sure why they made this design decision), so we need to lock it during the request to make sure a request for a different Vault namespace swaps out the name mid-request.

What's an identity TTL? I'm not sure I get what it is

The identity.ttl is the TTL for the signed JWT's that Nomad generates for workloads (aka Workload Identity). This lets the workload use that token to talk to Nomad or to present the token to an external service which can verify the token came from Nomad. When you have a vault block in the job, Nomad automatically provides a WI intended to login to Vault (using that /auth/jwt/login endpoint). The Nomad client uses this WI on the workload's behalf to do so and set up the renewal loop. This means the Nomad client doesn't need a Vault token of its own. The Vault token and the identity have their own TTLs.

What's the benefit(s) of allowing Vault tokens to age-out quickly after an allocation exits?

This means if the token is leaked by the application, it won't be valid once the allocation is done.

Can we make the 30s ttl configurable?

It is! But the configuration is on the Vault side, in the auth configuration and roles you've configured.

[nicoche]

Thanks for those detailed answers!

✅ Noted for the lock.
✅ Noted for the TTL to be configured on Vault's side.

I'll check to see if that works for us, I'll circle back!