Vault Agent flooding the server with "/v1/sys/leases/renew" requests after the lease expired

[jrutkowski]

I have noticed a strange pattern, when looking through vault server load balancer metrics. There were tons of request, with "403 Forbidden" response code recorded. Quick investigation led me to instances of vault agent, configured with AWS IAM auto_auth and dynamic credentials caching enabled.

tcpdump showed, that after some time from obtaining first auth token and a set of dynamic credentials, agent started flooding the server with the lease renewal request (/v1/sys/leases/renew), that at that time were no longer in vault (evicted due to the auth token expiration). The server kept responding with "403 Forbidden", because the lease was no longer valid and the agent kept repeating the request over and over again.
The interesting part was that those request were not visible in agent logs, even with most detailed "TRACE" log level.

I think, that this happens, when vault agent tries to determine whether cached credentials are still valid or if they can be renewed. In usual use case a response is returned, which is then used to decide if data should be evicted from the cache or renewed. When the auth token (with which credentials were generated) has expired, that is no longer possible. Program gets trapped in a loop and keeps sending the "/v1/sys/leases/renew" request.

Details specific to the configuration which triggers the issue:

agent-config.hcl:

pid_file = "./agent.pid"
log_level = "trace"

vault {
  address = "http://vault_addr:8200"
  retry {
    num_retries = 5
  }
}

auto_auth {
  method "aws" {
    mount_path = "auth/aws"

    config = {
      type = "iam"
      role = "auth_role"
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address = "0.0.0.0:8100"
  tls_disable = true
}

auth_role:

Key                               Value
---                               -----
allow_instance_migration          false
auth_type                         iam
bound_account_id                  []
bound_ami_id                      []
bound_ec2_instance_id             <nil>
bound_iam_instance_profile_arn    []
bound_iam_principal_arn           [arn:aws:iam::xxxxxx:role/auth_role]
bound_iam_principal_id            [...]
bound_iam_role_arn                []
bound_region                      []
bound_subnet_id                   []
bound_vpc_id                      []
disallow_reauthentication         false
inferred_aws_region               n/a
inferred_entity_type              n/a
resolve_aws_unique_ids            true
role_id                           ...
role_tag                          n/a
token_bound_cidrs                 []
token_explicit_max_ttl            0s
token_max_ttl                     1m
token_no_default_policy           false
token_num_uses                    0
token_period                      0s
token_policies                    [policies]
token_ttl                         1m
token_type                        service

database role details:

Key                      Value
---                      -----
creation_statements      [...]
credential_type          password
db_name                  ...
default_ttl              1m30s
max_ttl                  1m30s
renew_statements         []
revocation_statements    [...]
rollback_statements      []

What's important to note here is that the issue appears when the token type is set to "service" and max_ttl is applied (not a periodic token).

I see 2 possible solutions:

• When auto_auth token expires and new one is obtained - make sure that all the cache related to the previous token is evicted.
• When "403 Forbidden" is received, when trying to renew cached lease, treat it as it was no longer valid, and evict from cache.

If needed, I am willing to invest some time into helping to solve this issue (including writing some code). Please let me know if that would be welcome.

[aitch2o-development]

I'm seeing this as well, even with token not expired, we think it might be throttling from AWS on auth calls from instance profile

[matthisholleville]

We are experiencing exactly the same issue ! We went from 30 request per seconds to over 1100 for 1 hour and 10 minutes.

We switched leader and that seems to have solve the flood.

PS: I don't think this is specific to AWS. We use GCP, and we're experiencing exactly the same issue.

cc. @jefferai

[matthisholleville]

We have just experienced the same behavior again. Moreover, we notice that this issue occurs regularly.

I wonder if the problem might originate here: https://github.com/hashicorp/vault/blob/main/api/lifetime_watcher.go#L264. Shouldn't we attempt a maximum renew attempt?

[golgoth31]

Still present in v2.0.3. We hit this on GCP/Kubernetes with the Kubernetes auth method, so adding a data point that it is not specific to AWS IAM, plus a root-cause trace and a PR.

Where it comes from

RegisterAutoAuthToken only ever adds:

https://github.com/hashicorp/vault/blob/v2.0.3/command/agentproxyshared/cache/lease_cache.go#L1763-L1819

If the token is already indexed the call is a no-op; otherwise a new index is created with a context derived from the cache's base context. The previous auto-auth token's index is never removed and its context is never cancelled.

Its only caller does nothing else:

https://github.com/hashicorp/vault/blob/v2.0.3/command/agentproxyshared/sink/inmem/inmem_sink.go#L36-L44

The cascade needed to clean this up already exists — a lease's renewal context is derived from its token's context:

https://github.com/hashicorp/vault/blob/v2.0.3/command/agentproxyshared/cache/lease_cache.go#L544

so cancelling the token's context would stop every lease watcher derived from it and let startRenewing's defer evict them. Nothing calls it on re-authentication. lease_cache.go has had four commits since January 2024 (static-secret caching race, versioned-secret fix, a Proxy bug, a license header), none touching this path.

Note this is distinct from #26383, which fixed api.LifetimeWatcher discarding its exponential backoff on the error path. That fix landed in 1.17.0 and is present in v2.0.3; it bounds how fast a doomed watcher retries, but does not stop it from being doomed. It was not backported — we checked the v1.15.16 and v1.16.13 changelogs.

What it looks like from the outside

Agent 1.14.0, Kubernetes auth, dynamic database credentials rendered through a template, cache enabled. Reproduced deliberately by setting the auth role's token_max_ttl to 300s and the database lease TTL to 600s, so the lease outlives its parent token. Three agent pods, ~16 auto-auth generations each.

One rotation, from the agent's own debug logs:

15:29:28   storm begins server-side (token already dead)
15:29:38   lifetime watcher done channel triggered
15:29:38   authentication successful, sending token to sinks   <- healthy new token, 10s later
15:29:38   initiating renewal db-creds                         <- new watchers started
   ...     storm continues against the OLD token
15:30:54   renewal halted; evicting from cache  db-creds          (+76s)
15:31:20   renewal halted; evicting from cache  sys/leases/renew  (+102s)

Auto-auth recovers within 10 seconds. The orphaned watchers keep going for another 76 and 102 seconds, and only stop when each one gives up on its own.

From the Vault audit log, one token/lease pair, one pod:

34 088 failing sys/leases/renew in ~150s   (~227 req/s average, ~356 req/s peak)

Across three pods rotating out of phase, roughly 1 100 req/s sustained in ~2 minute bursts on every rotation, against a leader running on 2 vCPUs. That matches @matthisholleville's "30 request per seconds to over 1100" almost exactly.

Two things that made this hard to find, echoing the original report:

• Nothing is visible at info. The lease cache logs renewal activity at debug, and the failing requests are never logged at all — only the eventual renewal halted; evicting from cache. We had to raise the sidecar to debug to see the mechanism.
• Vault's own telemetry does not show it. Over the exact window of a storm, sum(rate(vault_core_handle_request[2m])) read 0.02–1.5 and sum(rate(vault_expire_renew_token[2m])) read 0–0.25, while the audit log recorded ~1 100 req/s. Three orders of magnitude apart, so a request-rate alert on the exported metrics will not catch this.

Conditions

The lease has to outlive its parent token for the window to exist. If the secret engine's TTL is shorter than the token's, the watcher reaches its own deadline while the parent is still alive, fails cleanly and is evicted — we measured that case and it is well behaved. The exposure is therefore worst with long-lived dynamic secret leases capped by the token's remaining TTL, which is the common configuration (TTL of "24h" exceeded the effective max_ttl of "2h57m34s"; TTL value is capped accordingly).

PR

#32060 records the current auto-auth token and cancels the context of the one it replaces, driving the cascade that already exists.

Two details in there that may be worth a maintainer's opinion:

1. The cancellation has to run before the "token already cached" short-circuit, otherwise it never arms after a restart with a persistent cache, since restoreTokens has already re-added the previous token. That is the case where stale watchers are most likely to be running.
2. It cancels on any token change, not only on expiry. auth.go also re-authenticates on new credentials (credCh) or after a transient watcher error, where the old token may still be valid; its leases are then dropped and re-fetched rather than kept renewed. The narrower alternative is to drive this from the InvalidToken path instead. Happy to go that way if preferred.

One case the PR deliberately does not cover: if the persisted token is rejected at startup, the agent re-authenticates without the cache ever being told which token was live, so the restored leases keep renewing against it. Fixing that means seeding the tracker from restoreTokens, which felt like a separate change.