SPIRE agent can exceed the RPC timeout when synchronising bundles with many federations

[markgoddard]

SPIRE agent periodically syncs entries, federated bundles and SVIDs from the SPIRE server. This happens every 5 seconds, with a backoff applied on failure. There is a 30 second timeout for the update process (in pkg/agent/client/client.go):

const rpcTimeout = 30 * time.Second

If there are many federations for the entries allowed for the agent, this timeout can be exceeded. If the timeout is exceeded, then the update fails and no entries, bundles or SVIDs are updated. If this happens persistently then SVIDs may expire. This is particularly apparent in resource constrained environments.

Part of the problem here is that federated bundles are fetched sequentially (in fetchBundles()).

[sorindumitru]

Thanks for the issue and the pr for it, @markgoddard! The approach in the PR should be ok, but I guess users may still run into problems after a certain number of federation relationships.

One other idea here could be to extend the SyncAuthorizedEntries (and maybe GetAuthorizedEntries) RPC to also include the bundles for the entries the agent has synced. This way the agent gets everything in one single RPC and we may be able to batch some bundles together in the packet. There's a bunch of things that need to be discussed about how that would work, but before doing that, do you think something like that would help in your case?

[markgoddard]

@sorindumitru agreed - this only gets us to the next limit. There are certainly a few other ways this could be improved, but this one was the simplest and quickest to implement.

From what I can see, SyncAuthorizedEntries already looks like quite a complex bidirectional streaming API and could be complicated by the addition of bundles. Perhaps a similar sync API could be built for bundles? It would definitely be good to avoid downloading bundles unnecessarily as it can amount to quite a lot of data.

Some other things I had thought of:

• add a BatchFetchFederatedBundle RPC
• increase caching of bundles in the server and agent
• add a revision field to bundles to allow fetching only changes (this would probably be a prerequisite for a sync RPC)

Given that we have a PR ready to go, should we go with that approach for now?

[sorindumitru]

Discussed this today, we're ok with having this change and leaving bigger changes for the future since those may take longer due to API changes.