-
Notifications
You must be signed in to change notification settings - Fork 2k
Description
After an allocation is terminal, the Nomad client needs to keep the allocation directory around so that users can read their logs to debug failed allocations. But to do so, Nomad keeps the entire AllocRunner alive, which uses up memory resources we should be freeing for the user's workloads (sometimes substantially, ex. #25269), and complicates doing the actual GC (ex. #25123). At the end of the day users want the alloc dir to persist as long as they don't otherwise need the disk space, but shouldn't have to pay for all this overhead.
Nomad should instead split the allocrunner and the alloc directory into separate objects, so that we can abandon the allocrunner once the allocation is terminal, and keep the alloc directory around until GC so we can serve /v1/client/fs APIs like Stream Logs.
There are several subtasks to figure out, mostly around the cleanup operations in allocrunner hooks:
- The
csi_hookhas aDestroyhook method (triggered on GC). It looks like this only cancels any in-flight requests. Can we safely move this to aPostrunhook? - The
consul_hookhas aDestroyhook method. This exactly copies thePostrunhook method. Why do we feel like we need both here? Do we have a resource cleanup problem that we're trying to paper over? - The
identity_hookhas aDestroyhook method. This exactly copies thePreKillandShutdownhook methods. Why do we feel like we need all three here? Why isn't there aPostrunhook? - The
allocdir_hookhas aDestroyhook method which is used to cleanup on GC. We would need to move this out of the allocrunner entirely and into whatever "alloc directory proxy" we build. - The alloc FS RPCs defined in
fs_endpoint.gowill need reworking to use some new client object as a proxy to the allocdir. It looks like there are a small number of methods to implement (ex. we need the allocation ID to find the allocdir and its namespace to check auth). - There will be a bunch of refactoring to do in
client/client.goandclient/gc.goso that we're ensuring the allocrunner is completely dead before cleaning up the disk for GC.
(internal ref https://hashicorp.atlassian.net/browse/NET-12306)