Runaway memory usage

[jrudolph]

Mountpoint for Amazon S3 version

mount-s3 1.19.0

AWS Region

us-east1

Describe the running environment

Access a TB-sized dataset in EKS using mountpoint-s3 using the csi driver (recent al2023 ami).

Mountpoint options

Mostly default settings


--metadata-ttl indefinite --read-part-size 5242880 --allow-other --region us-east-1

What happened?

For a workload where we mount a large (~2.4TB / 1200 * 2GB files) dataset that is accessed mostly in a random-access fashion, we see rampant memory usage after a short but heavy burst of access leading ultimately to OOM situations. We've seen the mount-s3 process use more than 15GB of memory and cannot find a way to limit its memory usage.

We are using mountpoint-s3 via the CSI driver.

Things we tried:

• decrease --read-part-size to its lower limit of 5242880 (no effect)
• use an empty dir cache (seemingly no effect)
• updating to the new v2 csi-driver (I tried that after the old driver let k8s nodes just die because the systemd service just gobbled up all memory and the node went into unresponsiveness). With the v2 driver and setting mountpointContainerResourcesLimitsMemory I can see the mppods being OOMKilled (which is better for the cluster than hanging nodes, but doesn't help the application, restarting the mppod does not gracefully reestablish access for the application). Gradually increasing mountpointContainerResourcesLimitsMemory, it becomes harder and harder to schedule the application at all.

Not tried:

• changing UNSTABLE_MOUNTPOINT_MAX_PREFETCH_WINDOW_SIZE because it's not supported by the CSI driver. Also, it's not clear how the default of 2GiB could still lead to >10GiB used.

In general, if there would be any limit we could use, it might just be good enough for us for the time being (even if it might make scheduling in a K8s cluster harder). In general, it would be better to limit memory usage cleanly and in the best case pick up the max memory from the cgroup environment.

Relevant log output

[c-hagem]

Hi, sorry to hear that you are seeing such high memory usage with Mountpoint. As you already discovered, our general recommendation would be to try using the UNSTABLE_MOUNTPOINT_MAX_PREFETCH_WINDOW_SIZE , however this is currently not possible with CSI Driver V2, you can follow the progress on enabling this flag for V2 here. For V1 of csi driver, you should be able to inject the environment variable, as described here.

We are currently working on memory optimisations to Mountpoint, like sharing the memory pool with the CRT library for acessing S3 (c.f. PR for Memory Pool). While this optimisation is not released yet (and we do not have an ETA for that), in our current testing it seems to reduce Mountpoint's memory consumption. So we hope that use cases like yours will work better with Mountpoint in the future.

[passaro]

Hi @jrudolph,

We are looking into optimizing memory usage in Mountpoint and we'd like to better understand your issue. Would you be able to provide a little more detail?

For a workload where we mount a large (~2.4TB / 1200 * 2GB files) dataset that is accessed mostly in a random-access fashion, we see rampant memory usage after a short but heavy burst of access leading ultimately to OOM situations. We've seen the mount-s3 process use more than 15GB of memory and cannot find a way to limit its memory usage.

• "short but heavy burst of access": how many concurrently open files (roughly)?
• could you tell whether the memory usage goes up during a single burst (and eventually OOM) or keeps growing across multiple bursts and does not decrease?

If it is the latter, we suspect the cause may be a memory fragmentation issue we are currently working to address.

[jmccl]

My understanding is the 2GiB is per request, so with sufficient concurrent requests it could eat up more than 10GiB.

[renanmagagnin]

Mountpoint for Amazon S3 v1.20.0 is now available with significant reduction in overall memory usage. Would you be able to upgrade to check how it improves this use case? We would love to hear your feedback on this!

[jrudolph]

Hey, sorry I dropped that a bit after vacation. For the problematic workload, we moved away from mountpoint (but still use it in lots of other places), if I find some time I'll try the new release.