'Timeout after {} seconds while waiting for mount process to be ready'

[thundergolfer]

This is an incomplete issue report. To get things started I'm just interested in learning how to prioritize observability based on this problem. This timeout is being hit <1% of the time, so I can't really enable debug logging globally to catch it.

All I see in mountpoint logs is this:

[ERROR] mountpoint_s3::cli: timeout after 30 seconds waiting for message from child process

---

Mountpoint for Amazon S3 version

mount-s3 1.15.0-unofficial+fd1d51e-dirty

AWS Region

several

Describe the running environment

Running on EC2, OCI, and GCE hosts in various regions. The error doesn't seem to be correlated to a specific cloud or specific region.

Mountpoint options

mount-s3 --user-agent-prefix modal-task-id-ta-01K22EYDDC9PYV7HNRWYXDQ7PD --part-size=67108864 --maximum-throughput-gbps=25 --allow-delete --allow-overwrite --endpoint-url $REDACTED --region auto --cache /tmp/task-data/ta-01K22EYDDC9PYV7HNRWYXDQ7PD/mountpoint-disk-cache1k4d35 --max-cache-size 524238 --upload-checksums=off bioqa /tmp/task-data/ta-01K22EYDDC9PYV7HNRWYXDQ7PD/mountpoint-E8Ot3r

What happened?

Hitting the 'Timeout after {} seconds while waiting for mount process to be ready' timeout after 30s (default on main) and also now 60s after bumping the timeout.

Failed to mount cloud bucket: mount-s3 command failed: Error: Timeout after 60 seconds while waiting for mount process to be ready

Relevant log output

[c-hagem]

Hi, without logs this is very hard for us to investigate. Since you are using the cache it would be interesting to know if the cache directory is empty before you run Mountpoint, or if you are relying on Mountpoint to delete the cache? We know that cache cleanup not being finished sometimes causes Mointpoint to fail to start, especially with large cache directories. Could that be the case in your deployment?