Cloud storage plugin socket usage at capacity

[TimLanzi]

Link to reproduction

No response

Describe the Bug

I am using the cloud storage plugin with an S3 adapter pointing to DigitalOcean Spaces. I have users making a lot of edits relating to images lately. After a significant amount of uploads and image swapping, I get reports of performance issues on the admin dashboard and uploads failing. I looked at the logs and I found these error messages:

@smithy/node-http-handler:WARN socket usage at capacity=50 and 102 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 114 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 120 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 121 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 125 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 135 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 145 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 156 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 158 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 161 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 163 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 166 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 169 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 170 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.

These socket capacity warnings are sometimes preceded by error messages like this:

ERROR (payload): UnknownError
    err: {
      "type": "NoSuchKey",
      "message": "UnknownError",
      "stack":
          NoSuchKey: UnknownError
              at de_NoSuchKeyRes (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4809:21)
              at de_CommandError (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4747:19)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async /home/node/app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
              at async /home/node/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/index.js:225:18
              at async /home/node/app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
              at async /home/node/app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:173:18
              at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:97:20
              at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:120:14
              at async /home/node/app/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
      "name": "NoSuchKey",
      "$fault": "client",
      "$metadata": {
        "httpStatusCode": 404,
        "requestId": "[request id]",
        "attempts": 1,
        "totalRetryDelay": 0
      },
      "Code": "NoSuchKey",
      "BucketName": "bucket-name",
      "RequestId": "[request id]",
      "HostId": "[host id]"
    }

I have my Payload app hosted on a basic DigitalOcean Droplet (1 vCPU, 1GB RAM, 25GB storage, added 1GB swap space) if that helps at all.

To Reproduce

It seems like it happens when a lot of uploading is done. I haven't seen a clear-cut way to reproduce, but the best I can do is:

1. Set up a payload app with the cloud storage plugin
2. Point to a DigitalOcean Spaces bucket
3. Create an upload collection
4. Upload images in quick succession. Upload new images into existing records. Whatever can be done to trigger the errors.

Payload Version

2.14.1

Adapters and Plugins

"@payloadcms/bundler-webpack": "^1.0.6", "@payloadcms/db-mongodb": "^1.5.1", "@payloadcms/plugin-cloud-storage": "^1.1.2", "@payloadcms/plugin-nested-docs": "^1.0.12", "@payloadcms/plugin-seo": "^2.3.1", "@payloadcms/richtext-lexical": "^0.9.1", "@aws-sdk/client-s3": "^3.556.0", "@aws-sdk/lib-storage": "^3.556.0", "aws-crt": "^1.21.2"

[Stroi]

Also having this problem, after a while(not more than hour) we stop getting images and the same socket warning start showing up... Increasing the socket number didn't help.
We are hosting on render, images in S3 and DB is Mongo

[denolfe]

@TimLanzi @Stroi Can either of you give me an idea of your throughput for uploads? Seems like this might be tricky to reproduce.

[TimLanzi]

It's been a while so I can't really give exact numbers. If I had to guess, I'd say 15 upload events in 10 minutes. Image sizes were around 1MB each. I also have imageSizes set up as follows:

    imageSizes: [
      {
        name: 'thumbnail',
        width: 400,
        height: 300,
        position: 'centre',
      },
      {
        name: 'card',
        width: 768,
        height: 1024,
        position: 'centre',
      },
      {
        name: 'tablet',
        width: 1024,
        height: null,
        position: 'centre',
      },
    ],

[blobmold]

I have the same issue with almost the same payloadcms settings as @TimLanzi, same payloadcms and s3 errors when uploading images.

Edit: looks like the plugin is creating s3 socket connections that its not closing @denolfe

[creative-andrew]

I'm having exactly the same issue.

"name":"payload","err":{"type":"NoSuchKey","message":"The specified key does not exist.","stack":"NoSuchKey: The specified key does not exist.\n at de_NoSuchKeyRes (/home/node/backend/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4791:21)\n at de_CommandError (/home/node/backend/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4729:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/node/backend/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n at async /home/node/backend/node_modules/@aws-sdk/middleware-signing/dist-cjs/index.js:226:18\n at async /home/node/backend/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38\n at async /home/node/backend/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:174:18\n at async /home/node/backend/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:98:20\n at async /home/node/backend/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:121:14\n at async /home/node/backend/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22","name":"NoSuchKey","$fault":"client","$metadata":{"httpStatusCode":404,"requestId":"17E092AADF93FA23","extendedRequestId":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","attempts":1,"totalRetryDelay":0},"Code":"NoSuchKey","Key":"BPRO-Spender-EN.pdf","BucketName":"production","Resource":"/production/BPRO-Spender-EN.pdf",,"},"msg":"The specified key does not exist."}

2024-07-09T15:19:53.037446982Z stdout F {"level":50,"time":1720538393036,"pid":7, "name":"payload","err":{"type":"Error","message":"Socket timed out without establishing a connection within 10000 ms","stack":"TimeoutError: Socket timed out without establishing a connection within 10000 ms\n at Timeout._onTimeout (/home/node/backend/node_modules/@smithy/node-http-handler/dist-cjs/index.js:67:21)\n at listOnTimeout (node:internal/timers:573:17)\n at process.processTimers (node:internal/timers:514:7)","name":"TimeoutError","$metadata":{"attempts":3,"totalRetryDelay":146}},"msg":"Socket timed out without establishing a connection within 10000 ms"}

[denolfe]

I found a thread that deals with this specific issue: aws/aws-sdk-js-v3#3560

I would give a few of these solutions a try.

[creative-andrew]

My cloud hosting provider pointed also other issues related to this:

aws/aws-sdk-js-v3#6263

https://github.com/aws/aws-sdk-js-v3/blob/main/supplemental-docs/CLIENTS.md#request-handler-requesthandler

I believe I have finally managed to recreate the issue with my test setup. It required me to upload a video to payload, I then ran 120 curl instances at once with a timeout shorter than the video length and that seems to trigger the problem.
There is an interesting comment on the AWS SDK regarding socket exhaustion which helped me identify the problem.