API container runs out of memory when under pressure

[demisx]

We are seeing this pattern lately with the Freesky Gateway API container (Content Publishing Service). The memory usage starts under 1Gi, then gradually grows to the container’s limit of 8Gi causing this container to be OOMKilled and restarted. Then, the pattern repeats (see attached image). The Gateway image: v1.5.10.

Attached is the memory usage graph and these are the last messages in the container log right before it’s killed, if helps:

🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
🚀 Skipping global interceptor for this route
[2025-09-23 14:10:39.234 +0000] INFO (18): [HTTP] request aborted
    reqId: 9546
    req: {
      "id": 9546,
      "method": "POST",
      "url": "/v2/asset/upload",
      "query": {},
      "params": {
        "0": "v2/asset/upload"
      },
      "headers": {
        "accept": "application/json",
        "content-type": "multipart/form-data; boundary=--------------------------920030329608237099776511",
        "content-length": "2285587",
        "user-agent": "axios/1.9.0",
        "accept-encoding": "gzip, compress, deflate, br",
        "host": "freesky-gateway-content-publishing-api",
        "connection": "keep-alive"
      },
      "remoteAddress": "::ffff:10.184.10.195",
      "remotePort": 43602
    }
    res: {
      "statusCode": null,
      "headers": {
        "x-powered-by": "Express"
      }
    }
    responseTime: 7163
[2025-09-23 14:10:39.235 +0000] INFO (18): [HTTP] request aborted
    reqId: 9547
    req: {
      "id": 9547,
      "method": "POST",
      "url": "/v2/asset/upload",
      "query": {},
      "params": {
        "0": "v2/asset/upload"
      },
      "headers": {
        "accept": "application/json",
        "content-type": "multipart/form-data; boundary=--------------------------685911435340763964675746",
        "content-length": "2585087",
        "user-agent": "axios/1.9.0",
        "accept-encoding": "gzip, compress, deflate, br",
        "host": "freesky-gateway-content-publishing-api",
        "connection": "keep-alive"
      },
      "remoteAddress": "::ffff:10.184.10.195",
      "remotePort": 43592
    }
    res: {
      "statusCode": null,
      "headers": {
        "x-powered-by": "Express"
      }
    }
    responseTime: 7163
[2025-09-23 14:10:39.235 +0000] INFO (18): [HTTP] request aborted
    reqId: 9548
    req: {
      "id": 9548,
      "method": "POST",
      "url": "/v2/asset/upload",
      "query": {},
      "params": {
        "0": "v2/asset/upload"
      },
      "headers": {
        "accept": "application/json",
        "content-type": "multipart/form-data; boundary=--------------------------166933008538460882381288",
        "content-length": "2369989",
        "user-agent": "axios/1.9.0",
        "accept-encoding": "gzip, compress, deflate, br",
        "host": "freesky-gateway-content-publishing-api",
        "connection": "keep-alive"
      },
      "remoteAddress": "::ffff:10.184.10.195",
      "remotePort": 43590
    }
    res: {
      "statusCode": null,
      "headers": {
        "x-powered-by": "Express"
      }
    }
    responseTime: 7162
[2025-09-23 14:10:39.235 +0000] INFO (18): [HTTP] request aborted
    reqId: 9545
    req: {
      "id": 9545,
      "method": "POST",
      "url": "/v2/asset/upload",
      "query": {},
      "params": {
        "0": "v2/asset/upload"
      },
      "headers": {
        "accept": "application/json",
        "content-type": "multipart/form-data; boundary=--------------------------941532794754161450863739",
        "content-length": "2483804",
        "user-agent": "axios/1.9.0",
        "accept-encoding": "gzip, compress, deflate, br",
        "host": "freesky-gateway-content-publishing-api",
        "connection": "keep-alive"
      },
      "remoteAddress": "::ffff:10.184.10.195",
      "remotePort": 43612
    }
    res: {
      "statusCode": null,
      "headers": {
        "x-powered-by": "Express"
      }
    }
    responseTime: 7165
[2025-09-23 14:10:39.235 +0000] INFO (18): [HTTP] request aborted
    reqId: 9544
    req: {
      "id": 9544,
      "method": "POST",
      "url": "/v2/asset/upload",
      "query": {},
      "params": {
        "0": "v2/asset/upload"
      },
      "headers": {
        "accept": "application/json",
        "content-type": "multipart/form-data; boundary=--------------------------884909884495295432736898",
        "content-length": "2635205",
        "user-agent": "axios/1.9.0",
        "accept-encoding": "gzip, compress, deflate, br",
        "host": "freesky-gateway-content-publishing-api",
        "connection": "keep-alive"
      },
      "remoteAddress": "::ffff:10.184.10.195",
      "remotePort": 43618
    }
    res: {
      "statusCode": null,
      "headers": {
        "x-powered-by": "Express"
      }
    }
    responseTime: 7166
[2025-09-23 14:10:39.236 +0000] INFO (18): [HTTP] request aborted
    reqId: 9635
    req: {
      "id": 9635,
      "method": "POST",
      "url": "/v2/asset/upload",
      "query": {},
      "params": {
        "0": "v2/asset/upload"
      },
      "headers": {
        "accept": "application/json",
        "content-type": "multipart/form-data; boundary=--------------------------923330792595314763305135",
        "content-length": "3052179",
        "user-agent": "axios/1.9.0",
        "accept-encoding": "gzip, compress, deflate, br",
        "host": "freesky-gateway-content-publishing-api",
        "connection": "keep-alive"
      },
      "remoteAddress": "::ffff:10.184.10.195",
      "remotePort": 42402
    }
    res: {
      "statusCode": null,
      "headers": {
        "x-powered-by": "Express"
      }
    }
    responseTime: 5982

[demisx]

In case this helps, I wanted to point out that after Jeetu cleared batch files from S3 and I bounced the batch-announcer pod yesterday, the Gateway API container memory usage has been consistently low, 20% of the 1Gi request and 0 container restarts. I am attaching the chart for the past 12 hours where you can see the memory usage is very low. So, the leak appears to happen when the gateway is under pressure.

[shannonwells]

I took a couple of approaches to analyzing this, one, I used Claude Code to look for a couple of the most obvious types of things that can cause a memory leak, but found none:

• there are about 15 event registrations, which are all automatically cleaned up
• there were no promises that went unresolved
• Launching the content-publishing-api shows, when looking at prometheus running locally, even without hitting endpoints, a slow linear-ish increase in process memory.

Next I wrote a k6 script that hits the v2/asset/upload endpoint (see #978 ) and looked at the Prometheus for signs of a problem. I saw the memory increases reported. Larger file uploads caused larger memory increases. Sometimes the memory was eventually deallocated and sometimes it wasn't.

Then I found a profiler called clinic and fired it at the endpoint using the k6 test- actually I switched between a v1 endpoint using small files and this one. Clinic Doctor analysis indicated a potential IO bottleneck no matter which endpoint it was hitting. It recommended to use the bubbleprof command (which didn't work) and/or flame command for further analysis.

I ran the flame command using the same k6 script. I noticed the following (see screenshot):

• The “hottest” functions were at the bottom of the graph or very low, or didn't run for long. So nothing really seemed to be hogging compute time.
• Very tall stack frames existed whenever we call the polkadot-js api
• Lots of idle time in the crypto lib C++ bindings

It was agreed to set this aside for now given that cleaning up the batches seems to have resolved the issue for now.

[demisx]

Just wanted to confirm @shannonwells findings, that even without pressure the Gateway API container on Freesky eventually runs out its current 12Gi limit and container restarts. It just takes longer for this to happen (3-6 hrs vs 45 mins). Attached is the Gateway API container memory usage graph for the past 12 hours.

[aramikm]

@shannonwells @demisx I'm curious if we found the issue for this or still under investigation? Do we know at least which gateway service might causing this memory leak it it does happen on all of them?

[demisx]

@aramikm This abnormality has been observed in the Freesky project, which currently runs only a content publishing service. My understanding is that this issue is currently on hold due to other higher-priority tasks.