High CPU and Memory usage due to develop container

[jakubgs]

Summary:

Today I found the develop container using most of the CPU and Memory on the host:

And spewing these logs:

RuntimeError: unreachable
    at wasm://wasm/005420d6:wasm-function[1077]:0x135a01
    at fM (/app/.next/server/pages/api/og.js:66:56317)
    at e.wbg.__wbindgen_string_get (/app/.next/server/pages/api/og.js:66:61187)
    at wasm://wasm/005420d6:wasm-function[30]:0x3fcde
    at new fH (/app/.next/server/pages/api/og.js:66:58408)
    at new f1 (/app/.next/server/pages/api/og.js:66:62302)
    at pr (/app/.next/server/pages/api/og.js:66:64887)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.start (/app/.next/server/pages/api/og.js:66:65273)
RuntimeError: unreachable
    at wasm://wasm/005420d6:wasm-function[27]:0x38c4e
    at wasm://wasm/005420d6:wasm-function[30]:0x3fdcc
    at new fH (/app/.next/server/pages/api/og.js:66:58408)
    at new f1 (/app/.next/server/pages/api/og.js:66:62302)
    at pr (/app/.next/server/pages/api/og.js:66:64887)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.start (/app/.next/server/pages/api/og.js:66:65273)

Restart helped, at least for now.

Environment:

• Operating System: Alpine Linux 3.17
• Project Version: sha256:221faa7c3b63afc28ddb3788800874d001ef5cc720a37539e510de7161e8331c

[jakubgs]

It appears this issue has not really been resolved, this is from today:

Very high CPU and Memory usage spike. CPU eventually settles but memory is not freed.

[jakubgs]

I moved logos-press-engine and free-technology setups to a separate host due to these memory issues over a week ago.

• infra-misc#e7be53f6 - aptly-server: add broken Status logo image

Which was a good move, because now the persistent memory issues do not affect other websites.

This is the node-01.do-ams3.press.misc host right now:

Can devs here investigate this?

[jakubgs]

Here's free-technology container:

And here's logos-press-engine container over 7 days:

[jinhojang6]

Thanks for the updates. Did you find any logs related to this issue?

@jakubgs

[jakubgs]

You can find all the logs on Kibana:
https://kibana.infra.status.im/goto/7939e460-d740-11ee-aaa4-85391103106b

But currently there's barely any logs, and the containers are behaving so far:

Here's the current trend:

https://grafana.infra.status.im/d/QCTZ8-Vmk/single-host-dashboard?orgId=1&refresh=1m&var-host=node-01.do-ams3.press.misc&from=1708633580552&to=1709238380552

Ignore the drop off a the end, we are having issues with our metrics cluster: https://github.com/status-im/infra-hq/issues/125

[jinhojang6]

Thanks @jakubgs

Does the develop container mean there is an issue with the develop(staging) branch, and the production (live) website is okay?

I think there is an issue with vercel/og given the logs you provided

RuntimeError: unreachable
    at wasm://wasm/005420d6:wasm-function[1077]:0x135a01
    at fM (/app/.next/server/pages/api/og.js:66:56317)
    at e.wbg.__wbindgen_string_get (/app/.next/server/pages/api/og.js:66:61187)
    at wasm://wasm/005420d6:wasm-function[30]:0x3fcde
    at new fH (/app/.next/server/pages/api/og.js:66:58408)
    at new f1 (/app/.next/server/pages/api/og.js:66:62302)
    at pr (/app/.next/server/pages/api/og.js:66:64887)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.start (/app/.next/server/pages/api/og.js:66:65273)

But we are using it both of the production & staging website to produce the og-image (link previews) on the server.

[jakubgs]

I don't believe this issue is specific to develop, it was simply first observed on develop.

If you look at a 30 day graph you see it always goes up to the limit of 4 GB and releases memory only after restart:

Ignore the two big spikes at the end, that's an artifact due to metrics cluster latency.

The host is constantly using pretty much all the SWAP memory:

[siddarthkay]

Hey @jinhojang6 : I was planning on assigning this to myself and to begin investigating this issue if you don't mind.
Also would you have any additional notes other than what is already logged in this issue ?

Thank you.

[jinhojang6]

Thanks for the help @siddarthkay .

I don't have further information yet. Just suspect that the vercel/og makes the problem as written in the error log.

RuntimeError: unreachable
    at wasm://wasm/005420d6:wasm-function[1077]:0x135a01
    at fM (/app/.next/server/pages/api/og.js:66:56317)
    at e.wbg.__wbindgen_string_get (/app/.next/server/pages/api/og.js:66:61187)
    at wasm://wasm/005420d6:wasm-function[30]:0x3fcde
    at new fH (/app/.next/server/pages/api/og.js:66:58408)
    at new f1 (/app/.next/server/pages/api/og.js:66:62302)
    at pr (/app/.next/server/pages/api/og.js:66:64887)

• reference: https://vercel.com/docs/functions/og-image-generation

[jinhojang6]

@siddarthkay @jakubgs

It seems that many people who are running NextJS with Docker experience the same issue.
vercel/next.js#61047

Updating to Next v14.1.3 (published last week) could solve the problem.

[siddarthkay]

yeah my money was on nextjs.. although upgrading to 14 would be a disruptive change no @jinhojang6 ?
since it uses an experimental router and everything ?

[jinhojang6]

@siddarthkay You are totally right. But I think we have no choice but to upgrade it to resolve this memory leak issue. What do you think?

[siddarthkay]

I am always up for upgrading stuff :D @jinhojang6
I was trying to recreate the problem locally by profiling the app and trying to find out exactly where the leak comes from.
But yeah planning for an upgrade is always good and something that we should often do.