FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[pmensik]

Description

I am running Sidecar alongside Polkadot AssetHub node (64 GB RAM machine, a decent CPU). One of the clients seems to keep posting an invalid extrinsic every minute which results in error: RPC-CORE: submitExtrinsic(extrinsic: Extrinsic): Hash:: 1012: Transaction is temporarily banned. I am not sure if it's related but the Sidecar subsequently crashes after exactly one hour with the out-of-memory error.

I tried to increase the memory with export NODE_OPTIONS="--max-old-space-size=8192" in both my .bashrc and and in my systemd running the Sidecar but it doesn't seem to take any effect. Posting my systemd config for clarity as well

[Unit]
Description=substrate api sidecar

[Service]
User=user
Group=user
ExecStart=/usr/local/bin/substrate-api-sidecar 


Restart=always
RestartSec=5

Environment="SAS_EXPRESS_BIND_HOST=0.0.0.0"
Environment="NODE_OPTIONS="--max-old-space-size=8192""

[Install]
WantedBy=multi-user.target

Steps to Reproduce

I installed a 2nd node from scratch and that's yielding the same error (both are behind the same load balancer).

sudo apt install nodejs
sudo apt install npm
sudo npm install -g @substrate/api-sidecar
sudo npm install -g @polkadot/api-augment

So the versions are following

node - v18.19.1
npm - 9.2.0
sidecar - v20.5.0

[Provide detailed steps for someone else to reproduce your bug.]

Expected vs. Actual Behavior

Node shouldn't crash after receiving such requests.

[TarikGul]

Hey @pmensik so there has definitely been memory issues in the past with sidecar - can you further explain some of your incoming requests? What is your most queried endpoints and is it mostly stuff at the tip of the chain, or is there a lot of historic queries as well?