504 Gateway Timeout on getMultipleAccounts Calls

[andreclaro]

Problem

The Solana RPC Node running agave validator is unable to answer to getMultipleAccounts calls.

Example:

 curl -X POST http://localhost:8899   -H "Content-Type: application/json"   -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getMultipleAccounts",
    "params": [
      [
        "Jito4APyf642JPZPx3hGc6WWJ8zPKtRbRs4P815Awbb",
        "SysvarC1ock11111111111111111111111111111111"
      ],
      {
        "encoding": "base64",
        "commitment": "finalized",
        "dataSlice": {
          "offset": 0,
          "length": 282
        },
        "minContextSlot": 331054443
      }
    ]
}'

Only after restarting the service, it is able to reply to this type of queries:

curl -X POST http://localhost:8899   -H "Content-Type: application/json"   -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getMultipleAccounts",
    "params": [
      [
        "Jito4APyf642JPZPx3hGc6WWJ8zPKtRbRs4P815Awbb",
        "SysvarC1ock11111111111111111111111111111111"
      ],
      {
        "encoding": "base64",
        "commitment": "finalized",
        "dataSlice": {
          "offset": 0,
          "length": 282
        },
        "minContextSlot": 331054443
      }
    ]
}'
{"jsonrpc":"2.0","result":{"context":{"apiVersion":"2.1.16","slot":331062721},"value":[{"data":["AUUePdUNO3uFNgRcK3rC7CWUc+vCWuO8vh++sX1S+8e+eXhXwruGsaac0PTcoWwisNzj3eyWuEBcCPHEcDrQj9NUtd6+o5sz4PHc+gqPYiqVuLTrluhPL6HjF2cPHpbB2P0j4HUJut3t/bUWqQuRl7tQR0MlXQ43xf9dzookHu3EMZ6naP7fZEyKrpuOIYit0GvFUPv3Fsgiuc5jx3g9lS4f/NFB6YMsrxCtkXSVyg8nG1spPNRwJ+pzcAftQOs5oL0J5qP+7PmQMuHB32uXItyzY057jjRAk2vDSwzByOtSHwbd9uHXZaGT2cvhRs7reawctIXtX1s3kTqM9YV+/wCpOteHlOkPPAD9t6jbE2EyAP4CAAAAAAAA","base64"],"executable":false,"lamports":5143440,"owner":"SPoo1Ku8WFXoNDMHPsrGSTSG1Y47rzgn41SLUNakuHy","rentEpoch":18446744073709551615,"space":611},{"data":["wp27EwAAAABx0O1nAAAAAP4CAAAAAAAA/wIAAAAAAADBu+5nAAAAAA==","base64"],"executable":false,"lamports":1169280,"owner":"Sysvar1111111111111111111111111111111111111","rentEpoch":18446744073709551615,"space":40}]},"id":1}

Logs

I don't see any related log message in agave-validator logs...

What should I look for?

Version

agave-validator --version
agave-validator 2.1.16 (src:00000000; feat:3271415109, client:Agave)

Config

/usr/local/bin/agave-validator \
    --identity /var/lib/solana/keys/validator-keypair.json \
    --entrypoint entrypoint.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \
    --known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \
    --known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \
    --known-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ \
    --known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \
    --dynamic-port-range 8000-8020 \
    --no-voting \
    --private-rpc \
    --full-rpc-api \
    --rpc-port 8899 \
    --rpc-bind-address 0.0.0.0 \
    --enable-rpc-transaction-history \
    --enable-cpi-and-log-storage \
    --rpc-pubsub-max-active-subscriptions 10000 \
    --wal-recovery-mode skip_any_corrupted_record \
    --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
    --no-snapshot-fetch \
    --maximum-incremental-snapshots-to-retain 2 \
    --incremental-snapshot-interval-slots 500 \
    --full-snapshot-interval-slots 10000 \
    --accounts /var/lib/solana/accounts \
    --accounts-hash-interval-slots 1000  \
    --disable-accounts-disk-index \
    --ledger /var/lib/solana/data/ \
    --snapshots /var/lib/solana/snapshot/ \
    --limit-ledger-size 185000000 \
    --health-check-slot-distance 60  \
    --no-port-check \
    --no-poh-speed-test

[andreclaro]

logs-504-getaccount-issue.txt

[andreclaro]

"All queries to get account(s) information fail. For example:

curl -s -X POST http://localhost:8899/   -H "Content-Type: application/json"   -d '{"jsonrpc":"2.0","id":1,"method":"getAccountInfo","params":["Jito4APyf642JPZPx3hGc6WWJ8zPKtRbRs4P815Awbb", {"encoding":"base64"}]}'