[rpc] Polkadot assethub rpc's randomly stop providing chain_getBlockHash which breaks polkadot.js api.

[stakeworld]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Hi,

I'm noticing some strange behaviour on polkadot assethub rpc's, were sometimes the rpc server stops providing some information and polkadot.js api blocks. One symptom I notice is that an rpc call for chain_getBlockHash returns no value. A restart fixes the problem. The problems keeps happening randomly over different nodes, without problems with disk space, resources etc. No specific errors in the logs.

Looking at polkadot.js it is affecting multiple providers: polkadot-js/apps#12094

This is expected:

root@monitor:/# websocat -1 -k wss://dot-rpc.stakeworld.io/assethub <<< '{"id":1,"jsonrpc":"2.0","method":"chain_getBlockHash","params":[0]}'
{"jsonrpc":"2.0","id":1,"result":"0x68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f"}
root@monitor:/#

This happens in error:

root@monitor:/# websocat -1 -k wss://rpc-asset-hub-polkadot.luckyfriday.io <<< '{"id":1,"jsonrpc":"2.0","method":"chain_getBlockHash","params":[0]}'
root@monitor:/# 

Also the parity endpoint is erroring but that seems maybe a different problem:

root@monitor:/# websocat -1 -k wss://polkadot-asset-hub-rpc.polkadot.io <<< '{"id":1,"jsonrpc":"2.0","method":"chain_getBlockHash","params":[0]}'
websocat: WebSocketError: WebSocketError: Received unexpected status code (429 Too Many Requests)
websocat: error running
root@monitor:/#

When in error a command like:

root@monitor:/# polkadot-js-api --ws wss://rpc-asset-hub-polkadot.luckyfriday.io rpc.system.version
2026-01-03 11:05:44        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2026-01-03 11:05:44        RPC-CORE: getRuntimeVersion(at?: BlockHash): RuntimeVersion:: No response received from RPC endpoint in 60s
2026-01-03 11:05:44        API/INIT: Error: FATAL: Unable to initialize the API: No response received from RPC endpoint in 60s
    at #onProviderConnect (file:///root/.nvm/versions/node/v20.16.0/lib/node_modules/@polkadot/api-cli/node_modules/@polkadot/api/base/Init.js:387:27)

times out.

Steps to reproduce

No response

[bkchr]

@stakeworld please provide some logs with jsonrpsee=trace,rpc=trace enabled. I don't know how much logs this will generate, but it would be perfect that you capture the logs exactly on when it hangs.

[stakeworld]

@stakeworld please provide some logs with jsonrpsee=trace,rpc=trace enabled. I don't know how much logs this will generate, but it would be perfect that you capture the logs exactly on when it hangs.

This is the last minute of log before my script restarts the node because it has 2 empty rpc responses (with options --log info,jsonrpsee=trace,rpc=trace). p.s. it was just after the reboot of my node to include the tracing so there is chance that could also be the cause, i will keep monitoring a bit longer and send logs if it happens again after a while running.

20260103T164504-sw-rpc-dot-asset-00.service-error.log.gz

[bkchr]

Someone is hammering author_pendingExtrinsics, otherwise I can not see anything suspicious.

[sigurpol]

We have started observing something potentially similar on the staking miner too where on different polkadot ah RPC nodes we see errors like

2m2026-01-03T19:50:11.332537Z�[0m �[33m WARN�[0m �[2mpolkadot-staking-miner�[0m�[2m:�[0m Non-critical listener error, continuing: Subxt(Rpc(ClientError(User(UserError { code: -32801, message: "Invalid block hash", data: None }))))

Or failure to connect on RPC nodes on restart.
This has started to happen recently and seems to happen so far on polkadot AH and not (yet ?) on kusama paseo or westend asset hub.
We have observed the issue on parity , dwellir and IBP1 nodes so far

[stakeworld]

@bkchr I've got a new log from this morning. Script checks chain_getBlockHash like in the description, when it doesn't get a response twice then it restarts the node and save the logs. I've now stopped the trace.

20260104T051505-sw-rpc-dot-asset-00.service-error.log.gz

For what it is worth: chatgpt's analysis:

chatgpt.md

Could it be that automatic scripts (like the scripts mentioned in the element group to regain access to funds) are overloading rpc subsystems? (maybe also related to the parity pokadot assethub rpc being overloaded)

Assethub polkadot rpc usage has increased significantly since around the end of december.

[bkchr]

@bkchr I've got a new log from this morning. Script checks chain_getBlockHash like in the description, when it doesn't get a response twice then it restarts the node and save the logs. I've now stopped the trace.

Under normal operations you are seeing chain_getBlockHash call took in the logs?

Assethub polkadot rpc usage has increased significantly since around the end of december.

Yeah this is probably because of this guy and its script.

[stakeworld]

Under normal operations you are seeing chain_getBlockHash call took in the logs?

They appear only in the trace logs so I honestly don't know... Maybe there is someone from the rpc devs who can take a look?

From an rpc provider perspective I see mainly that (only) polkadot assethub nodes randomly start having problems connecting to polkadot.js apps and one of the symptoms seems to be the chain_getBlockHash is not returned via rpc call.

Main problem is that after this the nodes seem to get in a bad state for rpc services, but further keep growing blocks etc so regular alerts don't get triggered. I've mitigated now by restarting by script after the above symptom arises but of course this is suboptimal.

[bkchr]

Yeah I could verify on my own using the Parity RPC node that chain_getBlockHash call took appears at some point in time.

[bkchr]

@stakeworld would it be possible that you test this #10757 on your RPC node?