`eth_getBlockByNumber` returns empty response on Ethereum mainnet

[rgelinas-polygon]

Using the latest dshackle release 0.15 with the following minimal reproduction config:

version: v1
port: 2449

proxy:
  host: 0.0.0.0
  port: 8545
  routes:
    - id: ethereum
      blockchain: ethereum

cluster:
  upstreams:
    - id: quicknode-mainnet
      chain: ethereum
      disable-validation: true
      connection:
        ethereum:
          rpc:
            url: "<functioning_quicknode_endpoint>"
          ws:
            url: "<functioning_quicknode_endpoint>"

Despite upstreams providing responses for eth_getBlockByNumber (I verified by dumping upstream responses in WsConnectionImpl) dshackle returns an error to downstream clients. As an example reproduction using cast:

$ cast rpc eth_getBlockByNumber 0x160d0c4 true
Error: server returned an error response: error code -32002: No response or no available upstream for eth_getBlockByNumber

Sent against a known-good upstream directly, that produces the expected response.

[splix]

Does it say in the console anything about the upstream? it's likely "no available upstream" but it should say in the logs if it cannot connect, etc

[rgelinas-polygon]

Does it say in the console anything about the upstream? it's likely "no available upstream" but it should say in the logs if it cannot connect, etc

No, the logs reflect the upstreams being available, and other head-verified methods work fine when sent at the exact same time (eth_getTransactionReceipt for example). I disabled validation in the reproduction config to ensure there wasn't anything else contributing to it.

As a sanity check, I just ran the 0.15 docker image with this config using the public DRPC Mainnet endpoints to see if it was coincidentally isolated to the two providers we use:

version: v1
port: 2449

proxy:
  host: 0.0.0.0
  port: 8545
  routes:
    - id: ethereum
      blockchain: ethereum

cluster:
  upstreams:
    - id: drpc-mainnet
      chain: ethereum
      disable-validation: true
      connection:
        ethereum:
          rpc:
            url: "https://eth.drpc.org"
          ws:
            url: "wss://eth.drpc.org"

But the same cast fails, again when sent to the DRPC endpoints directly is fine.

$ cast rpc eth_getBlockByNumber 0x160d0c4 true
Error: server returned an error response: error code -32002: No response or no available upstream for eth_getBlockByNumber

[rgelinas-polygon]

@splix Any update on reproduction for this? We are able to reproduce with basically any provider, public or private, and it renders dshackle functionally useless for production use for us.

[splix]

Yes, I found the reason. It's a in a dependency which processes JSON, and I'm updating it right now. Apparently there was a suboptimal code and with the growth of the ethereum it became noticeable.

I'll finish the update today or tomorrow, but as a temp workaround I suggest to add more CPU for dshackle.

[rgelinas-polygon]

Do you have a suggested target you've observed it working correctly at? We've bumped our instances to 2 full cores each and still get a timeout in that query

[splix]

I guess this would not be enough. Anyway, I have a fix locally, it works. I didn't make a release yet just to make sure nothing else is broken. Will make one in a day or two.

If it helps, you can send false and make separate requests per transaction.

[splix]

@rgelinas-polygon Please try v0.16