Too many data sets make many commands unusable

[beck-8]

0xa108be4331296ec8b8c47c2cd2fbfddf06e27523

This wallet was used by me before for stress testing. It has a large number of data sets. I'm not sure exactly how much right now.
I obtained it through the contract method clientDataSets and now there are 4214

Now it seems that this method getClientDataSets cannot hold so many entries?

➜  /tmp filecoin-pin add out2287372121
┌  Filecoin Pin Add
│
◇  ✓ File validated (102.0 B)
│
◇  ✓ Connected to Filecoin - Calibration testnet
│
◇  ✓ Minimum payment setup verified (~0.066 USDFC required)
│
◇  ✓ File packed with root CID: bafybeihpws5pknsqfpfmqufxchoww4m5ry5jidy6s3zu22ac2v63tkediq
│
◇  ✓ IPFS content loaded (297.0 B)
│
◇  ✓ Payment capacity verified for 297.0 B
│
◇  ✗ Add failed: The contract function "getClientDataSets" reverted with the following reason:
message execution failed (exit=[33], revert reason=[none], vm error=[message failed with backtrace:
00: f0179532 (method 3844450837) -- contract reverted at 943 (33)
01: f0176641 (method 3844450837) -- out of gas (7)
02: f0176641 (method 6) -- out of gas (7)
 (RetCode=33)])

Contract Call:
  address:   0x53d235D474585EC102ccaB7e0cdcE951dD00f716
  function:  getClientDataSets(address client)
  args:                       (0xa108Be4331296Ec8b8C47c2Cd2FbfDDF06E27523)

Docs: https://viem.sh/docs/contract/readContract
Version: viem@2.47.4



➜  /tmp filecoin-pin data-set ls --all --rpc-url http://127.0.0.1:1234/rpc/v1
┌  Filecoin Onchain Cloud Data Sets
│
◇  ✗ Failed to list data sets
│
│
│  Error: Failed to get details for data set 11184: HTTP request failed.
│
│  URL: http://127.0.0.1:1234/rpc/v1
│  Request body: {"method":"eth_call","params":[{"data":"0xbdaac0560000000000000000000000000000000000000000000000000000000000002bb0","from":"0xa108Be4331296Ec8b8C47c2Cd2FbfDDF06E27523","to":"0x53d235D474585EC102ccaB7e0cdcE951dD00f716"},"latest"]}
│
│  Raw Call Arguments:
│    from:  0xa108Be4331296Ec8b8C47c2Cd2FbfDDF06E27523
│    to:    0x53d235D474585EC102ccaB7e0cdcE951dD00f716
│    data:  0xbdaac0560000000000000000000000000000000000000000000000000000000000002bb0
│
│  Contract Call:
│    address:   0x53d235D474585EC102ccaB7e0cdcE951dD00f716
│    function:  getDataSet(uint256 dataSetId)
│    args:                (11184)
│
│  Docs: https://viem.sh/docs/contract/readContract
│  Details: fetch failed
│  Version: viem@2.47.4
└  Listing failed


➜  /tmp filecoin-pin data-set show 11174
┌  Filecoin Onchain Cloud Data Set Details for #11174
│
◇  ✗ Failed to inspect data set
│
│
│  Error: Failed to get details for data set 2811: The request took too long to respond.
│
│  URL: wss://wss.calibration.node.glif.io/apigw/lotus/rpc/v1
│  Request body: {"method":"eth_call","params":[{"data":"0xbdaac0560000000000000000000000000000000000000000000000000000000000000afb","from":"0xa108Be4331296Ec8b8C47c2Cd2FbfDDF06E27523","to":"0x53d235D474585EC102ccaB7e0cdcE951dD00f716"},"latest"]}
│
│  Raw Call Arguments:
│    from:  0xa108Be4331296Ec8b8C47c2Cd2FbfDDF06E27523
│    to:    0x53d235D474585EC102ccaB7e0cdcE951dD00f716
│    data:  0xbdaac0560000000000000000000000000000000000000000000000000000000000000afb
│
│  Contract Call:
│    address:   0x53d235D474585EC102ccaB7e0cdcE951dD00f716
│    function:  getDataSet(uint256 dataSetId)
│    args:                (2811)
│
│  Docs: https://viem.sh/docs/contract/readContract
│  Details: The request timed out.
│  Version: viem@2.47.4
└  Inspection failed

[beck-8]

filecoin-project/curio#892

@BigLep We have previously tested a dataset with many pieces. I'm not sure where have we encountered the above situation before? Or has it been tested?

[BigLep]

Thanks for reporting @beck-8 ! I don't think this has been tested before.

@rvagg : can you please give a quick triage and provide suggested next steps?

[rvagg]

Going to need a contract upgrade to deal with this, or at least an FWSS view contract upgrade. We have a call, getClientDataSets that, on chain, iterates through the client's data sets one by one and loads the state to build a return type. This should be paginated to deal with the case of a client with this many data sets.

FilOzone/filecoin-services#443

[rvagg]

FilOzone/filecoin-services#444 but getting that out and deployed and then updated into synapse will take a bit. I'm not sure when our next deploy will be.