Skip to content

share/getters: Consider removing / not using IPLDGetter for requests of historic data #3653

Open
@renaynay

Description

@renaynay

Related to #3630

If we pursue the solution suggested in the above linked issue:

For the described data existence pre-checks, FN/BNs always report like they have the data, without doing lookups into inverted index.

This means that FNs + BNs of either type (archival or pruned) would always respond that they have the data regardless of whether they do or not. Archival nodes absolutely will have the data as they sync and retain all blocks, but pruned nodes will not have any data that is older than historic, so if a request for shares / EDS that is historic falls back to IPLDGetter, pruned nodes in the network will answer that they have the data even if they do not, so it would be misleading and could result in more roundtrips for the LN (e.g. requesting CID from a pruned node for historic shares by namespace when that data has already been deleted from the node, so LN has to go and ask another node).

The benefits of the solution quoted above are to reduce DAS' impact on the CPU of BNs + FNs in the network. DAS only occurs in the sampling window which is mostly what IPLDGetter is needed for (specifically, the GetShare request).

So when it comes to shares by namespace or EDS requests, the ShrexGetter should be sufficient to handle that as it's lighter-weight and the happy path. This would require some improvements in the robustness of the peer manager for shrex, but I think it's possible and can be done fairly easily (needs to be tested though of course).

FWIW, I would even consider removing the IPLDGetter as a fallback for requests for shares by namespace and EDS within the sampling window as well, but I think turning IPLDGetter off incrementally (for historic requests as they're less frequent and lower volume) is a first step approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ipldIPLD pluginarea:sharesShares and samplesshrexFor issues and PRs related to shrex protocol

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions