Discuss upload process to Piri and limitations of CommP hash function throughput.

[frrist]

The CommP hash function used during data upload: https://github.com/storacha/piri/blob/main/pkg/pdp/service/piece_upload.go#L55 has a throughput of ~1Gbps on a modern CPU. This causes the upload rate to be limited by the max throughput of the hash function, rather than the network throughput, which will generally be over 1Gpbs in practice. The purpose of this discussion is to find mitigations to this limit, and/or come up with an alternative upload process.

[alanshaw]

network throughput, which will generally be over 1Gpbs in practice

Is this right? Perhaps for server to server uploads but average home internet upload speeds are not nearly approaching 1Gbps.

[frrist]

True, though I was initially thinking about two scenarios where we'd see higher throughput:

1. Server-to-server transfers for replication between storage providers
2. Business customers who tend to have better connectivity, and who I assume will be the initial customers of the warm storage network.

[alanshaw]

To document what we spoke of before - I think we're currently tied to that hash function (in the sense that it needs to be calculated) due to needing everything to go into a Filecoin deal. However it could be calculated async.

I think the best we can do is to be agnostic to the CID we're accepting for uploads. I understand there is some work that would need to be done to allow that.