Allow multiple IPFS nodes to advertise their CIDs into IPNI through the same instance of index-provider

[ischasny]

index-provider allows IPFS nodes to advertise their CIDs to IPNI via Delegated Routing protocol. index-provider is supposed to be run as a sidecar to the IPFS node that it is publishing for. Such architecture is inefficient for IPFS clusters with more than one node as each node has to have its own instance of index-provider running. There are no limitations in the protocol itself that would prevent multiple IPFS nodes from sharing the same instance of index-provider.

As a part of this work:

• enable index-provider to support multiple nodes sharing a single instance via HTTP Delegated Routing;
• optimise index-provider in-memory data structure that is used to facilitate CID expiration. Right now the amount of RAM required is linearly correlated to the number of active CIDs which will be a bottleneck for larger deployments.

An alternative to that approach could be using Elastic IPFS as it supports direct publishing to IPNI. The downside however is that it has a complex deployment architecture and is tight to AWS only.

Related thread.