feat!: add ipns reproviding/republishing

[2color]

What

This PR adds the functionality necessary for a Helia node to reprovide records it created before expiry.

Fixes #367

Related: ipshipyard/roadmaps#1

Why

Currently, the ipns.republish method which runs the republishing loop does not actually handle republishing.

How

• Add the the ability to iterate on records in the data store.

Open question about storing keys

• We don't store the private key when publish is called. This means that we can only reprovide valid IPNS records, but with the current defaults (48 hour lifetime and 23 hour republish interval), that means one round of republishing, after which all stored IPNS records become invalid.
  -This begs the question of whether we should store the keys so that we can create new valid records for stored IPNS names.
• After looking at how we manage the self key for the libp2p peer we create, it seems like we probably want to add the keychain. The question then is whether to add the keychain to the Helia interface or just the IPNS module.
• Adding a keychain would allow us to iterate on keys rather than IPNS names when republishing. Though I'm not sure if there's any obvious benefit to this.

helia/packages/ipns/src/index.ts

Line 450 in 1e20e38

publish(key: PrivateKey, value: CID | PublicKey | MultihashDigest<0x00 | 0x12> | string, options?: PublishOptions): Promise<IPNSRecord>

TODO

• Add protobuf for serialising IPNS record metadata in the datastore
• Configure @helia/http to use a libp2p instance that's only configured with HTTP routers to ensure we always have the js-libp2p keychain here
• republish multiple records concurrently (see example)

[2color]

Following a discussion offline with @achingbrain, we need to move key management internally to implement reproviding.

Key management is typically (and store encrypted at rest) handled by the js-libp2p keychain, however we don't have libp2p in @helia/http (which can also be used for @helia/ipns with delegated routers). This means that we probably need a dedicated keychain instance in the IPNS class, so that we don't depend on the libp2p instance.

[achingbrain]

This means that we probably need a dedicated keychain instance in the IPNS class

@helia/http could also use a libp2p instance that's only configured with HTTP routers? Then you don't need to handle things differently at this level.

It'd also be good to see just how lightweight we can make libp2p for this sort of use-case.

[2color]

@helia/http could also use a libp2p instance that's only configured with HTTP routers? Then you don't need to handle things differently at this level.

Yeah we could try that first and see how it feels. It may increase the bundle size.

I also vaguely remember there being a bug with libp2p whereby instantiating it without the DHT, causes an infinite random walk loop with the defaults, as it's looking for a circuit relay reservation by default.

Would this also allow Helia to make use of the libp2p over http stuff you've be working on?

[achingbrain]

It may increase the bundle size.

Looking at the breakdown, the usual suspects are already there - @libp2p/crypto, @noble/hashes, etc so it might not increase it by that much.

I also vaguely remember there being a bug with libp2p whereby instantiating it without the DHT, causes an infinite random walk loop with the defaults, as it's looking for a circuit relay reservation by default.

I'd imagine this configuration wouldn't configure a relay listener so it wouldn't try to find a relay?

We need this landing, really - ipfs/specs#476

Would this also allow Helia to make use of the libp2p over http stuff you've be working on?

Yes, though you'd need transports etc in order to use libp2p for the transport layer.

[2color]

As discussed in #807, the next steps here:

• Take named keys (string), persisted in the keychain using a separate namespace) as input to publish. Keys will be generated internally.
• Republish will iterate over records in datastore
• Add unpublish(‘keyname’) for deleting records
• Add the public key to the return type of publish, because we can't derive the name when the key is managed by the keychain.

[2color]

I have an open question: we currently store the marshalled DHT record for IPNS records in the datastore.

As part of republishing, we need to also persist some additional local metadata associated with the IPNS record, like the keychain’s keyname (the key name string that the user passes) associated with the record and potentially also the lifetime. Both of these are needed for republishing.

The two ways I’ve explored storing this metadata are:

• Take the current libp2p DHT record we store and make it part of an envelope which also stores some local metadata (lifetime and keychain’s keyname)
• Store the local metadata using a separate key in the datastore specific for ipns metadata.

The latter approach seems a bit easier and less likely to have downstream effects or unintended consequences (like polluting /dht/record/ namespaced keys in the datastore with other data that isn’t part of a DHT record)

Since the datastore expect binary values, what are the conventions around storing such data in binary format? Should I use Protobuf for this or can I use cbor (which we already depend on)

For the sake of the example, let's assume I have the following struct that I want to persiste:

export interface IPNSRecordMetadata {
  keyName: string
  lifetime: number
}

@achingbrain

[achingbrain]

Most of the datastore values are protobuf encoded - it's nice because it has predictable decoding behaviour and uses schemas which will protect us from ourselves.

I think storing extra metadata under a separate key prefix is the way to go, the /dht/record/ prefix is populated and read by at least kad-dht and ipns-pubsub so we can't change the data structures stored there, not without a very tedious upgrade process.