Trying to Dial Kubo Node with Web Transport

[TheGreatAlgo]

Recently upgraded a node to support web transport near end of april and I cannot dial via web transport. Was able to connect at first but seemingly unable to connect or only very limited successes at attempts now. Was attempting to dial /ip4/45.55.32.80/udp/4001/quic-v1/webtransport/certhash/uEiAM_1-R7yfSAtnm9g_fzzcIccsT1RzKPfXPr5lKfqmVVQ/certhash/uEiAcLZS4mabYQl9V1K0QjWsToU21DtsWarGTjZQzecHFmg/p2p/12D3KooWG7itEPAHut3xsVo7CwyD8sKeQXKgQizotNhPsToCssXQ

and get this error

[TheGreatAlgo]

Any updates on this? This is preventing a connection to our IPFS node that is supporting web transport and preventing us from implementing helia on browser side

[achingbrain]

It looks like the certhash(es) are invalid? There was a bug around this in certain versions of Kubo - are you using the latest version?

Otherwise, are you using the same certhashes as in April? They expire after a couple of weeks (limitation of the transport - see the certificates section of the spec) but the Kubo node should rotate them and report addresses with updated certhashes over time.

[TheGreatAlgo]

ahh so that version of Kubo wasn't rotating them as expected? We are using 0.19 i think. Seems like using 0.21 should resolve this?

[achingbrain]

Tediously the Kubo release notes don't say when the bug was fixed so I can't be certain but it's definitely worth updating.

The certhashes should change over time though - if you're using the exact same versions as in April they will definitely have expired by now.

[BigLep]

@TheGreatAlgo : did upgrading resolve the issue?

[TheGreatAlgo]

@BigLep Been a little swamped to try at the moment after upgrading. Hope to give it another look soon

[BigLep]

Transferring this issue from https://github.com/libp2p/js-libp2p-webtransport/issues/82 was requested by dClimate David on behalf of @TheGreatAlgo

[TheGreatAlgo]

After upgrading the node we still are having issues with certificate unknown. I am running the latest Helia and Libp2p versions as of yesterday but I suspect this is a kubo thing. How can we ensure the certificates are being rotated? does this mean /ip4/45.55.32.80/udp/4001/quic-v1/webtransport/certhash/uEiAM_1-R7yfSAtnm9g_fzzcIc... should change every two weeks? Do we have to just manually pay attention when they update and update on the frontend every single time it updates?

Also I tried https://github.com/libp2p/js-libp2p/tree/master/examples/libp2p-in-the-browser/webtransport/fetch-file-from-kubo to at least see if I could get the example working but connections were timing out with the example address.

We are running https://github.com/ipfs/kubo/releases/tag/v0.21.0 on the backend and trying to connect to it

[p-shahi]

This was discussed in today's js-libp2p maintainers call.

Do we have to just manually pay attention when they update and update on the frontend every single time it updates?

Right now the answer is sort of yes. If you've hardcoded the whole multiaddr into your application, then you'll have to pay attention and update manually since the addr will eventually be invalid.
Ideally, instead of hardcoding the multiaddr, it's better to encode the peerid of the nodes to dial and do a peer lookup via the dht.
However, browser dht queries are challenging atm because the whole query will stop when we encounter nodes that don’t have WebTransport addresses.
So the solution of encoding peerid & doing a lookup is limited until more of network supports WebTransport.
What percentage of the network supports WebTransport is something we're looking to get insight into. The ProbeLab team does network measurements but we don't have concrete numbers on WebTransport atm.

[aschmahmann]

Do we have to just manually pay attention when they update and update on the frontend every single time it updates?

Maybe a more flexible answer then the one above is that browser webtransport doesn't want the certificate hashes to be fixed and yet you want fixed addresses so something's got to give. Some examples:

• Use your application as the mutability point and update it every two weeks
• Use the peerID -> multiaddr mapping as the mutability point
  • Find some system to do peer routing for you in a way the browser can access (DHT + Delegated routing, IPNI, etc.)
• Use a non-peerID -> multiaddr mapping as the mutability point
  • For example, DNSLink + DoH resolution in the browser

However, browser dht queries are challenging atm because the whole query will stop when we encounter nodes that don’t have WebTransport addresses.

Yeah, but then just ask someone else to do it for you 😄. Sure, it'd be nice to do it yourself but for now I don't see why not. There's IPIP ipfs/specs#417 for adding peer routing to the /routing/v1 API that should be better supported going forward, but if you needed something sooner you should be able to use https://github.com/libp2p/js-libp2p-delegated-peer-routing which has been around for a while.

To me this ^ seems like the easiest way to handle the certificate hash updates today. Although it comes with the caveat that you should make sure you've got a plan for swapping out even the peerIDs in the event say a private key is leaked.

[TheGreatAlgo]

Thanks for you help on this! helped clarify a little more. I attempted to use the libp2p delegated peer routing with my HeliaClient and ran into issue with connecting and getting data
So I applied the peer routers as suggested as below

Then I find the peer, and then connect to it. (peerid: 12D3KooWG7itEPAHut3xsVo7CwyD8sKeQXKgQizotNhPsToCssXQ)

And it shows a connection made.

So that part is great so far

However I keep getting webtransport issues on connection. It appears to try and connect but errors on the ipv6 address. CID that exists on this node is bafzbeib46bee2znfxnsl6wdnfey6b52i6527s3iyaj73j3eufsfnirqeq4

So switching to a different node of ours on 0.22 (different from the 0.21 tried before) and we manage to get connection! (peerid : 12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy)

but when fetching for different data that exists on this node, nothing happens. No error, no timeouts.

CID being fetched bafyreicvczjixk5g7gs4rdd3sjvt7wab7mqoqcj6mqkh3fq7rnpoyc5ati

Are there two different communication protocols (one for peers and one for content routing)? If we want to fetch data from a connected peer does it try to send out a request via the content routing?

When turning on the contentRouter via cid.contact our node is recognized and it tries to dial the data via webtransport but fails. I turned it off when we had success dialing because my thoughts is that we shouldn't need to rely on a external service if our node has the data and we are peered to it

I thought that if we had a successful peer connection we shouldn't have certhash issues or web transport issues?

Thanks once again in helping with this!

Current config

    const libp2p = await createLibp2p({
        // transports allow us to dial peers that support certain types of addresses
        transports: [webTransport()],
        connectionEncryption: [noise()],
        streamMuxers: [mplex()], // streamMuxers: [yamux(), mplex()],
        peerDiscovery: [
            bootstrap({
                list: [
                    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
                    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
                    "/dnsaddr/bootstrap.libp2p.io/p2p/QmZa1sAxajnQjVM8WjWXoMbmPd7NsWhfKsPkErzpm9wGkp",
                    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
                    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
                    // "/ip4/45.55.32.80/udp/4001/quic/p2p/12D3KooWG7itEPAHut3xsVo7CwyD8sKeQXKgQizotNhPsToCssXQ",
                    // "/ip4/45.55.32.80/udp/4001/quic-v1/webtransport/certhash/uEiAM_1-R7yfSAtnm9g_fzzcIccsT1RzKPfXPr5lKfqmVVQ/certhash/uEiAcLZS4mabYQl9V1K0QjWsToU21DtsWarGTjZQzecHFmg/p2p/12D3KooWG7itEPAHut3xsVo7CwyD8sKeQXKgQizotNhPsToCssXQ",
                ],
            }),
        ],
        contentRouters: [ipniContentRouting("https://cid.contact")],
        // services: {
        //     // the identify service is used by the DHT and the circuit relay transport
        //     // to find peers that support the relevant protocols
        //     identify: identifyService(),
        //     // the DHT is used to find circuit relay servers we can reserve a slot on
        //     dht: kadDHT({
        //         clientMode: true,
        //         validators: {
        //             ipns: ipnsValidator,
        //         },
        //         selectors: {
        //             ipns: ipnsSelector,
        //         },
        //     }),
        // },
        peerRouters: [delegatedPeerRouting(client)],
    });

config for the better working node

	"ID": "12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
	"PublicKey": "CAESIFmQ1jfIE0EMrmzNOIwl9Xbvoml2JYt7ZpLcXMSXY3ai",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip4/127.0.0.1/udp/4001/quic-v1/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiCEWEj4tRXTc07CTSG-85iceKDtGkuotwBEST5vbHrT_g/certhash/uEiD9zjXupPyyym8K9ddVdL1c-JjmZmgp26VBGP29uk2t1Q/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip4/127.0.0.1/udp/4001/quic/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip4/167.71.180.28/tcp/4001/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip4/167.71.180.28/udp/4001/quic-v1/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip4/167.71.180.28/udp/4001/quic-v1/webtransport/certhash/uEiCEWEj4tRXTc07CTSG-85iceKDtGkuotwBEST5vbHrT_g/certhash/uEiD9zjXupPyyym8K9ddVdL1c-JjmZmgp26VBGP29uk2t1Q/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip4/167.71.180.28/udp/4001/quic/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip6/::1/tcp/4001/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip6/::1/udp/4001/quic-v1/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiCEWEj4tRXTc07CTSG-85iceKDtGkuotwBEST5vbHrT_g/certhash/uEiD9zjXupPyyym8K9ddVdL1c-JjmZmgp26VBGP29uk2t1Q/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy",
		"/ip6/::1/udp/4001/quic/p2p/12D3KooWFqzYkjofVzvo7MdYy4h3cZwixJC1f6NFWGYPELU7yBNy"
	],
	"AgentVersion": "kubo/0.22.0/",
	"Protocols": [
		"/floodsub/1.0.0",
		"/ipfs/bitswap",
		"/ipfs/bitswap/1.0.0",
		"/ipfs/bitswap/1.1.0",
		"/ipfs/bitswap/1.2.0",
		"/ipfs/id/1.0.0",
		"/ipfs/id/push/1.0.0",
		"/ipfs/kad/1.0.0",
		"/ipfs/ping/1.0.0",
		"/libp2p/autonat/1.0.0",
		"/libp2p/circuit/relay/0.2.0/hop",
		"/libp2p/circuit/relay/0.2.0/stop",
		"/libp2p/dcutr",
		"/libp2p/fetch/0.0.1",
		"/meshsub/1.0.0",
		"/meshsub/1.1.0",
		"/x/"
	]
}```

[p-shahi]

@TheGreatAlgo can you try on the latest version of js-libp2p libp2p-v0.46.5 and webtransport-v3.0.5? The reason I ask this is because there are fixes like reducing dial activity in browsers and a refactor WebTransport session closures that I'd like to see if it improves the following error you saw

Do you also see the following #1896 w/ the above error

#1969 helps with that (If you're trying to debug in the browser, and you leave it running for too long then you hit that 64 WebTransport session limit and no matter what you do, even if you got a new address youd' be like to dial, it still may not work because youv'e hit this other bug. So keeping the num of WebTransport sessions to minimum will make debugging this easier.)
Thanks for your patience

[TheGreatAlgo]

@p-shahi I tried upgrading to the latest but now I get this error. Seemingly finding the peer no longer works

swarm connecting the peer does work

I may have gotten the error at one point with the session limit