Pubsub flood due to same message propagated multiple times

[stbrody]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.tech

Version

it's a spread.  Different nodes on the network are running different versions.  We cannot control when all node operators upgrade their nodes.

Config

n/a

Description

Incident report from 3Box Labs (Ceramic) Team

Incident summary

The Ceramic pubsub topic has been experiencing a flood of pubsub messages beyond our usual load for the last several days now. We log every pubsub message we receive on the nodes that we run, and running analysis on those logs using LogInsights shows us that we are receiving messages with the exact same `seqno` multiple times - one message can show up upwards of 15 times in an hour. During normal operation we do not åsee this issue with seqnos showing up multiple times. This dramatic increase in the number of messages that need processing is causing excess load on our nodes that is causing major performance problems, even with as much caching and de-duplication as we can do at our layer.

Evidence of the issue

Graph of our incoming pubsub message activity showing how the number of messages spiked way up a few days ago. The rate before 2/20 was our normal, expected amount of traffic:

AWS LogInsights Query demonstrating how the majority of this increased traffic is due to seeing the same message (with the same seqno) re-delivered multiple times. Before the spike we never saw a msg_count greater than 2.

Steps to reproduce

Connect to the gossipsub topic `/ceramic/mainnet`. Observe the messages that come in, keep track of the number of times you see a message with each `seqno`. You'll see that over the span of an hour you see the same message with the same `seqno` delivered multiple times

Historical context

We have seen this happen before, in fact it's happened to us several times over the last year, and we've reported it to PL multiple times. You can see our original report here (at the time we were still using js-ipfs): https://github.com/libp2p/js-libp2p/issues/1043. When this happened again after we had migrated to go-ipfs, we reported it again, this time on slack: https://filecoinproject.slack.com/archives/C025ZN5LNV8/p1661459082059149?thread_ts=1661459082.059149&cid=C025ZN5LNV8

We have since discovered a bug in how go-libp2p-pubsub maintained the seenMessage cache and worked to get a fix into kubo 0.18.1: libp2p/go-libp2p-pubsub#502

We have updated our nodes to 0.18.1, but of course we have no direct control over what versions of ipfs/kubo the rest of the nodes on the Ceramic network are running, so even if the above bugfix would resolve the issue if every single node on the network were to upgrade to it, we have no real way to enforce that and no idea how long it will be (if ever) before there are no older ipfs nodes participating in our pubsub topic. Not to mention the possibility of a malicious node connecting to our pubsub topic and publishing a large volume of bogus messages (or re-broadcasting valid messages). So no matter what, we need a way to respond to incidents like this that goes beyond "get your users to upgrade to the newest kubo and pray that that makes the problem go away", which has been what we've been told every time we're reported this issue so far.

Our request from Protocol Labs

This is an extremely severe incident that has affected us multiple times over the last year. It strikes without warning and leaves our network crippled. Every previous time this happened it cleared up on its own within a day or so, but this one has been going on for 5 days now without letting up. We need some way to respond to incidents like this, and to potential malicious attacks in the future where someone intentionally floods our network with pubsub traffic.

So our questions for PL are:

• What short term options can we take on the nodes that we operate, or that we can tell our community of node operators to take, to resolve this immediate issue that is currently affecting our production network?
• Do you have any tools for inspecting the p2p network that would let us identify which node(s) are the source of the issue? If we knew that the issue was because of one problematic node that was, for instance, running a very old version of ipfs or running on very underpowered hardware, we could potentially reach out to them directly and get them to upgrade or take down their node. Or perhaps we could tell existing node operators to block connections from that problematic peer.
• What is the recommended way in general to respond to nodes that (either intentionally through malice or accidentally through a bug) spam a pubsub topic with bogus or re-broadcast messages?
• What additional steps can we take going forward to prepare our network to be more resilient to issues like this in the future?

Thank you for your time and attention to this important issue!

-Spencer, Ceramic Protocol Engineer

[stbrody]

This is a companion report to a report that was already filed against go-libp2p-pubsub here: libp2p/go-libp2p-pubsub#524

The main advice we received from @vyzo working on go-libp2p-pubsub was to utilize libp2p validators for our incoming pubsub messages. Is that something that kubo exposes currently?

Also, even if the libp2p validators are exposed via kubo APIs, it will take time to set up the right ones and get our community of node operators to upgrade to utilize them. In the meantime our network is still in a bad state right now so I'm very interested in any ideas for what we can do in the short term to reach a stable state again.

Thank you!

[vyzo]

A possible solution that can solve your current predicament is to provide a default validator in ipfs that uses the message envelope seqno as the nonce.

This works for all peers/topics, and can reject/ignore old messages without needing to look into the message payload itself.

This can be implemented quite easily and provide the bandaid needed to stop the bleeding and buy time to design the proper abstractions for a validator api.

[vyzo]

A bit more on the mechanics of this.

The proposed validator relies on message signing and use of the seqno, which are default features used by ipfs.
This means that every message has an origin peer, a signature (validated by pubsub lib itself) and a monotonically incrementing seqno in the envelope (constructed by the pubsub lib, initialized with the current unix timestamp in nanos and incremented thereafter).

Thus we can use this seqno as a per peer nonce and extinguish the kind of floods that is being observed here.
Note that the per peer nonce shouldnt in generally be kept in memory (modulo caching) to avoid creating a dos vector.

The validator can be registered by every topic thats is joined, or we can add an api in pubsub for default validators for all topics.

[smrz2001]

Thanks for the ideas, @vyzo! This is very helpful.

So we could expose RegisterTopicValidator and UnregisterTopicValidator like we did with WithSeenMessagesTTL (but as a function, of course)?

[vyzo]

I plan on working on this tmrw, as it is holiday here today.
Unless you want to pick it up today of course.

I will add a new api to pubsub for default validators, and write a default validator that behaves as described in kubo.
No need to expose anything new.

[smrz2001]

I plan on working on this tmrw, as it is holiday here today. Unless you want to pick it up today of course.

I will add a new api to pubsub for default validators, and write a default validator that behaves as described in kubo. No need to expose anything new.

Would really appreciate the help, @vyzo 🙏🏼 Our whole team is all-hands-on-deck debugging and testing in preparation for a major product launch at Eth Denver over the next couple of days (hence our concern about Pubsub flooding at this time).

[p-shahi]

The current plan: kubo maintainers will try to get out a patch release that includes Vyzo's proposed solution this week.

[stbrody]

Really appreciate the attention here @vyzo and @p-shahi!!!

[BigLep]

So there are a couple of parties involved here:

1. vyzo@ for the go-libp2p-pubsub work
2. ipfs/kubo-maintainers for doing a Kubo release.

This is on the Kubo maintainers' 2023-02-28 standup agenda to discuss to figure out what we can do here. We understand this is time sensitive for you all. We'll update the issue after standup (by 19UTC on 2023-02-28).