Questions on Thanos architecture based on alert that we got

[TessaIO]

Hi,
A few days ago, we had a networking alert related to the Packet Per Second exceeded on a specific node. For context, we're using AWS as our cloud provider and Thanos version v0.39.2.

Upon checking the alert that happened on a couple of nodes, we found that the store gateway is using a lot of the networking bandwidth, and you can see that the number of packets is kinda big (at least bigger than how it was historically).

After that, we exec'ed into the node to see which connection with the store gateway has the most of the packets (using conntrack), and we found this:

You can check the dst has 10.100.184.213 as IP, which is the same IP as one of our Prometheus instances (which also has the sidecar)

However, from my understanding, the store gateway and the sidecar shouldn't technically communicate with each other. In fact, the querier is the mediator in Thanos and is the one who communicates with other components, and there should be no direct communication between the store gateway and the querier.

Am I missing something here?

[GiedriusS]

This is interesting. Store and Sidecar shouldn't have any connections between themselves except for one common component - object storage. I'm not sure how this could happen based on your information 🤔