Listening Addresses on an AWS EC2 don't include public IPs

[hayotensor]

Summary

When using AWS (Amazon Web Services) EC2 remote servers, the public IP doesn't make it into the listening addresses, even when manually included in the list. Therefor, peers will broadcast their private and local IPs to other peers.

Expected behavior

Public IP should be one of the listening addresses and shared to other peers.

Actual behavior

What we're testing is using DHT, random walk, and gossip. But I believe the issue stems from the host layer.

For example (Identify demo):

This might not be relevant to the claims, but shows that connecting via public IP isn't the issue, it's the broadcasting that is. You will see that the listening_addr doesn't include the public IP on the messaging.

When starting a server peer_a:

First host listening (using length-prefixed format).
Listener ready, listening on:

/ip4/172.31.36.5/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4
/ip4/127.0.0.1/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4
/ip6/::1/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4

Run this from the same folder in another console:

identify-demo -d /ip4/172.31.36.5/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4

Waiting for incoming identify request...

These are the logs ^, but the public IP is not in the listening address list.

From peer_b (connecting via peer_a's public IP):

dialer (host_b) listening on /ip4/172.31.42.167/tcp/31330/p2p/12D3KooWQX2bjnjTuuEZBFmGNsuVds3sBSTcYK3uykJDUn7cLSu1
Second host connecting to peer: 12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4
Starting identify protocol...
Identify response:
  Public Key (Base64): CAESIGt1jMZBk6omhEirTLi6V04H2s+cRRUjjaiu3cQP7Db9
  Listen Addresses: ['/ip4/172.31.36.5/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4', '/ip4/127.0.0.1/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4', '/ip6/::1/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4']
  Protocols: ['/ipfs/id/1.0.0', '/ipfs/ping/1.0.0']
  Observed Address: ['/ip4/18.1xx.2xx.1xx/tcp/53298']
  Protocol Version: ipfs/0.1.0
  Agent Version: py-libp2p/0.6.0

=== Envelope ===
Payload Type: b'\x81\x06'
Signature: cd96b0ef1860273149e8c220b62e436ce0463bc9093b190d5ae3be8afd3492f4ef8f928ef3c62736a8b618f9fc8a1e63f7b335f9bf91113b2de11f7a4e8fcd00 (64 bytes)
Raw Payload: 0a260024080112206b758cc64193aa268448ab4cb8ba574e07dacf9c4515238da8aeddc40fec36fd10e89ebbf5df8291cc181a330a3104ac1f2405067a62a503260024080112206b758cc64193aa268448ab4cb8ba574e07dacf9c4515238da8aeddc40fec36fd1a330a31047f000001067a62a503260024080112206b758cc64193aa268448ab4cb8ba574e07dacf9c4515238da8aeddc40fec36fd1a3f0a3d2900000000000000000000000000000001067a62a503260024080112206b758cc64193aa268448ab4cb8ba574e07dacf9c4515238da8aeddc40fec36fd (221 bytes)

=== Parsed PeerRecord ===
PeerRecord(
  peer_id=12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4,
  multiaddrs=['/ip4/172.31.36.5/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4', '/ip4/127.0.0.1/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4', '/ip6/::1/tcp/31330/p2p/12D3KooWH3qkhNpEx3gr4GNNovqyqipEWyUfaNXtNGvSz3fMVBn4'],
  seq=1772241314628161384
)

peer_a then logs:

🔗 Received identify request from peer: 12D3KooWQX2bjnjTuuEZBFmGNsuVds3sBSTcYK3uykJDUn7cLSu1
   Remote address: /ip4/18.1xx.2xx.1xx/tcp/53298/p2p/12D3KooWQX2bjnjTuuEZBFmGNsuVds3sBSTcYK3uykJDUn7cLSu1
✅ Successfully processed identify request from 12D3KooWQX2bjnjTuuEZBFmGNsuVds3sBSTcYK3uykJDUn7cLSu1

Echo and ping examples both do work. So the issue is likely how a peer stores/chooses/broadcasts its listening addr's.

Main Issue

It seems like the main issue is the listening addresses and how they are broadcasted and identified internally. For example, we had a handful of peers connected gossiping to one another (using Kad-Dht, random walk). But the only reason this swarm worked was because there was one non-AWS server that was connecting everyone together.

This is their logs from a second non-AWS sever (connected to 1 AWS and 1 non-AWS):

(peer_id: [multiaddresses])

'12D3KooWKv3zskz6Eg4eMEi5D1xNBasmk1jQosznknw6et6wFYnz [/ip4/THEIR PUBLIC IP/tcp/31330/p2p/12D3KooWKv3zskz6Eg4eMEi5D1xNBasmk1jQosznknw6et6wFYnz, /ip4/127.0.0.1/tcp/31330/p2p/12D3KooWKv3zskz6Eg4eMEi5D1xNBasmk1jQosznknw6et6wFYnz]',

'12D3KooWSukUTaByb8ctEkgkb9Py4TssVCwwPncvnLGxmRu7CXhD [/ip4/127.0.0.1/tcp/31332/p2p/12D3KooWSukUTaByb8ctEkgkb9Py4TssVCwwPncvnLGxmRu7CXhD, /ip6/::1/tcp/31332/p2p/12D3KooWSukUTaByb8ctEkgkb9Py4TssVCwwPncvnLGxmRu7CXhD, /ip4/172.31.42.167/tcp/31332/p2p/12D3KooWSukUTaByb8ctEkgkb9Py4TssVCwwPncvnLGxmRu7CXhD]

As you can see, only one (non AWS server) has their public IP shown to everyone, the rest are receiving private and local IPs from peers broadcasting them.

Once they leave (the non AWS server), everything starts to fall apart and handshake errors begin.

For example, when servers try to connect (random walk):

Failed to open TCP stream: all attempts to connect to 172.31.42.167:31332 failed
Failed to open TCP stream: all attempts to connect to 172.31.42.167:31331 failed

Relevant log output

Possible Solution

No response

Environment

/usr/bin/python3: No module named eth_utils

Would you like to work on fixing this bug ?

Maybe

[hayotensor]

In TCPListener:

async def serve_tcp(
    handler: Callable[[trio.SocketStream], Awaitable[None]],
    port: int,
    host: str,
    task_status: TaskStatus[Sequence[trio.SocketListener]],
) -> None:
    """Just a proxy function to add logging here."""
    logger.debug("serve_tcp %s %s", host, port)
    await trio.serve_tcp(handler, port, host=host, task_status=task_status)

This will fail when sending through a public IP on the AWS server.

This fails because in open_tcp_listeners in trio I get [Errno 99] Cannot assign requested address when attempting to use the public IP (adding logs to trio).

This makes sense on why it won't acknowledge the public IP, but I think allowing the peer to broadcast the IP they enter should resolve this. Then other peers will assume that IP is correct until it fails on their end

[hayotensor]

Seems this issue was brought up on js:

libp2p/js-libp2p#230

[hayotensor]

#805