IPFS not announcing CIDs on indirect peers

[iumarabbas]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

ipfs version 0.25.0

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/__.___.100.178/tcp/4001/p2p/12D3KooJLq6rP69XfYWJkkKN29fsvhX5z5hZe5SYUai6guhMhLxa"
  ],
  "DHT": {
    "Server": true
  },
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "12D3KooWAAUpb6sUWUkdGngrMTb4QgjokRVxWghnG5FeG7QqJjMz"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null,
    "Type": "dhtserver"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {}
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

I have four nodes NodeB, NodeA, Node0, and Node3
NodeB is peer of NodeA,
NodeA is peer of NodeB and Node0
Node0 is a peer of NodeA and Node3
Node3 is a peer of Node3

When I upload file on NodeB it is accessible on NodeA but not on Node0 and Node3 as these nodes are not directly added as peer but have same swarm key. what might be the reason?
I have configured all nodes as dhtserver and turned the dht server check to “active” but its not working.

When we get content from direct peer node, content is retrieved but for indirect peer the content is not accessible directly.

[Jorropo]

@iumarabbas can you show please ipfs stats dht on all nodes of the network ?

[iumarabbas]

@Jorropo - Please see the response for both ipfs stats dht and ipfs swarm peers
As we have done the following

1. Updated config with Routing.Type: dhtserver and dhtclient
2. Updated config with DHT.Server: true
3. Updated config with Tested with AccelaratedDHTClient: true
4. manually announced the CID using ipfs dht provide then tried to fetch it using ipfs dht findprovs but no peer was returned.
5. Tried annoucing using ipfs dht provide -r
6. Tried with ipfs bitswap reprovide (command did not work)

All the nodes are connected through public IP and accessible publicly and the swarm key is same.

[ NodeB <--> NodeA ] <==>[ Node0 <--> Node3 ]

The nodes are hosted on azure cloud services:

NodeB and NodeA are in virtual network 1.
Node0 and Node3 are in the virual network 2.

NodeB:

ipfs stats dht

DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version
Error: no LAN client found

ipfs swarm peers
/ip4/..99.165/tcp/4001/p2p/12D3KooWSsdB2mJyQkq*****

NodeA

root@nodeA:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version

Error: no LAN client found
root@nodeA:/home/ssiadmin# ipfs swarm peers
/ip4/..99.199/tcp/4001/p2p/12D3KooWDpY9uHAEDxd*****
root@nodeA:/home/ssiadmin#

Node0

root@node0:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version

Error: no LAN client found
root@node0:/home/ssiadmin# ipfs swarm peers
/ip4/..98.222/tcp/4001/p2p/12D3KooWLPM8shRFvonSnn******
/ip4/..53.99/tcp/4001/p2p/12D3KooWNy35zuZFaqu3QXa******
root@node0:/home/ssiadmin#

Node3

root@node3:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version

Error: no LAN client found
root@node3:/home/ssiadmin# ipfs swarm peers
/ip4/..98.178/tcp/4001/p2p/12D3KooWETehntUtx6i63HBvrme*****
root@node3:/home/ssiadmin#

[Jorropo]

@iumarabbas the issue here is that the accelerated dht client is constantly failing to crawl your network, so it has no peer to issue queries to. It could also be that it hasn't finished yet however I would find that weird given your network is so slow it should crawl it in no times.
Can you also show ipfs stats dht wanserver on all nodes please, trying to see if this is a dht server or accelereated dht client issue ?

[iumarabbas]

@Jorropo - As we are using Azure Cloud network so the network bandwidth/speed should not have issue here appearantly.

We are getting the following response to the above query: ipfs stats dht wanserver on all nodes,

root@node3:/home/ssiadmin# ipfs stats dht wanserver
DHT wanserver (0 peers):

[Jorropo]

Next steps would be to capture jaeger traces of the DHT to see what it is doing.
I don't have anything in mind and I think the thing I want to see (the server) isn't traced, would need to write some more code.

[mzuhaq7]

@Jorropo Judging from the documentation https://github.com/ipfs/kubo/blob/master/docs/experimental-features.md#private-networks . IPFS private network is an experimental feature. I suppose it's still not stable enough to be used for heavy load production environments

P.S: Same project as @iumarabbas

[Jorropo]

@mzuhaq7 it is the inverse problem, we struggle on small networks, if your network is bigger (and is mainly routed over WAN) it should work like the amino network.

[mzuhaq7]

@Jorropo I see. I tested it out on a network of 6 nodes but files were not being fetched from indirect nodes. I doubt if i increase network size more that'll help.

Is this a known issue in private IPFS? This seems to be a very basic functionality that should be working fine. We followed the simple steps to setup a private IPFS network here https://medium.com/@s_van_laar/deploy-a-private-ipfs-network-on-ubuntu-in-5-steps-5aad95f7261b

I see similar issues reported with no solution
https://discuss.ipfs.tech/t/cant-fetch-files-from-other-nodes/10893