"Timed Out Fetching CID 'QmaPE7..................'" During Sync/Block Processing on Private DID ION Nodes

[Shariat-dotnet]

Subject: Issue: "Timed Out Fetching CID" During Sync/Block Processing on Private DID ION Nodes

Description:
I have deployed two DID ION mainnet nodes on different servers, both inside a private network. While I can successfully create and publish DIDs on these nodes, the nodes are unable to resolve any external DIDs.

Whenever I attempt to resolve external DIDs in any of those nodes, I consistently receive the following message:

DID not found

Upon reviewing the logs, I consistently encounter the following error during sync/block processing in ion-core:

2025-01-30 04:59:27 Fetched 3 unresolvable transactions to retry in 2 ms.
2025-01-30 04:59:27 Downloading core index file 'QmaPE7xsFQ8HxhfQyAFdf3nWk7AidH3F9nQ12j3up4RB2S', max file size limit 1000000 bytes...
2025-01-30 04:59:27 Downloading file 'QmaPE7xsFQ8HxhfQyAFdf3nWk7AidH3F9nQ12j3up4RB2S', max size limit 1000000...
2025-01-30 04:59:27 Downloading core index file 'QmQyrFpvpEjw558LP9tLARRgUgqMTJXLmZL8bsXyUcXHxP', max file size limit 1000000 bytes...
2025-01-30 04:59:27 Downloading file 'QmQyrFpvpEjw558LP9tLARRgUgqMTJXLmZL8bsXyUcXHxP', max size limit 1000000...
2025-01-30 04:59:27 Downloading core index file 'QmUvfGKksdK58ucaBT5UB599UfukGCSxzMdNakrF9cXsrG', max file size limit 1000000 bytes...
2025-01-30 04:59:27 Downloading file 'QmUvfGKksdK58ucaBT5UB599UfukGCSxzMdNakrF9cXsrG', max size limit 1000000...
2025-01-30 04:59:37 Timed out fetching CID 'QmaPE7xsFQ8HxhfQyAFdf3nWk7AidH3F9nQ12j3up4RB2S'.
2025-01-30 04:59:37 Timed out fetching CID 'QmQyrFpvpEjw558LP9tLARRgUgqMTJXLmZL8bsXyUcXHxP'.
2025-01-30 04:59:37 Timed out fetching CID 'QmUvfGKksdK58ucaBT5UB599UfukGCSxzMdNakrF9cXsrG'.
2025-01-30 04:59:37 Event emitted: sidetree_download_manager_download: {"code":"content_not_found"}
2025-01-30 04:59:37 Event emitted: sidetree_download_manager_download: {"code":"content_not_found"}
2025-01-30 04:59:37 Event emitted: sidetree_download_manager_download: {"code":"content_not_found"}
2025-01-30 04:59:37 Transaction 2.QmaPE7xsFQ8HxhfQyAFdf3nWk7AidH3F9nQ12j3up4RB2S is confirmed at 66594
2025-01-30 04:59:37 Transaction 1.QmQyrFpvpEjw558LP9tLARRgUgqMTJXLmZL8bsXyUcXHxP is confirmed at 66588
2025-01-30 04:59:37 Transaction 1.QmUvfGKksdK58ucaBT5UB599UfukGCSxzMdNakrF9cXsrG is confirmed at 66604

Steps Taken:

• Verified the node configurations and ensured they point to the mainnet.
• Successfully created and published DIDs from these private nodes.
• Tested internal resolution of the DIDs created within these nodes, which works as expected.
• External DIDs always return "DID not found."

Additional Information:

• Both nodes are in a private network but have internet access for syncing.
• The CID fetching issue appears to block the ability to sync with the rest of the ION network.

Questions:

1. What could be causing the "Timed out fetching CID" error during block processing?
2. Are there specific configurations or network settings required to enable smooth synchronization with the public ION network?
3. Could this issue be related to private network restrictions on IPFS or other underlying services used by ION nodes?

Environment:

• DID ION Node version: custom build based on sidetree repository.
• Operating system: Ubuntu 20.04
• Deployment: Docker-based

I would appreciate insights from the community, particularly from @csuwildcat , @thehenrytsai and @transmute-industries, as their expertise on similar issues would be invaluable.

Thank you in advance for your support.