Some network error occurred when fetching erasure chunk

[m-saxemberg]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

The Validator SAXEMBERG3 has found the bug following the update to v 1.22.0

2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(4a924309e7119738b4ad6cc6fcaa871e6aad68313f1f92bc6ef4b227098f1343 (12gmzZAL...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(97) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0xdf6d923ec7126867045155261ebec2e5819d05b8e1d42fb9ac220361bd17fd9d err=Refused traceID=296986769623174515542970339148854903525 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(b0b93789fa924191f7266e4d989b7a009432d4580a99f7090377c056f007516f (14ziT2Cz...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(96) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0xa6ddfb558ac97fb78ccba9061429d8fcbffc54dd11d649a311adb72abd3bd7da err=Refused traceID=221804442568189090661213356217684580604 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(3e95a43b61c076c8884ceec23b7058f264ae7f86ee4662c8bc481bec7a08920e (12R4RTWC...)) relay_parent=0x3de145b14f8b7715905b3e82f9e45c510ef34bb526acf45fecc98dc9b0687ac4 group_index=GroupIndex(101) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0xf32d45fa99bb3ce8a4fe610fab5741a573239eb3181c40de912636bd56dab755 err=Refused traceID=323237475675249468154709013965387022757 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(ac0b241bc0084aad52a6721f210be6efff06a4a28a1dc8cbbe68cc8e38be9a64 (14taYeuj...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(98) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0x537ecd3a4f57272ccb2cafb53de1605451523d618ba5cecceb45849d4f618559 err=Refused traceID=110984315568080257472761892161290592340 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(daaf1e41b0617000326c098aaa05826b0be68a2ef955f55f31e7cbbadcf34329 (15wjTft8...)) relay_parent=0xda2a4823748d3e511240288b00e7ef12835d93bcef9057f214c37bd1af1771ec group_index=GroupIndex(76) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0xe3793efe50e43004bfdeea8c2bfe6326d351150178e33a69fc1f2ed0f111f88c err=Refused traceID=302364300621441984952708067010669339430 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(ba50d793c4bdd0e6e1ee205ccbdc5c27a58dabee8a84a5d414fdaaf4a755fb08 (15DHvTWm...)) relay_parent=0x3de145b14f8b7715905b3e82f9e45c510ef34bb526acf45fecc98dc9b0687ac4 group_index=GroupIndex(115) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0xe89accdbca9e6459a242d2840e9c2b72d7e80f7eb74891aa8ffff645505563b4 err=Refused traceID=309184663763549037686644816509037980530 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(54158aad6cb3852e7bf594ef17f87ee3534a05d25bf815e1a35f1116e07f4677 (12uFRT4P...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(7) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0x5210cb244e6252e9f032d585d459b4a2037027a6c2d9a01a09283fda58b77c73 err=Refused traceID=109083892609721747668619126137444480162 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(844cb2d01393ef274fed7715204d9c6b7a30e599eb727e50fc7bc77e34b27a07 (13zU76Bz...)) relay_parent=0x3de145b14f8b7715905b3e82f9e45c510ef34bb526acf45fecc98dc9b0687ac4 group_index=GroupIndex(74) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0x0b5b13d94361459ca39be0aa55e9fb11eac542dbb3adff943febe71cea03af03 err=Refused traceID=15094409546907569088538861236333968145 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(2e7a3cddd4ab8ad9c52c6f6467681864a73d6dc4c3750ca10ef2ecb10ab5cb2b (123wWs7N...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(1) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0x0dee1edb84976dfb2f30d97bd77825d7ffd2ad1ed8f518b074422f193f0ba9de err=Refused traceID=18516356461825983964001611527723951575 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(74f7a4cc3db21c19046f0f5c4325a9c76fd6c5c3941f6ba973bdf69fb5dc7435 (13eN7yJB...)) relay_parent=0xf202f3b0044775adcd013820a15506ec1ce13bbfbf7f9320129b8035d9c9a4f8 group_index=GroupIndex(80) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0x42115a785e0e83731ea656fb7fdf5d1cacb03cfa074084303490f709a6cc18ca err=Refused traceID=87819151721752506681669364868139932956 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(0abbc88ac382a2438ab7bc553ca60d2124da53dfcda50fd9d306c0e0e722b07f (1F5GCuNk...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(5) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0xc7a1b0db06558e26d48d855a4181407c70c75d081def7fcfc4c92a4bf359da89 err=Refused traceID=265355918012440539319863763320375951484 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(e4108dbb40047e8437b46abf6fe36d22fef9dc74b231f98abb599c29fea87132 (16A2qSWm...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(104) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0x5cabd12f82f9b01bb2eb6f02b2606ed5f7540e9ffcc98e8ed609d360c32ae59d err=Refused traceID=123181101162887422654485933352263118549 2026-04-07 14:50:37 Some network error occurred when fetching erasure chunk origin=Public(1056f10065e442a64d4edacab1f30453ab506de1e4adbae283de02b150d8c578 (1NRbQMZz...)) relay_parent=0x91a83bbc03cce1cbf6017d75c74060c8be9cda84532db1772ff49c6e2d679370 group_index=GroupIndex(116) session_index=12851 chunk_index=ValidatorIndex(552) candidate_hash=0x10d87c5c577d4c90009135c54f408eafb5801703442f8ec7bf1b0d6981f158f8 err=Refused traceID=22391706388860653991808695563083353775 2026-04-07 14:50:37 Recovery of available data failed. candidate_hash=0xb500d1fc3b1e23ca6f3ee9bf9b96297dd0b12528562f64229de101312ab80977 traceID=240594526244469942160453859112176200061 2026-04-07 14:50:37 Error during recovery: Data is unavailable 2026-04-07 14:50:37 Data unavailable for candidate (0xb500d1fc3b1e23ca6f3ee9bf9b96297dd0b12528562f64229de101312ab80977, 2094) para_id=2094 candidate_hash=0xb500d1fc3b1e23ca6f3ee9bf9b96297dd0b12528562f64229de101312ab80977 attempts_remaining=16 traceID=240594526244469942160453859112176200061 2026-04-07 14:50:38 Recovery of available data failed. candidate_hash=0x6652ed16b38d6315dd6f818614b5a5375f4a3f5c5672a54057aec84fecaf9ed4 traceID=136011832642125658877639697479929013559 2026-04-07 14:50:38 Error during recovery: Data is unavailable 2026-04-07 14:50:38 Data unavailable for candidate (0x6652ed16b38d6315dd6f818614b5a5375f4a3f5c5672a54057aec84fecaf9ed4, 2037) para_id=2037 candidate_hash=0x6652ed16b38d6315dd6f818614b5a5375f4a3f5c5672a54057aec84fecaf9ed4 attempts_remaining=16 traceID=136011832642125658877639697479929013559

EDIT April 8th
At this moment the percentage of missed votes was 37% according to Turboflakes' One-T tool, showing it had a significant impact on performance and it was not just a minor performance issue.

after reboot it continues with:

fetch_pov_job err=FetchPoV(NetworkError(Refused)) para_id=3367 pov_hash=0x289bb7d685fdc80f344b51ee34ed35ee6845e4e70d4d66166a00aba8c44318a2 authority_id=Public(d82399fc4cc54d618c47974d1ab29153fc861d75bd35d39a883c5f21e32ce77d (15tPvJMf...))
After the next epoch it seems that this message has gone. The same machine is now producing votes and blocks now. It will be considered for change if this issue reappears.

It was reported already on the Element channel without any answer.

Previous network or connectivity errors were detected without any logs where a reboot was attempted but produced no result.

Steps to reproduce

No response

The hardware is new and above the minimum and recommended requirements. No memory or I/O issues were detected using diagnostics tools

Peering doesn't seem to be a problem either as it is connected to around 36 peers

These are the monitoring images at the beginning of the connectivity issue: These are a little misleading as the connectivity issue continues after the ravine of activity.

The biggest tale tell of the issue is the task monitor as it shows that no new tasks have been active since the ravine. All tasks go flat though yesterday's issue seemed to have had only two tasks go flat.

[m-saxemberg]

Let us know if any extra flag can help find the cause of this bug. It could be ran with some other machine at the same location too. Probably just for a little while with this exact machine as it's a validator.

[sigurpol]

@alindima , @sandreim , @pgherveou can you please take a look when you have time 🙏 ?

[sandreim]

Regarding connected peers, your picture does not help as I think it charts the sync protocol peers. We want to see the parachain consensus protocol peers. So please use this query: avg(polkadot_parachain_peer_count{chain="$chain", node=~".*validator.*"}) by(protocol)

err=FetchPoV(NetworkError(Refused)) this means that the target node is refusing the connection indicating connectivity issues on your side or their side.

But this one below looks more interesting. We seem to not be able to recover a parachain candidate. If this happened after restart, could be just a symptom of poor connectivity / not enough of the network discovered.

(0xb500d1fc3b1e23ca6f3ee9bf9b96297dd0b12528562f64229de101312ab80977, 2094) para_id=2094 candidate_hash=0xb500d1fc3b1e23ca6f3ee9bf9b96297dd0b12528562f64229de101312ab80977 attempts_remaining=16 traceID=240594526244469942160453859112176200061 2026-04-07 14:50:38 Recovery of available data failed. candidate_hash=0x6652ed16b38d6315dd6f818614b5a5375f4a3f5c5672a54057aec84fecaf9ed4 traceID=136011832642125658877639697479929013559 2026-04-07 14:50:38 Error during recovery: Data is unavailable 2026-04-07 14:50:38 Data unavailable for candidate

With that being said, is this issue still happening now ?

[sandreim]

Are you overriding the now default litep2p network backend with --network-backend libp2p on the cmd line ?

[m-saxemberg]

(0xb500d1fc3b1e23ca6f3ee9bf9b96297dd0b12528562f64229de101312ab80977, 2094) para_id=2094 candidate_hash=0xb500d1fc3b1e23ca6f3ee9bf9b96297dd0b12528562f64229de101312ab80977 attempts_remaining=16 traceID=240594526244469942160453859112176200061 2026-04-07 14:50:38 Recovery of available data failed. candidate_hash=0x6652ed16b38d6315dd6f818614b5a5375f4a3f5c5672a54057aec84fecaf9ed4 traceID=136011832642125658877639697479929013559 2026-04-07 14:50:38 Error during recovery: Data is unavailable 2026-04-07 14:50:38 Data unavailable for candidate

With that being said, is this issue still happening now ?

After the next epoch (12852) and two machine reboots the message(s) is/are gone and activities/tasks are recorded again. The same machine is now producing votes and blocks now. A replacement at a central Europe/US east coast location will be considered if/when this issue reappears as most networking issues on Polkadot disappear when a validator is located in central/west Europe and US east coast.

Are you overriding the now default litep2p network backend with --network-backend libp2p on the cmd line ?

no

[sandreim]

As most networking issues on Polkadot disappear when a validator is located in central/west Europe and US west coast.

This sounds surprising.

Where is your validator located now ? And can you share the data that supports this conclusion ?

[m-saxemberg]

Where is your validator located now ?

Johor Malaysia, right next to Singapore
at Equinix JH1

And can you share the data that supports this conclusion ?

It's been already referenced at #10921 with reduced performance for non-EU / west-US locations and still impacting newer releases. Attestations: #10504 (comment)
Importance of this issue when DAP phase 2 happens
https://forum.polkadot.network/t/changes-on-polkadot-in-march-2026/17101/2

There are some older technical posts by Rotko when them as an ISP dealt with performance in Thailand as well

There are other cross-references about the importance of location on other threads as well. Probably the old W3F DN program has the full collection of this data as it was cross referenced with known geolocation. Live data from Turboflakes one-t should also support this.

[sandreim]

One reason this might be happening is that votes propagation is slower for these nodes due to latency. One way to find out if this is true or not is this histogram: sum(increase(polkadot_parachain_provisioner_backable_vs_in_block_bucket{chain="$chain", node=~".*validator.*"}[$__rate_interval])) by (le)

Can you please share it for any validators running in these regions ?