Kusama Validators randomly(?) missing ~30% of votes in 1 session

[florentina57]

I have 2 iterations of 2 Kusama validators in the same group missing ~30% of the votes in 1 session, resulting in a bad performance grade (C) on https://apps.turboflakes.io/. While all other previous and following sessions remain with excellent performance (A+ or A).

In each case I have verified connectivity, no packet loss for the given session on IPv4 or IPv6 (monitoring from 3 continents). Due to latency my validators do miss sometimes a couple of votes in a session, but NOT >30%, and their historic MVR <0.007%.

Logs show many of the following during the specific session:
fetch_pov_job err=FetchPoV(NetworkError(NotConnected)) ,
fetch_pov_job err=FetchPoV(NetworkError(Refused)) and
fetch_pov_job err=FetchPoV(NetworkError(Network(Timeout)))

1st instance: Session 47380:
Validators:
"🌎 ValidAndina 🏔️ LIM 🇵🇪 KSM": MVR: 37.1%
"DOZENODES.COM/DN1": MVR: 33.7%

Remarks:
Parachains w/ missed votes: 12/17
Validator geographical location: Peru and Thailand

2nd instance: Session 48130
Validators:
"🌎 ValidAndina 🏔️ BOG 🇨🇴 KSM": MVR: 32.1%
"Brightlystake/01": MVR: 28.3%

Remarks:
Parachains w/ missed votes: 7/8
Validator geographical location: Colombia and ?

[florentina57]

Update: I just noticed that in both instances a 3rd validator(JKzwkX6JcARutZvJpArnUFZnYnbsZedCTJfbhrqTtPuxezU) was present in the group...

[bkchr]

CC @alexggh

[infrachris]

Including my logs from this time frame when I was also receiving "fetch_pov_job err=FetchPoV(NetworkError(Network(Timeout)))".
The node is running on dedicated baremetal hardware, 7950x, 10Gbps internet, located in Salt Lake City Utah, USA.

ic02-ksm-last-5-days-20250526_232916.log.gz

[florentina57]

Update: I just noticed that in both instances a 3rd validator(JKzwkX6JcARutZvJpArnUFZnYnbsZedCTJfbhrqTtPuxezU) was present in the group...

I've been looking at other sessions with the validator mentioned above (Ice) and I identified a 3rd and 4th instance of other validators in the group missing a large percentage of votes on most assigned parachains and resulting in bad performance during the given session.

3rd instance:
Session 48152

4th instance:
Session 48160

Although, in the above 4 cases "Ice" validator was present in the group, it does not seem to happen in every session the validator is in. To support this I also identified a session (48150) where "Ice" validator was in the same group as one of my validators and NOBODY missed any votes.

[alexggh]

The way the backing statement distribution protocol works, you do rely on other nodes to distribute your statements toward the block author, so depending on the latency between your node and the collators + your node and the other nodes in the backing group, it might be possible that your statements don't make it in time on the block author, if they arrive late then they won't be noted on-chain and you won't get backing points.

This backing groups get randomised every session, so across many sessions, everyone will be affected similarly.

[sandreim]

it might be possible that your statements don't make it in time on the block author, if they arrive late then they won't be noted on-chain and you won't get backing points.

I have looked at this a while back, and the statements have around 5s (minus pvf exec time) to arrive at the block author. I would expect this to be in general enough time, unless connectivity is low for which we have some hints in this ticket:

fetch_pov_job err=FetchPoV(NetworkError(NotConnected)) ,
fetch_pov_job err=FetchPoV(NetworkError(Refused)) and
fetch_pov_job err=FetchPoV(NetworkError(Network(Timeout)))

[hitchhooker]

missing votes due to timeouts is common for validators outside Europe. We need peer IP addresses in timeout logs to diagnose network issues.

currently logs only show:

fetch_pov_job err=FetchPoV(NetworkError(Network(Timeout))) para_id=Id(2239) authority_id=Public(0x70c2e207...)

without peer IPs, we can't:

• identify geographic/routing patterns
• debug ISP routing issues
• fix problematic AS paths

even with significant network infrastructure (3M routes, redundant transit via HGC/Cogent, peering at AMSIX BKK/HK/AMS + BKNIX 10G ipv6 transit with HE), we still see 10-20% vote misses from timeouts.

like having global transit in region like SEA is really expensive(1-4 usd/mbits) so if we really want to continue on this path of pushing direct networking with higher bandwidth and not move whole network to be run from europe, having easy way to determine those poor networking conditions would help a lot. there is after all provides that provide affordable point to point transit based on /24 and /48 ip:s(f.g. megaport)

[burdges]

missing votes due to timeouts is common for validators outside Europe.

Intresting, but which votes do they miss?

Are these FetchPoV errors only for backing? Or are they the first attempt for approval checking too? Are these nodes becoming no-shows? If they're becomoing no-shows then we should increase the no-show timeout. Approval checks could be given enough time imho.

If otoh they're only missing backing checks, then we could probably increase some timeouts there too, as async backing should permit doing this, but timeouts require more care in backing.

We've a major bug right now that we reward backing checks far too highly, but do not reward the much more important approval checks. We've a solution outlined in polkadot-fellows/RFCs#119 but we could speed up implementation so that node operators miss fewer rewards. We'd want the better reports this provides anyways.

[florentina57]

Update: I just noticed that in both instances a 3rd validator(JKzwkX6JcARutZvJpArnUFZnYnbsZedCTJfbhrqTtPuxezU) was present in the group...

Today I identified 2 more examples of 2 validators missing votes in the group, while the SAME 3rd validator was present in the group ("Ice" JKzwkX6JcARutZvJpArnUFZnYnbsZedCTJfbhrqTtPuxezU) but, again, he was not missing any votes.

5th instance:
Session 49833

6th instance:
Session 49839

[florentina57]

The performance issues with high MVR of one of my Kusama validators (located in South America) appeared 2 more times and once again they include the same validator in the set during the given session:

Identity: Ice (JKzwkX6JcARutZvJpArnUFZnYnbsZedCTJfbhrqTtPuxezU)

I'm documenting these instances here for the future. Without knowing the location of the other validator it's hard to tell if the geographical distance is a potential cause or not.

7th instance:
Session 50,579

8th instance:
Session 50,655

[pmensik]

We are experiencing the same thing on our validators in Singapore - most of the time they are fine but a couple of times per week their rating drops to B or C. We tried switching to backup in the EU to see whether the issue disappears and yes - the performance was flawless afterwards.