Fix uplink to detect ec problems, and retry with more pieces, instead of failing

[mniewrzal]

It seems that if one of pieces requested for download is malformed then whole download will fail. We need to correctly detect such situation and request another piece for download.

Acceptance Criteria:

• update libuplink to download object successfully even if one of requested pieces is malformed

More context:
https://github.com/storj/customer-issues/issues/2990#issuecomment-3018549553

[jtolio]

Some prior work:

https://review.dev.storj.tools/c/storj/uplink/+/12032
https://review.dev.storj.tools/c/storj/uplink/+/12141
https://review.dev.storj.tools/c/storj/uplink/+/12157

[ifraixedes]

I've looked to the code to find this for several days. The last 2 days, I focused on reading the method StripeReader.ReadStripes because my understanding is that method that will detect if a piece is corrupted.

A corrupted piece should have corrupted shares, so infections decoding will return an error. That happens https://github.com/storj/uplink/blob/main/private/eestream/stripe.go#L409 and we handle the error on https://github.com/storj/uplink/blob/main/private/eestream/stripe.go#L421

Analyzing the logic, I don't understand how a corrupted piece will make everything to fail because a share will be corrupted, but if we have more pieces to read that share, it should be fine. The only way that I see that could fail is that there aren't more pieces correct pieces to read enough needed shares to reconstruct the stripe. However, I don't see how that error will get out from that method because every recursive call should end in another recursive call at line https://github.com/storj/uplink/blob/main/private/eestream/stripe.go#L424 and we found that uplink ends with an error, not hang forever.

On the other hand, infectious doesn't tell what share is corrupted, hence we cannot identify which pieces has a corrupted. As a result we cannot ask for another piece and replace it (in case that we could request that to the satellite). What we could try is to ask for new pieces (order limits) for the segment and try again, but we have to signal the caller.

To signal the caller, we could have a specific error class and use it when infections returns TooManyErrors. TooManyErrors is only returned in one place https://github.com/storj/infectious/blob/v0.0.2/berlekamp_welch.go#L211 which a method called by the Correct method https://github.com/storj/infectious/blob/v0.0.2/berlekamp_welch.go#L112

Currently we don't distinguish between TooManyErrors and NotEnoughShares https://github.com/storj/uplink/blob/main/private/eestream/stripe.go#L447 (called from https://github.com/storj/uplink/blob/main/private/eestream/stripe.go#L421).

StripeReader.ReadStripes method looks fine after I read the code a lot of times and I have thoughts that it was wrong in some places to handle the mentioned problem, but I checkout changes because I realized that I was wrong all the times.

I'd like you hear if I should explore the code with the hope that we don't need to make too many changes and it will be feasible to get only orders for a segment and, in that case, apply the changes.

An important side note is that I didn't find from where the error message posted on https://github.com/storj/customer-issues/issues/2990#issuecomment-2996696492 comes.

[mniewrzal]

An important side note is that I didn't find from where the error message posted on https://github.com/storj/customer-issues/issues/2990#issuecomment-2996696492 comes.

It comes from encryption.ErrDecryptFailed which wraps non storj errors, most probably something related to AESGCM.

I noticed also that we have eventkit metric for this error on libuplink side and this problem is not as rare as I was thinking. It's like 15k-45k per day.