fix: claimed in staking info endpoint

[Imod7]

Description

Closes #1433

This PR restores the field claimed (under staking) in the staking-info endpoint's response which was "broken". The root cause of this is explained here.

For the Reviewer

To review this PR please refer to :

1. the hackMD: "Staking PR - claimed field - Guide".
2. the README: "STAKING_IMPLEMENTATION_DETAILS.md".

Todos

• Implement new logic based on comment
• Add a test that includes the era when the migration to the new logic happened
• Add a test that includes also partially claimed eras & erasStakersOverview = null
• Should return a total amount of eras (and their status) that are equal or less than depth.
• Merge master branch & update all historical tests (with new response type)
• Handle the case in early blocks/eras where there is lastReward instead of claimedRewards under stakingLedger
• Add the nominator logic mentioned in this comment
• Update docs
• Add IMPLEMENTATION_DETAILS guide
• Add reviewer's HackMD

Changelog

This PR introduces breaking changes to the staking-info endpoint response by :

1. Removing the returned field lastReward for early eras and adding claimedRewards updated accordingly.
2. Restoring the claimedRewards field name after the breaking change of renaming it to legacyClaimedRewards
3. Changing the structure/type of the returned claimedRewards field (under staking) : before we had an array of eras that were claimed, now we have an object with the list of eras and their corresponding status
4. Refactoring the whole logic that returns the claimedRewards
5. Adding the possibility to query nominator's claimed Rewards apart the validator's.

These changes should be explicitly mentioned in the corresponding Changelog and Release Notes of the release in which this PR is included.

[IkerAlus]

The change is reasonable. Sidecar user does not care of internal renaming changes since the retrieved info has the same meaning.

Is this PR ready for review or are we waiting for the Todos list to be checked?

[Imod7]

The change is reasonable. Sidecar user does not care of internal renaming changes since the retrieved info has the same meaning.

Is this PR ready for review or are we waiting for the Todos list to be checked?

Yes, ideally the todos needs to be completed before a review. However, if any review will speed things up then please feel free to add them.

[Imod7]

Update
After getting feedback from @Ank4n, for the 2nd check mentioned in the logic section above, I concluded that the logic and thus the implementation in this PR should change to the following:

• If staking.erasStakersOverview.pageCount == query.staking.claimedRewards -> era claimed
• If staking.erasStakersOverview.pageCount != query.staking.claimedRewards. length -> era partially claimed
• If overview == null && erasStakers > 0 -> pageCount = 1 : so then it depends on query.staking.claimedRewards value to see if era claimed or unclaimed

The response will also needs to be changed from a Vec of eras to a Vec of objects of type {era: eraNumber, status : claimed | unclaimed | partially claimed}

Related Resource
polkadot-js/api#5859 (comment)

[marshacb]

LGTM! Just had a question about unwraps

[IkerAlus]

Where/how can the user find the relevant linked info?

[Imod7]

Added specific links of the PR and linked issue. Let me know if this covers your question. I also updated the schema description since I no longer return lastReward. Changes included in this commit.

[IkerAlus]

The response time of the endpoint is significantly slower now

Do we have specific data for this statement? How long does it take to return the staking info in the any worst case scenario (for example, 84 eras in different pages and/or with different logic)?

[Imod7]

The response time of the endpoint is significantly slower now

Do we have specific data for this statement? How long does it take to return the staking info in the any worst case scenario (for example, 84 eras in different pages and/or with different logic)?

You are right, I didn't have any formal data when I wrote that. It was based on my observations from the tests I did and also on my knowledge of the calculations of each implementation.
After adding some simple code to measure the execution time of fetchAccountStakingInfo, it looks like there are cases with similar runtimes (old code vs new code) and others have significant differences. However, these results can be influenced by factors such as the connection and the machine used. I am including below some results but to back this statement I need to run proper benchmarks.

• http://127.0.0.1:8080/accounts/GpyTMuLmG3ADWRxhZpHQh5rqMgNpFoNUyxA1DJAXfvsQ2Ly/staking-info?at=16869643 - kusama

  • old code : 608 ms
  • new code : 1.612 ms

• http://127.0.0.1:8080/accounts/CmjHFdR59QAZMuyjDF5Sn4mwTgGbKMH2cErUFuf6UT51UwS/staking-info?at=22869643 - kusama

  • old code : 372 ms
  • new code : 5.312 ms [this is an edge case- goes super slow]

• http://127.0.0.1:8080/accounts/11VR4pF6c7kfBhfmuwwjWY3FodeYBKWx7ix2rsRCU2q6hqJ/staking-info?at=18157800 - polkadot

  • old code : 433 ms
  • new code : 408 ms

[Imod7]

@IkerAlus After our last convo on whether the status partially claimed is valid or not and some more feedback from the staking master @Ank4n :

For a validator to get all commission, they should claim reward for all pages of an era.

Here is the next round of conclusions about claimed:

• When a user queries the staking-info endpoint with a validator address and at a specific era&block
  • The possible status is still : claimed, partially claimed or unclaimed
• When a user queries a nominator address and at a specific era&block
  • @IkerAlus you were right, the possible status can only be : claimed or unclaimed

Currently, I am also considering that in both scenarios, it is useful to add the undefined status which is the case when we only have lastReward available (early eras). In such cases, we cannot determine if the queried era was claimed unless it matches the era noted in lastReward. If the queried era does not correspond to the lastReward era then claimed should be set as undefined.

[filvecchiato]

I need to understand oldCallsCheck flag a bit more but in the meanwhile a few minor nits that might help readability

[filvecchiato]

I have added some comments, or possible updates to the codebase to increase performance ( I ran a few tests and it seems like the response time halved). Quite a complex PR, good job in getting it out!

[filvecchiato]

All good on my side! I think the oldCallCheck flag can be moved outside the loop in a later stage (maybe create another tracking issue for that), great job overall, very complex stuff!!

[Imod7]

Thank you @filvecchiato for the super thorough and detailed review. All your notes on the performance improvements were well noted and implemented. Your questions, doubts and thoughts on the logic were excellent and super helpful in improving the implementation guide for future readers. I greatly appreciate the time, effort and brain matter (due to its complexity) you put into this review. A big thank you!

[filvecchiato]

Thanks for the kind words! I'm glad I could help! LGTM 🚀