Node Issue: MissingTrieValue Error Encountered During Version Upgrade from 2.5.0-rc.1 to 2.5.0-rc.3

[edescourtis-c1]

Contact Details

edescourtis@chorus.one

Node type

RPC

Which network are you running?

testnet

What happened?

When upgrading a NEAR testnet node from version 2.5.0-rc.1 to 2.5.0-rc.3 (skipping 2.5.0-rc.2), the node failed to process blocks after restart, reporting a MissingTrieValue(TrieStorage, XMuesBVj3SqcHXSrWax4Ca6TjKivHpMgUwXdjcwPGJU) error. The node got stuck attempting to download blocks without making progress, requiring a complete data reset and resync from genesis to resolve the issue.

Important: This issue does not occur on every node. So far, we've only observed it on one node (testnet03) while other nodes appear to upgrade without this problem, suggesting there may be specific conditions or node states that trigger the issue.

Note: It's unclear whether this issue is specifically caused by 2.5.0-rc.3, pre-existed in 2.5.0-rc.1, or is related to the upgrade process itself.

Version

Previous version: 2.5.0-rc.1
Current version: 2.5.0-rc.3

Relevant log output

### Errors
Mar 04 07:55:06 testnet03 neard[2038871]: 2025-03-04T07:55:06.043693Z  WARN chain: Error in applying chunk for block shard_id=9 hash=EZagfNjs8LT9WKiqVN5a56AQLuzCfyRkJMqDFBeTke6k err=Storage Error: MissingTrieValue(TrieStorage, XMuesBVj3SqcHXSrWax4Ca6TjKivHpMgUwXdjcwPGJU)

Mar 04 07:55:06 testnet03 neard[2038871]: 2025-03-04T07:55:06.043764Z ERROR client: try_process_unfinished_blocks got errors errors={EZagfNjs8LT9WKiqVN5a56AQLuzCfyRkJMqDFBeTke6k: StorageError(MissingTrieValue(TrieStorage, XMuesBVj3SqcHXSrWax4Ca6TjKivHpMgUwXdjcwPGJU))}

### The node got stuck in a download loop, continuously falling further behind
Mar 04 07:55:08 testnet03 neard[2038871]: 2025-03-04T07:55:08.197729Z  INFO stats: #189173187 Downloading blocks 0.00% (71 left; at 189173187) 32 peers ⬇ 438 kB/s ⬆ 487 kB/s 0.00 bps 0 gas/s CPU: 256%, Mem: 5.75 GB

Mar 04 07:55:18 testnet03 neard[2038871]: 2025-03-04T07:55:18.197152Z  INFO stats: #189173187 Downloading blocks 0.00% (81 left; at 189173187) 32 peers ⬇ 826 kB/s ⬆ 693 kB/s 0.00 bps 0 gas/s CPU: 166%, Mem: 5.79 GB

Mar 04 07:55:28 testnet03 neard[2038871]: 2025-03-04T07:55:28.198940Z  INFO stats: #189173187 Downloading blocks 0.00% (91 left; at 189173187) 32 peers ⬇ 1.11 MB/s ⬆ 915 kB/s 0.00 bps 0 gas/s CPU: 84%, Mem: 5.74 GB

Node head info

We did not capture the output of RUST_LOG=warn $NEARD --home $NEAR_HOME --unsafe-fast-startup view-state scan-db-column --column BlockMisc before resolving the issue. However, we have preserved the problematic data directory (named data-old-20250304) for further analysis if needed.

Node upgrade history

- Resharding and upgrade from 2.4.x
- Previous version: 2.5.0-rc.1
- Upgraded to: 2.5.0-rc.3 (skipping 2.5.0-rc.2)
- Date and time of upgrade: March 4, 2025, around 07:53 UTC

DB reset history

- Resharding and upgrade from 2.4.x