Lotus Daemon gets out of sync daily

[plbertrand]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Local: lotus version 1.17.1+mainnet+git.e3894de

Describe the Bug

1. Nothing
2. It was synching properly and then the splitstore started compacting.

Oct 19 02:34:05 filecoin conmon[3484088]: 2022-10-19T02:34:05.012-0400        INFO        chain        chain/sync.go:625        block validation        {"took": 0.91325791, "height": "2261828", "age": 5.012851525}
Oct 19 02:34:05 filecoin conmon[3484088]: 2022-10-19T02:34:05.051-0400        INFO        chain        chain/sync_manager.go:323        worker 11319 done; took 1.046932789s
Oct 19 02:34:09 filecoin conmon[3484088]: 2022-10-19T02:34:09.366-0400        INFO        net/identify        identify/id.go:364        failed negotiate identify protocol with peer        {"peer": "12D3KooWB8kWpjWFyZJBYPQvbHpcY6Wn3adpny2YfcDuToRvei8Y", "error": "stream reset"}
Oct 19 02:34:10 filecoin conmon[3484088]: 2022-10-19T02:34:10.023-0400        INFO        net/identify        identify/id.go:364        failed negotiate identify protocol with peer        {"peer": "12D3KooWGviNgrirRSPscB3m4yc3TUMiKtdwKWwTA7i7Ca4xtY7H", "error": "stream reset"}
Oct 19 02:34:10 filecoin conmon[3484088]: 2022-10-19T02:34:10.598-0400        INFO        splitstore        splitstore/splitstore_compact.go:447        protecting transactional refs done        {"took": 9.660075708, "protected": 17621}
Oct 19 02:34:10 filecoin conmon[3484088]: 2022-10-19T02:34:10.598-0400        INFO        splitstore        splitstore/splitstore_compact.go:396        protecting transactional references        {"refs": 13925}
Oct 19 02:34:10 filecoin conmon[3484088]: 2022-10-19T02:34:10.810-0400        INFO        splitstore        splitstore/splitstore_compact.go:447        protecting transactional refs done        {"took": 0.212303441, "protected": 68}
Oct 19 02:34:10 filecoin conmon[3484088]: 2022-10-19T02:34:10.810-0400        INFO        splitstore        splitstore/splitstore_compact.go:396        protecting transactional references        {"refs": 401}
Oct 19 02:34:10 filecoin conmon[3484088]: 2022-10-19T02:34:10.853-0400        INFO        splitstore        splitstore/splitstore_compact.go:396        protecting transactional references        {"refs": 116}
Oct 19 02:34:10 filecoin conmon[3484088]: 2022-10-19T02:34:10.857-0400        INFO        splitstore        splitstore/splitstore_compact.go:822        waiting for sync

Then these messages started to appear and the synching stopped:

Oct 19 02:34:31 filecoin conmon[3484088]: 2022-10-19T02:34:31.131-0400        WARN        splitstore        splitstore/splitstore_compact.go:258        missing object reference bafy2bzaced54dpjrbsv4mf6rcp4rrjmzetz22yis4mc2qm7g4hanbekvmv4mk in bafy2bzaceddk44vzpstayi7fgsddma2t5usouyvvknm3q2ckhfkivtt5hamai
Oct 19 02:34:31 filecoin conmon[3484088]: 2022-10-19T02:34:31.131-0400        INFO        chain        chain/sync_manager.go:233        selected sync target: [bafy2bzaceddk44vzpstayi7fgsddma2t5usouyvvknm3q2ckhfkivtt5hamai]
Oct 19 02:34:31 filecoin conmon[3484088]: 2022-10-19T02:34:31.131-0400        INFO        chain        chain/sync_manager.go:315        worker 11320 syncing in [bafy2bzaceddk44vzpstayi7fgsddma2t5usouyvvknm3q2ckhfkivtt5hamai]
Oct 19 02:34:31 filecoin conmon[3484088]: 2022-10-19T02:34:31.159-0400        WARN        splitstore        splitstore/splitstore_compact.go:258        missing object reference bafy2bzaced54dpjrbsv4mf6rcp4rrjmzetz22yis4mc2qm7g4hanbekvmv4mk in bafy2bzacebuuvv67jffknyg6gh2d4nlupvpnhkhrniizqrerufk6emc34tndy

3. Not prune the chain too much.
4. N/A
5. N/A
6. N/A

Logging Information

https://drive.google.com/file/d/19EIifLIsLjt1jDGFkCUe5coYXw7J3NRo/view?usp=sharing

Repo Steps

Run lotus daemon with split store enabled and cold store as discard. Feel free to ask more specific questions, I'll add to the description as needed.

[dd45e640b42e6da7da96faee3996ef7c]

whats the hardware this is running on?

[TippyFlitsUK]

As Patrick said, it would be very useful to get an idea of your system spec. Can you also please disable SpiltStore and see if this error persists?

Many thanks Pierre 🙏.