SPX SplitStoreV2 - Testing & Call Outs #8927

TippyFlitsUK · 2022-06-27T17:16:33Z

TippyFlitsUK
Jun 27, 2022
Maintainer

Hello SPX Fellows

Thank you for your participation!

We recommend testing SplitStoreV2 on a low-priority server first depending on resources available. If you do run into any issues please refer to the risk mitigation guidance below.

This test can be performed on both Mainnet & Calibration instances.

Please reach out to @jennijuju or @TippyFlits on Slack if you have any questions! 🙏

⚠️ Please backup ALL lotus wallets before doing anything! (lotus wallet list) ⚠️

lotus wallet export <wallet-address-1> > file-1
lotus wallet export <wallet-address-2> > file-2
lotus wallet export <wallet-address-3> > file-3
etc...

Upgrade to: stable 1.16.0
Set the following values in your lotus daemon config.toml: (you can run lotus config default to see all SplitStore config options)

[Chainstore]
  EnableSplitstore = true
  ColdStoreType = "discard"

Set the Lotus daemon log level for SplitStore to debug:

lotus log set-level --system splitstore debug

🚀 This post if you are participating in this test assignment.
Fill out the Reply card - report any bugs, if any.

Reply card:

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your ./lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

Directory	Current Size in GiBs
/.lotus/datastore/splitstore
/.lotus/datastore/chain

Reminders and Risk Mitigations

Importing a new lightweight snapshot with SplitStore enabled requires an additional step:

Prepare your /.lotus/datastore folder as per usual by deleting the contents of the /.lotus/datastore/chain folder
Run lotus-shed splitstore clear --repo </path-to/.lotus>
Import your new lightweight snapshot in the normal manner

In the event of a critical failure or data corruption, you will need to re-import a lightweight snapshot with the following steps:

❗ Ensure that all lotus wallets are fully backed up.
Stop lotus and delete your /.lotus /datastore folder
Download a new lightweight snapshot
Disable SplitStoreV2 in your /.lotus/config.toml - EnableSplitstore = false

📓 In the event that you are unable to start your daemon after trying the above steps, please delete just the folders in your /.lotus directory and retry the lightweight snapshot import. ❗ ONLY DELETE THE FOLDERS - the api, token and config.toml files should not be deleted!

Questions are welcomed in this discussion, the team will be monitoring and will check in regularly.

Issues can be reported at https://github.com/filecoin-project/lotus/issues. Issues should be submitted with:

Daemon logs
GO routine dumps lotus pprof goroutines and lotus-miner pprof goroutines
The operation/step at which the problem occurred
The stack trace or logs as relevant. As always, for large log files, you may upload them to this Google Drive folder and link them in the issue.

SBudo · 2022-06-27T22:40:32Z

SBudo
Jun 27, 2022

Calibnet (t029562)

If lotus daemon stays in sync
if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
if your ./lotus/datastore/chain folder is being successfully garbage collected
if normal mining tasks appear to be processed as expected
if normal daemon tasks appear to be processed as expected

Directory	Current Size in GiBs
/.lotus/datastore/splitstore	5.5GB
/.lotus/datastore/chain	60GB

1 reply

SBudo Jun 28, 2022

Garbage collection ran and reduced the size of the splitstore folder from 5.5GB to 5.4GB

SBudo · 2022-06-28T10:52:58Z

SBudo
Jun 28, 2022

Mainnet

If lotus daemon stays in sync
if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
if your ./lotus/datastore/chain folder is being successfully garbage collected
if normal mining tasks appear to be processed as expected
if normal daemon tasks appear to be processed as expected

Directory	Current Size in GiBs
/.lotus/datastore/splitstore
/.lotus/datastore/chain

Note: we use wallet servers

3 replies

SBudo Jun 28, 2022

Node is not in sync after enabling splitstore... seems to be stuck:

$ lotus sync wait
Worker: 5; Base: 1936938; Target: 1936939 (diff: 1)
State: message sync; Current Epoch: 1936939; Todo: 0
Validated 0 messages (0 per second)

The node is filling up the logs (about 5 messages per second) with:

2022-06-28T21:07:19.447+1000 DEBUG splitstore splitstore/splitstore_reify.go:160 reification aborted; reify limit reached

SBudo Jun 28, 2022

Seems the node caught up with the chain up to the last block, then started a splitstore compaction. Nothing else since, so my guess is that it's not going to try to sync until the compaction is completed (?)

2022-06-28T21:09:44.459+1000    DEBUG   splitstore      splitstore/splitstore_reify.go:160      reification aborted; reify limit reached
2022-06-28T21:09:44.472+1000    DEBUG   splitstore      splitstore/splitstore_reify.go:160      reification aborted; reify limit reached
2022-06-28T21:09:49.587+1000    INFO    chain   chain/sync.go:624       block validation        {"took": 6.307834444, "height": "1936938", "age": 49.587586602}
2022-06-28T21:09:49.599+1000    INFO    chain   chain/sync.go:624       block validation        {"took": 6.319399242, "height": "1936938", "age": 49.5991516}
2022-06-28T21:09:49.603+1000    INFO    chain   chain/sync.go:624       block validation        {"took": 6.324105978, "height": "1936938", "age": 49.603839537}
2022-06-28T21:09:49.604+1000    INFO    chain   chain/sync.go:624       block validation        {"took": 6.324997547, "height": "1936938", "age": 49.604753705}
2022-06-28T21:09:49.610+1000    INFO    chain   chain/sync.go:624       block validation        {"took": 6.330547655, "height": "1936938", "age": 49.610299713}
2022-06-28T21:09:49.620+1000    INFO    chainstore      store/store.go:646      New heaviest tipset! [bafy2bzaceb3gz7ycohtm5jkp76z5lclmm2nmqs5fszs372vndduakkmxbnxce bafy2bzacea6hsbiuk62cthcfqjbcmome2txfuw6ug27j6cskj5ktb4khua6wu bafy2bzaceb7l77lwxnif6cdawzypcevbo4bbfgofsflbaaxb4kdq6uypntsvy bafy2bzaceb3bd2uzcfncxerbj5j3txkpr2t7w562k6s3ydwqs6jtkkus4dkfg bafy2bzaceb3jufwagdpmrlnyxkeylqcaepm2zamxvhtlo4uxje2zkylivqhga] (height=1936938)
2022-06-28T21:09:49.620+1000    INFO    splitstore      splitstore/splitstore_compact.go:741    preparing compaction transaction
2022-06-28T21:09:49.621+1000    INFO    chain   chain/sync_manager.go:322       worker 2 done; took 33.637279417s
2022-06-28T21:09:49.621+1000    INFO    chain   chain/sync_manager.go:272       selected sync target: [bafy2bzacecxo7oof7pwqigktlmk4pd5pqoijrppgsw3mz6o5qapr6xam4ogtw]
2022-06-28T21:09:49.621+1000    INFO    chain   chain/sync_manager.go:314       worker 4 syncing in [bafy2bzacecxo7oof7pwqigktlmk4pd5pqoijrppgsw3mz6o5qapr6xam4ogtw]
2022-06-28T21:09:50.217+1000    INFO    splitstore      splitstore/splitstore_compact.go:119    compacting splitstore
2022-06-28T21:09:50.217+1000    INFO    splitstore      splitstore/splitstore_compact.go:480    waiting for active views to complete
2022-06-28T21:09:50.219+1000    INFO    splitstore      splitstore/splitstore_compact.go:483    waiting for active views done   {"took": 0.00174354}
2022-06-28T21:09:50.219+1000    INFO    splitstore      splitstore/splitstore_compact.go:511    running compaction      {"currentEpoch": "1936938", "baseEpoch": "1807980", "boundaryEpoch": "1933338", "inclMsgsEpoch": "1933338", "compactionIndex": 15}
2022-06-28T21:09:50.238+1000    INFO    splitstore      splitstore/splitstore_compact.go:526    protecting references with registered protectors
2022-06-28T21:09:51.037+1000    INFO    chain   chain/sync_manager.go:232       selected sync target: [bafy2bzacecxo7oof7pwqigktlmk4pd5pqoijrppgsw3mz6o5qapr6xam4ogtw bafy2bzacecfdim4e35kuhhiplbdob3mznkwcft2fsxk4ms3prymgm2twwuo7w bafy2bzaceakwjzj2nemy6nduvu45bxftmetgmmw5xgsqjgy4xc5b3jzsgo5ac bafy2bzacebsmfbretigby4lwx4yops54cqptwifdtlqywc7ft62wvzq25stcy bafy2bzaceagamhhd3azaqzf4yrw2adqeuphfsnxchzz3ag2ye5ren4vsnlqzc]
2022-06-28T21:09:51.037+1000    INFO    chain   chain/sync_manager.go:314       worker 5 syncing in [bafy2bzacecxo7oof7pwqigktlmk4pd5pqoijrppgsw3mz6o5qapr6xam4ogtw bafy2bzacecfdim4e35kuhhiplbdob3mznkwcft2fsxk4ms3prymgm2twwuo7w bafy2bzaceakwjzj2nemy6nduvu45bxftmetgmmw5xgsqjgy4xc5b3jzsgo5ac bafy2bzacebsmfbretigby4lwx4yops54cqptwifdtlqywc7ft62wvzq25stcy bafy2bzaceagamhhd3azaqzf4yrw2adqeuphfsnxchzz3ag2ye5ren4vsnlqzc]
2022-06-28T21:10:30.379+1000    DEBUG   splitstore      splitstore/splitstore_reify.go:160      reification aborted; reify limit reached
2022-06-28T21:10:30.517+1000    DEBUG   splitstore      splitstore/splitstore_reify.go:160      reification aborted; reify limit reached

SBudo Jun 28, 2022

To keep the issue documented here, I'm now getting this warning:
2022-06-28T21:34:47.537+1000 WARN splitstore splitstore/splitstore_reify.go:154 missing reference while reifying bafy2bzacedjxsikshohl3n6mei6dngdg7sxhfzzxcssyau76la7hkg6ngzl7q: bafy2bzaceaz7pyra5bz4peyffmfkyh4kkljgqoyp2wazoei5katjn7wu7ky2w

By the looks of it, I'm getting a warning for each message in each block

TippyFlitsUK · 2022-06-28T12:50:44Z

TippyFlitsUK
Jun 28, 2022
Maintainer Author

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your ./lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

Directory	Current Size in GiBs
/.lotus/datastore/splitstore
/.lotus/datastore/chain

0 replies

tmyuu · 2022-06-29T22:02:34Z

tmyuu
Jun 29, 2022

Daemon:  1.16.0-rc3+mainnet+git.824da5ea5+api1.5.0
Local: lotus version 1.16.0-rc3+mainnet+git.824da5ea5

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your ./lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

Directory	Current Size in GiBs
/.lotus/datastore/splitstore	141GB
/.lotus/datastore/chain	77GB

0 replies

benjaminh83 · 2022-07-01T07:17:52Z

benjaminh83
Jul 1, 2022

Daemon:  1.16.0+mainnet+git.01254ab32+api1.5.0
Local: lotus-miner version 1.16.0+mainnet+git.01254ab32

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your ./lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

Directory	Current Size in GiBs
/.lotus/datastore/splitstore	88GB
/.lotus/datastore/chain	91GB

2 replies

benjaminh83 Jul 3, 2022

Looks like these ones are growing. Shouldn't they stabilise @TippyFlitsUK ?

Directory	Current Size in GiBs
/.lotus/datastore/splitstore	131GB
/.lotus/datastore/chain	145GB

TippyFlitsUK Jul 3, 2022
Maintainer Author

Hey Ben

The chain folder will grow a small amount over the first few days so I wouldn't be too concerned about the 145GiB. Mine is currently 135GiB. It won't grow too much more than this in the long term.

The splitstore folder will vary in size over time but should stay within a range. My observed range over the last few months has been around 95GiB to 250GiB. The default garbage collection happens about once per week so you should see your 131GiB increase a bit over the next few days before returning to nearer 100GiB. This cycle then repeats on an ongoing basis.

Thanks for posting your results buddy!!

Angelo-gh3990 · 2022-07-25T08:43:14Z

Angelo-gh3990
Jul 25, 2022

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your ./lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

Directory Current Size in GiBs
/.lotus/datastore/splitstore : 201GB
/.lotus/datastore/chain : 81GB

0 replies

SPX SplitStoreV2 - Testing & Call Outs #8927

Uh oh!

Uh oh!

TippyFlitsUK Jun 27, 2022 Maintainer

Hello SPX Fellows

Reply card:

Reminders and Risk Mitigations

Replies: 6 comments · 6 replies

Uh oh!

Uh oh!

Calibnet (t029562)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mainnet

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TippyFlitsUK Jun 28, 2022 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TippyFlitsUK Jul 3, 2022 Maintainer Author

Uh oh!

TippyFlitsUK
Jun 27, 2022
Maintainer

Replies: 6 comments 6 replies

TippyFlitsUK
Jun 28, 2022
Maintainer Author

TippyFlitsUK Jul 3, 2022
Maintainer Author