500 GB of metadata

[RobQuistNL]

Checklist

• This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to boost.

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• Other

Boost Version

2.4.0

Describe the Bug

Massive amounts of files (300k) of ~2.1MB each in ~/.boost/datastore/metadata

Some of these files are back from 2022. Shouldn't this metadata be cleaned up? What is it from?

Logging Information

-

Repo Steps

1. Run boost
2. See disk usage

[LexLuthr]

This is the main levelDB. I am not sure why it is so large in your case but there is no clean up mechanism for it. Mostly likely suspect would be index-provider if you have large cache size for index-provider.

[RobQuistNL]

I've been a miner since before the spacerace and have 500.000+ legacy deals - some of those have a lot of data in them. Could that be it?

[LexLuthr]

That would still not explain the size. It is hard to guess what is taking up the space without access to the DB. You can try DB compaction. But ensure that Boostd is not running when you do this. Or you will need to check what kind of keys are the most.

[RobQuistNL]

Any debug tools / docs that I can use to explore that DB? Any query results you're interested in seeing?

[LexLuthr]

I use https://github.com/cions/leveldb-cli but it is limited. Also, can you give me the config.toml values for EntriesCacheCapacity? There is no specific commands that we can use to determine the size. This would mostly be trying to find what kind of keys are occupying the most space.

[RobQuistNL]

[IndexProvider]
  #EntriesCacheCapacity = 1024
  #EntriesChunkSize = 16384

so that's all at default values.

[LexLuthr]

In that case, you need to check what in DB is occupying the space.

[RobQuistNL]

The CLI app doesn't really seem to run;

~/leveldb-cli/cmd/leveldb$ go run . show
leveldb: error: file does not exist
exit status 1

how do i best read out contents of the DB and what would be interesting to look at?

[LexLuthr]

Use the directory flag. It is a simple KV DB so everything is in one place. "/index-provider" would be my first suspect but hard to guess. Only way to know is to look at the data.