How to properly reuse free space after clear?

[timotejroiko]

Hi, i have a couple questions about reclaiming free space:

My workload involves generating a dataset of about 500k records which results in ~100mb file size when stored in lmdb. Occasionally i need to update the dataset, and in certain cases it has to be done while the application that uses it is still running.

I tried doing this by running a transaction(() => { clearAsync(); put() 500k times; }) from a separate process, but by doing this, the file grows to ~200mb, and continues to increase by 100mb every time the update script runs.

Is there a proper way to make lmdb priorize reusing free space instead of growing the file, especially when writing after clearing?

these are my current settings:

lmdb.open({
    path: ...,
    noSync: false,
    noMemInit: true,
    remapChunks: true,
    useWritemap: false,
    sharedStructuresKey: Symbol.for("structures"),
    cache: true
});

[kriszyp]

You can try setting the maxFreeSpaceToLoad and maxFreeSpaceToRetain to higher settings:

lmdb.open({
    path: ...,
    maxFreeSpaceToLoad: 100000, // default is 50000
    maxFreeSpaceToRetain: 200000, // default is 75000

However, LMDB can have a difficult fully reusing free space since it doesn't have any defragmentation capability, so it is inherently prone to fragmentation with large records like these.

[kriszyp]

I guess I should also note that you should make sure you are awaiting the clearAsync() call, in case you aren't.

[timotejroiko]

You can try setting the maxFreeSpaceToLoad and maxFreeSpaceToRetain to higher settings:

lmdb.open({
    path: ...,
    maxFreeSpaceToLoad: 100000, // default is 50000
    maxFreeSpaceToRetain: 200000, // default is 75000

However, LMDB can have a difficult fully reusing free space since it doesn't have any defragmentation capability, so it is inherently prone to fragmentation with large records like these.

Hey, thanks for the reply, i am currently testing without the main process running, just running the update script by itself. The file grows to approx. 300mb then it stops growing, it seems like lmdb starts reusing the cleared free space after the third re-write.

I tried experimenting with those options, doesn't seem to make any difference. I am awaiting the clearAsync inside the transaction but not awaiting the put's, like this: await db.transaction(async () => { await db.clearAsync(); for(...) { db.put(...) } })

Could it be related to the transaction having to reserve enough space to write all entries before committing the operations? Thinking about it now it does make sense that it needs to keep the old data until the transaction commits, and it needs an equal amount of space to commit the new data simultaneously, plus some overhead i imagine.

Here's how the runs look like:

1st run: file size = 97202176
2nd run: file size = 194592768
3rd run: file size = 291983360
4th run: file size = 292179968

[timotejroiko]

I ran another test, this time instead of using clearAsync(), i am explicitly deleting only keys that need to be deleted, and overwriting the rest of the keys, like this:

await db.transaction(() => {
    for(const key of db.getKeys()) {
        if(!updateddata[key]) { db.remove(key) }
    }
    for(const [key, val] of Object.entries(updateddata)) {
        db.put(key, val);
    }
})

The result is exactly the same as before, the file grows for the first three runs, to about 290mb, and then stops growing. Even if there are no actual removals, rewriting existing keys with put still causes the file to grow up to 3x the size before it stops. At this point i'm just interested in figuring out an explanation for this behavior.