Send and retrieve dumps from an S3 store

[Kerollmops]

Bonjour Guillaume,

After a meeting with the @meilisearch/cloud-team about what would be the best/easiest way to update a Meilisearch server, we bring the idea of the engine being able to send and retrieve dumps from an S3 store.

When it comes to updating a Meilisearch server:

• you first need to create a dump, you must specify a path on the file system and allocate more space for it to fit on the disk
• you then need to create a new machine with a volume that has enough space for the dump and the final database
• once your machine with a bigger disk is up, you tranfert the dump and update a Meilisearch with it
• once the new Meilisearch server is up-to-date, you kill the previous one

This solution is cumbersome for the Cloud team and not really reliable in the sense that the disk space will always grow, you will always need more space from update to update it is because when you decide on the size of the new disk space you take the previous disk space and add a little bit more space for the dump to fit, but the new version has now more space (as you can't shrink the disk size). Doing a new update will force the new disk space to be even bigger than previously.

We imagined a basic solution where Meilisearch would be able to support sending and retrieving dumps from an S3 store. To make it simple: you do not need to change the size of the disk as the dumps are no more stored on it, the only thing you need is a read/write access to an S3 store and be able to specify it to the Meilisearch server.

---

We decided that we would try to mount a virtual disk based on an S3 store where we upload the dump to the S3 store once it is finished and read it from the S3 store from the new machine (with the default disk size) by using a virtual network disk using s3fs-fuse or equivalent.

[gmourier]

Hi @Kerollmops

I think I understand!

Is it not possible to point out the /dumps folder to S3FS directly without modifying the engine? I imagine that whatever happens, you have to be able to stream the dump because it will have to be stored locally before sending it to S3.

If I go back to the steps, which ones would be eliminated with this solution?

• You first need to create a dump, you must specify a path on the file system and allocate more space for it to fit on the disk; the idea here is to eliminate the extra disk allocation, right? For that, we need to be able to define an S3 endpoint instead of the /dumps path.
• You then need to create a new machine; There is still a need to spawn a new instance to have 0 downtimes, do we agree?
• Once your machine with a bigger disk is up, you tranfert the dump and update a Meilisearch with it; At that point, the simplicity would be able to write something like this, --import-dump https://my-dump-bucket.s3.us-west-2.meilisearch.com/20200813-042312213.dump right? Is it necessary to be able to read and process the file in streaming? Or do we just want to read the dump from S3 in its entirety and then load it? Will the size of the disks allocated at the beginning allow for this?
• once the new Meilisearch server is up-to-date, you kill the previous one; Nothing changes here if I'm not mistaken

If there is no need to stream the file, isn't there a way to do this at the machine level more than at Meilisearch level?

Thanks!

[Kerollmops]

Hey,

Is it not possible to point out the /dumps folder to S3FS directly without modifying the engine? I imagine that whatever happens, you have to be able to stream the dump because it will have to be stored locally before sending it to S3.

Absolutely, doing this is correct but maybe a little bit too custom. All of the listed steps you describe would be eliminated, we just found that, maybe, that could be the job of Meilisearch to store and write the dumps in an S3 store by itself instead of using a virtual network disk.

If there is no need to stream the file, isn't there a way to do this at the machine level more than at Meilisearch level?

We need to be able to stream the dump files from somewhere else to be sure that the dump doesn't take up space on the disk of the new machine, the s3fs solution described at the end of my main post does that by using a network file system.

[NoKriK]

Hi there,

Yann from Koyeb here.
We are looking for an efficient solution to recommend for Meilisearch data persistence and horizontal scaling on our Serverless platform.

From our understanding, a simple solution to handle data persistence, auto-healing, and horizontal scaling could be to rely on Meilisearch snapshots and S3-compatible object storage to distribute these snapshots.

This would enable highly resilient and performant deployments with horizontal scaling for use-cases that do not require real-time index updates:

• this would require a "main" Meilisearch instance receiving index updates and performing snapshots which would be automatically stored on object storage
• you could have plenty of autoscaling "follower instances", which would load the latest snapshot from the object storage at boot
• a re-deployment could be triggered each time a new snapshot is generated or on a time basis to update all Meilisearch instances (on Koyeb we provide natively Blue-Green deployments, so this is a zero-downtime operation)

This would be a highly scalable solution as long as you can afford a few minutes delay on the index update propagation.

I might have missed something, so let me know if that doesn't make sense.

[Kerollmops]

Hey @NoKriK,

Sorry for the very late reply. I would say that you didn't miss anything important here, doing this looks quite good in fact. The only thing that I can see is that doing a snapshot require disk space and this is mandatory I am not sure that we can change the engine in a way that would make the snapshot export to be streamable, but maybe you don't care about that.

Thank you very much,
Have a nice day!

[matthijs166]

Would be an awesome feature than we can deploy meilisearch next to our cloud deployments

[macraig]

Hi @matthijs166 , can you expand a bit more on your needs? What would you be hoping to achieve with this feature?

[matthijs166]

In a containerized environment without stateful storage (e.g., deployed on DigitalOcean App Platform), we need to store indexes in an S3 bucket to ensure they persist across deployments. This allows indexes to be restored from backups instead of requiring a full reindex, which is currently necessary due to data loss. Additionally, we aim to run this process inside a Docker container to facilitate configuration updates and local testing.

[Kerollmops]

Hey @matthijs166 👋

Thank you for the precision. However, you mix dumps, snapshots, and the database itself. Using an S3 (like S3FS) to store the database will be too slow. Meilisearch requires a fast disk, preferably fast NVMe, to access the internal databases quickly. The indexation and search will be infinitely slow using a network-based disk due to all the network latency.

However, you could store dumps in S3. This way, you keep a good enough save of your data. You can trigger the snapshot creation every now and then. We also plan to speedup the creation of dumps and snapshots that could be particularly slow today.