Thanks for the write-up kero!

Cc @gmourier

I am sharing this here for someone stumbling up on that issue. We have a dedicated product discussion to let us know what you need.

Here are some news about what we plan to do in the coming months.

We worked with @dureuill to create a fake Meilisearch program that behaves like Meilisearch, i.e., with a task queue and everything. This little experiment will run on top of datafuselabs/openraft. We followed the guide and wanted to make it work with LMDB and its commit and transactional systems.

We use this fake Meilisearch experiment to meet the Cloud team requirements and adapt the solution to something useful that can be easily deployed and maintained. Once everyone is happy with it, we will transpose the changes on Meilisearch itself. It's easier to work on a small codebase first to experiment than a code as bog as the Meilisearch one.

More news 🎉

We discussed the current implementation, and @irevoire and @ManyTheFish thought about the potential solution of not sending the whole update files through the Raft log. It would be too big. They think we can make the data.ms/update_files folder an S3fs-backed directory for all members.

When a new update arrives, the leader stores the file in this folder and sends the task info through the Raft log. The followers will then store the raw tasks with the same date-times as the leader (for task filters to be consistent across the cluster). We can use the register_raw_task function. Other members of the cluster will be able to read the file from there. We must only delete the file once the associated Raft log has been wiped out.

The mDNS communication system is relatively easy to implement. However, @irevoire suggested using the Quickwit/ChitChat solution.

We must commit at the right time and ensure that errors while writing on disk are well handled.

Meeting notes and potential next steps.

1. create a new task
   • leader creates a task
   • leader writes heavy files in a synchronized file system
   • leader sends the task skeleton
2. receive and commit task
   • followers receive the task and commit independently
3. create a batch
   • leader computes the task ids of the next batch to process
   • leader sends the batch's task ids to everybody
4. receive and process batch
   • followers receive the task ids and create the batch
   • followers process the batch
   • synchronize commit? (-> load a snapshot on failure)
   • document addition that needs an index creation?
   • behavior when every node fails?
5. snapshots of failures
   • write snapshot files in a synchronized file system
   • send a snapshot with a path
   • how to auto-clean old snapshots?
6. other
   • How to test? (https://github.com/mininet/mininet?)
   • API keys?

Main TODO:

• Index and process batch
• Ensure that the snapshot process works
  • when a machine fall
  • when a new machine is added to the cluster
• Ask for help with the consistency of the cluster at the end of an indexing process (Corum check?)
• auto-discovery (mdns)
• Ability to exclude a node when it can't resynchronize with the cluster
• if a node responds "BAD REQUEST" it should be excluded from the cluster

Hi there, I have a little bit of experience using Openraft. I had a similar ideia of using something else to distribute the snapshot datafuselabs/openraft#507, in this issue I talked with the author of Openraft. So if you guys want to take a look.

Also I'm free to help if needed, I know a little bit how Meilisearch deals with tasks and the task queue

Hello, I come back on the replication solution, Raft doesn't seem to be the best approach we found so far, to remind our must-haves in terms of replication:

• Meilisearch should be highly available on READ
• Meilisearch should not have any single point of failure
• Meilisearch should be geo-replicated

With the current Meilisearch architecture, RAFT doesn't play well with:

• geo-replication, which is a must-have
• Two-phase commit, needed to ensure indexing consistency
• An higher complexity for few benefits

A potential alternative would be to rely on Zookeeper to handle the leader election and the order consistency of the task to process and on S3 to store and propagates the updates and snapshots. Note that the S3 is necessary with a raft architecture too. This solution seems the easiest to implement, seems promising for a future HA on WRITE operations, and seems to allow having bigger machines specialized for indexing and smaller machines for Search.

Before starting to implement a POC, we should ensure the following needs:

• ensure the Meilisearch cloud infrastructure can handle a zookeeper (seems possible)
• ensure zookeeper and S3 are well geo-replicated and consistent
• prepare an architectural plan for the final solution
• provide a time estimation for each step of the solution

I would strongly vote for etcd over zookeeper if you're going to run a lockserver. A good majority of installations are going to happen on kubernetes, which runs on etcd, ensuring long term support. Zookeeper has stagnated for years.

Hello @derekperkins,
thank you for your suggestion, we will investigate the etcd solution too and see what is more suitable for us, we keep experimenting with Zookeeper because it is one of the most battle-tested solutions in terms of replication despite the stagnation.
See you!

zookeeper was definitely the original choice, but many popular datastores are eliminating it. etcd, mainly because it is the primary datastore for kubernetes, is easier to run in a cloud-native way, and is more likely to meet the needs for meilisearch for the next decade.

• https://www.infoworld.com/article/3660073/why-apache-kafka-is-dropping-zookeeper-for-kraft.html
• https://clickhouse.com/docs/knowledgebase/why_recommend_clickhouse_keeper_over_zookeeper

@ManyTheFish @derekperkins there is a paper discussing the use o etcd used by kubernetes to build a native consensus algorithm in kubernetes. paper

FWIW, I'm a maintainer of Vitess, distributed MySQL. We initially had an interface to enable any lockserver, and we had implementations for etcd, zookeeper, consul, and k8s. k8s and consul have already been deprecated, and we're really only keeping zookeeper around because of existing users. If we were starting over today, we would most likely require etcd. Starting with Zookeeper in a new project would be a mistake IMO.

Hey, I would love to hear more about what is wrong with Zookeeper; I’m struggling to understand the concrete issues with it?
The main resource I’ve read was the Jepsen analysis which was really positive https://aphyr.com/posts/291-call-me-maybe-zookeeper, and most people seem to be leaving Zookeeper for a custom solution crafted strictly for their use case (which we don’t have the time to develop right now).

Also, I looked a bit at etcd I think it fulfills our requirement, but it seems harder to use and deploy. I’m not especially against it, I would just like to understand better what’s the point in moving to etcd.

Also, I’ve heard a bunch of times that basic raft doesn’t play well with geo-replication. Since etcd relies on raft, do you have info to share on the subject?

Hey, I would love to hear more about what is wrong with Zookeeper; I’m struggling to understand the concrete issues with it?

There's nothing concretely wrong with Zookeeper. IIRC, we have encountered a few types of queries that were less performant in zookeeper, but nothing major.

most people seem to be leaving Zookeeper for a custom solution

This is my core concern. Zookeeper was already stagnating while it was required for many popular projects, and all of these migrations are just going to accelerate that. I'm not pitching for Meilisearch to write their own custom implementation, just to use etcd that by virtue of being the only datastore for kubernetes, is more likely to be around longer than zookeeper. To be fair, etcd is currently looking for some maintenance help, but again, the kubernetes connection makes it impossible to abandon.

I looked a bit at etcd I think it fulfills our requirement, but it seems harder to use and deploy

I guess that's subjective, I've had the opposite experience, though it admittedly has been a few years since I tried zookeeper. Bitnami maintains charts for both: etcd & zookeeper. I have experience with the etcd chart and it's really easy to deploy. Given the success I've had with it and some trust in bitnami, I'd guess that zookeeper is similarly easy to stand up.

Zookeeper runs in Java, and takes more resources to run than etcd written in Go does. Zookeeper was written pre-cloud, where etcd was built in a cloud-native era, which is very hand-wavy and not objective, but plays a role in my opinion.

Planetscale runs on Vitess, using etcd. This is a quote from someone on their team about how little it takes for them to manage it at scale.

I’ve heard a bunch of times that basic raft doesn’t play well with geo-replication

I'm not the expert in this sphere, but raft has been adopted by most of the NewSQL competitors, the two I mentioned above migrating away from zookeeper, Clickhouse and Kafka, and your closest competitor typesense:

• https://www.cockroachlabs.com/blog/raft-is-so-fetch/
• https://www.pingcap.com/blog/raft-and-high-availability/
• https://www.yugabyte.com/tech/raft-consensus-algorithm/
• https://developer.confluent.io/learn/kraft/
• https://clickhouse.com/docs/en/guides/sre/keeper/clickhouse-keeper
• https://typesense.org/docs/guide/high-availability.html

I don't think the choice of consensus algorithm is a reason to choose one over the other.

I would just like to understand better what’s the point in moving to etcd

At least from what is visible on this thread, the discussion didn't move to zookeeper until 2 days ago. If zookeeper was already built out and in place, I wouldn't suggest to anyone to do a rip and replace. My comments are based on the assumption that this is a pretty green field project.

Again, I'm not an expert in the weeds of zookeeper vs etcd, so I don't claim to 100% know the right choice for Meilisearch. I'm making some intuitive projections based on what I'm seeing in the distributed database world. As a ~2 year satisfied user of Meilisearch continuing to expand our usage, I want to see what's best for the project. If I have a specific bias, it's mostly based on operating etcd for several years in conjunction with our Vitess installation, all running on top of etcd in Kubernetes.

TL;DR: Zookeeper is probably fine, but choosing it now goes against trends in other distributed database systems

Hey, thanks a lot for your comment that helps a lot!

I'm not the expert in this sphere, but raft has been adopted by most of the NewSQL competitors, the two I mentioned above migrating away from zookeeper, Clickhouse and Kafka, and your closest competitor typesense:

The question was more « Is it ok to run instances all over the world in different dc »? Because a lot of people complain that raft is way too verbose and costs a lot of money. For example, afaik typesense isn’t geo-replicated in their cloud.
Kafka is, but they built a custom raft specifically for Kafka.

So, if you’re using etcd a lot I was just wondering how it is deployed and if it’s a good idea to have it synchronize for example one instance in Paris and one in Tokyo.

At least from what is visible on this thread, the discussion didn't move to zookeeper until 2 days ago. If zookeeper was already built out and in place, I wouldn't suggest to anyone to do a rip and replace. My comments are based on the assumption that this is a pretty green field project.

To keep it short, let’s say that the HA is a common effort with the meilisearch cloud team and thus we can’t swap our solution every week or no one is going to progress. But ideally, we would really like to take the time to try out etcd and probably other stuff at some point.

You also said you dropped support for consul, I see that it has been proposed to us as an alternative to zookeeper, do you have any info to share about this one as well? 👀

The question was more « Is it ok to run instances all over the world in different dc »? Because a lot of people complain that raft is way too verbose and costs a lot of money

My answer is the same as before - almost every new database system is using raft for running clusters around the world, so it's hard for me to believe that there's a fundamental problem with the algorithm. Vague references to "lots of people complain" don't help identify any specific issues you're facing.

if you’re using etcd a lot I was just wondering how it is deployed and if it’s a good idea to have it synchronize for example one instance in Paris and one in Tokyo.

The way that Vitess uses etcd / zookeeper is fundamentally different than what you're describing here. They aren't a part of the query serving path, they are only used for specific low-bandwidth operations like failovers. Actual data replication is handled by MySQL.

To keep it short, let’s say that the HA is a common effort with the meilisearch cloud team and thus we can’t swap our solution every week or no one is going to progress

I only know what's happening on this thread. My only goal was to add some more context so that zookeeper was chosen for its merits and not because it used to be the only choice.

If I were in your shoes, I would reach out to the people most actively committing in these other db systems and ask their opinion, what tradeoffs they've made, and how they would approach it for a db without existing replication like meilisearch. I'm sure you could find someone willing to get on a quick call to answer some of your questions, I just don't have the topical expertise to get any deeper.

• https://github.com/tikv/raft-rs
• https://github.com/yugabyte/yugabyte-db/commits/80e5f3fdb8967ab29273e0041e8d504b40415078/src/yb/consensus

Best of luck, I'm excited to continue watching Meilisearch evolve!

Vague references to "lots of people complain" don't help identify any specific issues you're facing.

Generally speaking, the kind of feedback I heard was like;

• Raft sends way too many logs everywhere, which costs a lot of money when going from one dc to another, especially when considering the kind of data we need to send
• Which brings us to the « multi-raft » implementation that can synchronize multiple raft clusters with fewer messages in between cluster => in rust, for example, openraft which is way easier to use than raft-rs doesn’t support it afaik
• Once you have a multi-raft cluster, people now complain that it’s still super expensive because you need raft cluster of 3 instances replicated all over the world instead of a single machine, for example. So if you just want to geo-replicate one instance in Tokyo, Paris, and SF, for example, instead of deploying only 3 instances and having degraded performances when one goes down you must deploy 3 instances in each place for a total of 9 instances.

Team has worked on this, but the work is not over
Moving this to the next Milestone (v1.5.0) since we will continue the work on it

• Raft sends way too many logs everywhere, which costs a lot of money when going from one dc to another, especially when considering the kind of data we need to send

Are you talking about replicating all the search index data (write/update/delete) via Raft?

My understanding was that Raft was only used to coordinate replica set health. But any synchronisation of actual DB content (search index data in this case) would be handled outside Raft (usually as a stream of commands from the master node to slave nodes).

Is anyone currently working on this?

Hello @andtii
This is a work the team made in the past months (implementation design and proof of concept). But we had to delay the final implementation to work on other priorities. This is something we will start again during the year probably.

To give you more insight, since we observed replication is more an enterprise need, we want to focus on making Meilisearch replicate on our cloud offer first. However, an opensource replication solution will come once we are sure we manage to make it available on the cloud.

@curquiza Thnx for the input. At the moment we are running tests with latest meilisearch and multi instance pods against a shared nfs disk in azure kubernetes. The problem we can see is that when you create a new index on one pod the other pods will not get the new index. Is there a way to run meilisearch like this? How are you running it in the cloud at the moment to support scale out?

Hey @andtii,

Is there a way to run meilisearch like this?

No, sorry, you absolutely cannot do that;

1. meilisearch is not designed to be run multiple times on the same database.
2. Meilisearch is not designed to work on shared nfs disks; this will cause database corruption.

As curqui stated, currently, we're focused on polishing our enterprise solution first, which I don’t know if I can share details about. But basically it comes down to having a big queue that replicates all the updates to as many meilisearch instances as you want.
You must take care that the queue always sends tasks in order.

If you want, you can also give a try to the two previous open solutions I made before. But keep in mind that I only did manual tests on it. There is a good chance it’s hiding some bugs, and as far as I know, no one else ever tried it.
We don’t consider it as production ready, so run it at your own risk.
But if you do run it, giving us feedback and bug reports would really help us to design a better open-source solution in the future.

Here’s the PR: #3971

To make it work, you must deploy a zookeeper + an S3 server, which all meilisearch instances will connect to.

Any progress in this area? We looked into typesense a bit since it supports scaleout using raft but its nowhere near the performance of meilisearch from our perspective but still a showstopper since our infra needs to scale

Yes for us too: I'm loving the Melisearch technology but without replication and a way to ensure high-availability we currently can not use this in production for any of our projects.

Hello @andtii and @dedene

Thank you both of you for using and appreciating Meilisearch.
Unfortunately, so far we don't have the resources to work on pure replication system on the opensource product, sorry.

I would like to know, what do you mean by "high-availability" for you? It can be very different depending on users.

You have to know, on our Cloud offer, we ensure a minimal strong SLA, and you can even ask for more if you fall into the Entreprise plan. In this case, our team will deploy the necessary to ensure the minimal uptime you expect, so the high availability you expect.