$$\ $$\ $$\ $$\ $$\ $$\ $$\
$$$\ $$$ | \__|$$ |\__| $$ | $$ |
$$$$\ $$$$ | $$$$$$\ $$\ $$ |$$\ $$$$$$$\ $$$$$$\ $$$$$$\ $$$$$$\ $$$$$$$\ $$$$$$$\ $$$$$$$\ $$$$$$\ $$$$$$$\ $$$$$$$\ $$$$$$\ $$$$$$$\ $$$$$$\ $$$$$$\ $$$$$$\
$$\$$\$$ $$ |$$ __$$\ $$ |$$ |$$ |$$ _____|$$ __$$\ \____$$\ $$ __$$\ $$ _____|$$ __$$\ $$$$$$\ $$ _____|$$ __$$\ $$ __$$\ $$ __$$\ $$ __$$\ $$ _____|\_$$ _| $$ __$$\ $$ __$$\
$$ \$$$ $$ |$$$$$$$$ |$$ |$$ |$$ |\$$$$$$\ $$$$$$$$ | $$$$$$$ |$$ | \__|$$ / $$ | $$ |\______|$$ / $$ / $$ |$$ | $$ |$$ | $$ |$$$$$$$$ |$$ / $$ | $$$$$$$$ |$$ | \__|
$$ |\$ /$$ |$$ ____|$$ |$$ |$$ | \____$$\ $$ ____|$$ __$$ |$$ | $$ | $$ | $$ | $$ | $$ | $$ |$$ | $$ |$$ | $$ |$$ ____|$$ | $$ |$$\ $$ ____|$$ |
$$ | \_/ $$ |\$$$$$$$\ $$ |$$ |$$ |$$$$$$$ |\$$$$$$$\ \$$$$$$$ |$$ | \$$$$$$$\ $$ | $$ | \$$$$$$$\ \$$$$$$ |$$ | $$ |$$ | $$ |\$$$$$$$\ \$$$$$$$\ \$$$$ |\$$$$$$$\ $$ |
\__| \__| \_______|\__|\__|\__|\_______/ \_______| \_______|\__| \_______|\__| \__| \_______| \______/ \__| \__|\__| \__| \_______| \_______| \____/ \_______|\__|
Version: 0.1
Author: dattd
- Synchronize data from your data source to Meilisearch.
- Perform change data capture (CDC) to remain data consistency.
- Refresh data in case of inconsistency between your data source and Meilisearch instance using Meilisearch's swap index feature (performing cost no downtime).
- Show consistent status between your datasource and Meilisearch instance (record wise).
- Config search policy of each index. For more documentation please refer to official Meilisearch documentation here .
Synchronize your datasource (currently support PostgresSQL) with Meilisearch instance.
Requirement:
- Support Postgres version: above version 12
-
Plugin wal2json must be installed associate with Postgres version on your local machine:
sudo yum install wal2json14
sudo apt-get install postgresql-14-wal2json
brew install wal2json
- Make sure to config Postgres wal_level and have enough replication slot:
wal_level = logical
#
# these parameters only need to set in versions 9.4, 9.5 and 9.6
# default values are ok in version 10 or later
#
# each table config to sync with Meilisearch required 1 replication slot.
max_replication_slots = 10
max_wal_senders = 10
Make sure to cover core concepts of Meilisearch such as index, document, search policy and more at: https://www.meilisearch.com/docs
cargo build --release
- Show help
- Sync data and perform CDC
cargo run -- --config <path> sync
- Check for consistency
cargo run -- --config <path> status
- Refresh data by swap index
cargo run -- --config <path> refresh
cargo run -- -h
meilisearch:
apiUrl: http://localhost:7700
adminApiKey: "4d210c85a27082683758d083d6c1298ee7e1e97385edeaad3250b9cd8aebd276"
dataSource:
sourceType: Postgres
host: localhost
port: 5432
username: dattran
password: dattran
database: postgres
synchronizeTables:
- indexName: users
primaryKey: user_id
limit: 1000
syncFields:
- user_id
- username
- created_at
meiliSetting:
displayedAttributes:
- user_id
- username
searchableAttributes:
- username
- indexName: employees
primaryKey: employee_id
limit: 10
syncFields:
- employee_id
- first_name
* Note: This is a POC and still in development process. Base source can change drastically in the future, please consider to use in production environment with real data.