Allow to mark vectors as "userProvided" #4633

dureuill · 2024-05-14T12:51:51Z

Pull Request

Related issue

Fixes #4606

What does this PR do?

See usage in PRD

Extends the shape of the special _vectors field in documents.
- previously, the _vectors field had to be an object, with each field the name of a configured embedder, and each value either null, an embedding (array of numbers), or an array of embeddings.
- In this PR, the value of an embedder in the _vectors field can additionally be an object. The object has two fields:
  1. embeddings: null, an embedding (array of numbers), or an array of embeddings.
  2. userProvided: a boolean indicating if the vector was provided by the user.
- The previous form embedder_or_array_of_embedders is semantically equivalent to:
```
{
    "embeddings": embedder_or_array_of_embedders,
    "userProvided": true
}
```
During the indexing step, the subfields and values of the _vectors field that have userProvided set to false are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template.
This allows importing embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change.
The dump process now uses this ability: it enriches the _vectors fields of documents with the embeddings that were autogenerated, marking them as not userProvided. This allows importing the vectors from a dump without regenerating them.

Tests

This PR adds the following tests

Long-needed hybrid search tests of a simple hf embedder
Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content.
Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected

- use the parsed_vectors module - only parse `_vectors` once per document, instead of once per embedder per document

dureuill · 2024-05-20T09:12:31Z

Looks like the vectors in the snapshots are different from those computed in the CI... Investigating, but we might need a different way to check here.

dureuill · 2024-05-21T10:00:06Z

I made sure the behavior when not passing embeddings in a document update (but only userProvided): it is the same as passing embeddings: null, which results in putting 0 vectors into arroy.

milli/src/error.rs

milli/src/index.rs

milli/src/update/index_documents/extract/extract_vector_points.rs

milli/src/vector/parsed_vectors.rs

milli/src/update/index_documents/typed_chunk.rs

milli/src/vector/parsed_vectors.rs

Kerollmops

Thank you very much @dureuill. LGTM 🍸
bors merge

4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <louis@meilisearch.com>

Kerollmops · 2024-05-22T12:38:02Z

bors cancel

meili-bors · 2024-05-22T12:38:05Z

Canceled.

Kerollmops · 2024-05-22T12:45:13Z

bors merge

4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <louis@meilisearch.com>

meili-bors · 2024-05-22T13:22:20Z

This PR was included in a batch that successfully built, but then failed to merge into main. It will not be retried.

Additional information:

{"message":"At least 1 approving review is required by reviewers with write access.","documentation_url":"https://docs.github.com/articles/about-protected-branches"}

Kerollmops

bors merge

4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <louis@meilisearch.com>

meili-bors · 2024-05-22T16:30:21Z

Build failed:

Tests on windows-2022

Kerollmops · 2024-05-22T16:34:58Z

bors merge

4633: Allow to mark vectors as "userProvided" r=Kerollmops a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <louis@meilisearch.com>

meili-bors · 2024-05-22T17:03:21Z

Build failed:

Tests on windows-2022

dureuill · 2024-05-22T20:50:21Z

bors merge

4633: Allow to mark vectors as "userProvided" r=dureuill a=dureuill # Pull Request ## Related issue Fixes #4606 ## What does this PR do? [See usage in PRD](https://meilisearch.notion.site/v1-9-AI-search-changes-e90d6803eca8417aa70a1ac5d0225697#deb96fb0595947bda7d4a371100326eb) - Extends the shape of the special `_vectors` field in documents. - previously, the `_vectors` field had to be an object, with each field the name of a configured embedder, and each value either `null`, an embedding (array of numbers), or an array of embeddings. - In this PR, the value of an embedder in the `_vectors` field can additionally be an object. The object has two fields: 1. `embeddings`: `null`, an embedding (array of numbers), or an array of embeddings. 2. `userProvided`: a boolean indicating if the vector was provided by the user. - The previous form `embedder_or_array_of_embedders` is semantically equivalent to: ```json { "embeddings": embedder_or_array_of_embedders, "userProvided": true } ``` - During the indexing step, the subfields and values of the `_vectors` field that have `userProvided` set to **false** are added in the vector DB, but not in the documents DB: that means that future modifications of the documents will trigger a regeneration of that particular vector using the document template. - This allows **importing** embeddings as a one-shot process, while still retaining the ability to regenerate embeddings on document change. - The dump process now uses this ability: it enriches the `_vectors` fields of documents with the embeddings that were autogenerated, marking them as not `userProvided`. This allows importing the vectors from a dump without regenerating them. ### Tests This PR adds the following tests - Long-needed hybrid search tests of a simple hf embedder - Dump test that imports vectors. Due to the difficulty of actually importing a dump in tests, we just read the dump and check it contains the expected content. - Tests in the index-scheduler: this tests that documents containing the same kind of instructions as in the dump indexes as expected Co-authored-by: Louis Dureuil <louis@meilisearch.com>

meili-bors · 2024-05-22T21:27:13Z

Build failed:

Tests on windows-2022

Kerollmops · 2024-05-23T08:17:36Z

bors merge

meili-bors · 2024-05-23T08:57:05Z

Build succeeded:

Kerollmops marked this pull request as draft May 15, 2024 15:19

Add parsed vectors module

98c8112

dureuill force-pushed the dont-regenerate-vecs-in-dump branch from 48776ca to 8a123a4 Compare May 20, 2024 08:35

dureuill added 7 commits May 20, 2024 10:36

Add function to get the embeddings of a document in an index

261de88

Refactor vector indexing

52d9cb6

- use the parsed_vectors module - only parse `_vectors` once per document, instead of once per embedder per document

Add vectors from vector DB in dump

02714ef

Don't write vectors that weren't autogenerated in document DB

2f7a8a4

Don't write an empty _vectors field

0462ebb

Fix tests

d05d49f

Add test with a dump

30cf972

dureuill force-pushed the dont-regenerate-vecs-in-dump branch from 8a123a4 to 1c7f252 Compare May 20, 2024 08:36

dureuill marked this pull request as ready for review May 20, 2024 08:37

dureuill force-pushed the dont-regenerate-vecs-in-dump branch from 1c7f252 to f25d590 Compare May 20, 2024 08:58

dureuill force-pushed the dont-regenerate-vecs-in-dump branch from f25d590 to 96e0de9 Compare May 20, 2024 12:22

dureuill added 3 commits May 20, 2024 14:44

Test hybrid search with hf embedder

afcd7b9

Test array of vectors

b17cb56

Add test on index-scheduler

9969f7a

dureuill force-pushed the dont-regenerate-vecs-in-dump branch from 96e0de9 to 9969f7a Compare May 20, 2024 12:44

dureuill added this to the v1.9.0 milestone May 20, 2024

dureuill requested a review from Kerollmops May 20, 2024 13:15

dureuill mentioned this pull request May 20, 2024

Only ever store vectors in the vector store + hide embeddings #4607

Open

8 tasks

Increase index-scheduler test timeouts

eccbcf5

Kerollmops requested changes May 22, 2024

View reviewed changes

dureuill added 3 commits May 22, 2024 12:23

Remove now unused error variant

8f7c8ca

Don't remove embedders that are not in the config from the document DB

16037e2

"[]" is deserialized as 0 embedding rather than 1 embedding of dim 0

3412e7f

dureuill requested a review from Kerollmops May 22, 2024 12:16

Kerollmops previously approved these changes May 22, 2024

View reviewed changes

Smaller review changes

8a941c0

dureuill dismissed Kerollmops’s stale review via 8a941c0 May 22, 2024 12:44

dureuill force-pushed the dont-regenerate-vecs-in-dump branch from 6ef21c7 to 8a941c0 Compare May 22, 2024 12:44

Kerollmops approved these changes May 22, 2024

View reviewed changes

meili-bors bot merged commit 14bc80e into main May 23, 2024
8 of 10 checks passed

meili-bors bot deleted the dont-regenerate-vecs-in-dump branch May 23, 2024 08:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow to mark vectors as "userProvided" #4633

Allow to mark vectors as "userProvided" #4633

dureuill commented May 14, 2024 •

edited

dureuill commented May 20, 2024 •

edited

dureuill commented May 21, 2024

Kerollmops left a comment

Kerollmops commented May 22, 2024

meili-bors bot commented May 22, 2024

Kerollmops commented May 22, 2024

meili-bors bot commented May 22, 2024

Kerollmops left a comment

meili-bors bot commented May 22, 2024

Kerollmops commented May 22, 2024

meili-bors bot commented May 22, 2024

dureuill commented May 22, 2024

meili-bors bot commented May 22, 2024

Kerollmops commented May 23, 2024

meili-bors bot commented May 23, 2024

Allow to mark vectors as "userProvided" #4633

Allow to mark vectors as "userProvided" #4633

Conversation

dureuill commented May 14, 2024 • edited

Pull Request

Related issue

What does this PR do?

Tests

dureuill commented May 20, 2024 • edited

dureuill commented May 21, 2024

Kerollmops left a comment

Choose a reason for hiding this comment

Kerollmops commented May 22, 2024

meili-bors bot commented May 22, 2024

Kerollmops commented May 22, 2024

meili-bors bot commented May 22, 2024

Kerollmops left a comment

Choose a reason for hiding this comment

meili-bors bot commented May 22, 2024

Kerollmops commented May 22, 2024

meili-bors bot commented May 22, 2024

dureuill commented May 22, 2024

meili-bors bot commented May 22, 2024

Kerollmops commented May 23, 2024

meili-bors bot commented May 23, 2024

dureuill commented May 14, 2024 •

edited

dureuill commented May 20, 2024 •

edited