[Question]: How to add OR metadata filter for RedisVectorStore retriever?

[dhruv2512]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

How can I add OR filter for the RedisVectorStore retriever?
I know currently only exact match is supported. But is there any workaround for this?

[dosubot[bot]]

🤖

Hello,

Currently, the LlamaIndex's RedisVectorStore retriever only supports exact match filtering as you've mentioned. This is due to the way the _to_redis_filters function is implemented, which constructs a Redis query string with the " & " operator joining each filter. This operator in Redis is equivalent to the logical AND operation, hence only exact matches are supported.

To implement an OR operation, you would need to modify the _to_redis_filters function to join the filter strings with the " | " operator instead of the " & " operator. However, this would apply the OR operation to all filters, not just specific ones.

Here's an example of how you might modify the function:

def _to_redis_filters(metadata_filters: MetadataFilters) -> str:
    tokenizer = TokenEscaper()

    filter_strings = []
    for filter in metadata_filters.filters:
        filter_string = "@%s:{%s}" % (filter.key, tokenizer.escape(str(filter.value)))
        filter_strings.append(filter_string)

    joined_filter_strings = " | ".join(filter_strings)  # changed from " & " to " | "
    return f"({joined_filter_strings})"

Please note that this is a simple workaround and might not cover all use cases. For more complex queries, you might need to implement a more sophisticated query builder that can handle different types of logical operations.

Also, please be aware that changing the function in this way will affect all queries that use the RedisVectorStore retriever, so you should thoroughly test your application after making this change to ensure it still behaves as expected.

I hope this helps! Let me know if you have any other questions.

Sources

• llama_index/vector_stores/redis.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

Yea I think the bot is correct here, there would need to be a PR to add this.

Another option is querying which each filter that you want to "or", and combine/dedup the resulting nodes

nodes = []
found_ids = set()
for filter in my_filters:
  retriever = index.as_retriever(..., filters=[filter])
  new_nodes = retriever.retrieve(query_str)
  for n in new_nodes:
    if n.node.node_id not in found_ids:
      nodes.append(n)
      found_ids.add(n.node.node_id)

[llermaly]

@logan-markewich We are seeing the same situation in the Elasticsearch Vector store.

I think we will need some kind of parameter from the MetadataFilters that says if it is an "or" or an "and"

What we are doing is assuming the keys with the same value are an OR, and then its an AND across different keys.

eg:

        ExactMatchFilter(key="artist", value="Taylor Swift"),
        ExactMatchFilter(key="artist", value="Katy Perry"),
        ExactMatchFilter(key="genre", value="pop"),

Generates (Taylor Swift OR Katy Perry) AND (pop)

Now, we need some way to say (Taylor Swift AND Katy Perry) OR (pop)

This is useful for multi valued fields (e.g multiple artists in a song)

Is it possible to do this today?

[logan-markewich]

Hmm, I think it depends on if the underlying vector store supports it.

Every vector store that uses filters would have to be updated 😅

[dhruv2512]

@logan-markewich We are seeing the same situation in the Elasticsearch Vector store.

I think we will need some kind of parameter from the MetadataFilters that says if it is an "or" or an "and"

What we are doing is assuming the keys with the same value are an OR, and then its an AND across different keys.

eg:

        ExactMatchFilter(key="artist", value="Taylor Swift"),
        ExactMatchFilter(key="artist", value="Katy Perry"),
        ExactMatchFilter(key="genre", value="pop"),

Generates (Taylor Swift OR Katy Perry) AND (pop)

Now, we need some way to say (Taylor Swift AND Katy Perry) OR (pop)

This is useful for multi valued fields (e.g multiple artists in a song)

Is it possible to do this today?

Hi @llermaly
How are you using ES?
I am getting below error:
TypeError: IndicesClient.create() got an unexpected keyword argument 'mappings'

[Spartee]

The redis team wants to eventually add the same filtering support we did for LC (langchain-ai/langchain#8612) here as well. We are working on it.