Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose use_index in Python #535

Open
wjones127 opened this issue Oct 3, 2023 · 2 comments · May be fixed by #560
Open

Expose use_index in Python #535

wjones127 opened this issue Oct 3, 2023 · 2 comments · May be fixed by #560
Assignees
Labels
enhancement New feature or request Python Python SDK

Comments

@wjones127
Copy link
Contributor

It looks like we started this in #433, but only did it for Node.

This functionality is critical for debugging and measuring recall.

@wjones127 wjones127 added enhancement New feature or request Python Python SDK labels Oct 3, 2023
@wjones127 wjones127 linked a pull request Oct 13, 2023 that will close this issue
@wjones127 wjones127 self-assigned this Oct 13, 2023
@westonpace
Copy link
Contributor

Since you're thinking about the topic, how should we handle "use scalar indices". Choices:

  • Just have a single use_index parameter that controls whether any kind of index is used
  • Have two boolean parameters, use_scalar_index and use_index (or use_vector_index)
  • Change use_index to accept a list of columns (or index names?) to use

@wjones127
Copy link
Contributor Author

It's an interesting question. For ANN/KNN search, this flag doesn't just flip between using an index and not; it also determines whether the results are approximate or not (since ANN indexes are approximate by nature). I think that makes it a little different than the scalar index case, where results are exact either way, right?

For this reason, I think it makes sense to keep this separate. It seems to me that disabling scalar indices is mostly for debugging purposes right? Makes me wonder if we shouldn't expose it in primary APIs at all and instead control it with an environment variable or global setting. What would you think of that?

alexkohler pushed a commit to alexkohler/lancedb that referenced this issue Apr 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Python Python SDK
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants