Issues API: Too many collections

[coszio]

Supersedes #3652, same thing but now event-based
Tracked in #3471
Needs #4139

This PR determines to have too many collections when all of the following are true:

• It has above 30 collections
• The density of points in the collections is low (avg_points / num_collections)

The current way of counting the amount of points in the collections is using the count api, aproximate.

All Submissions:

• Contributions should target the dev branch. Did you create your branch from dev?
• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

1. Does your submission pass tests?
2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
3. Have you checked your code using cargo clippy --all --all-features command?

[timvisee]

We might want to define a number in kilobytes instead. It represents the actual size in bytes, which is more interesting than just a count. We use it in many places, including for our indexing threshold:

qdrant/config/config.yaml

Line 112 in 97c107f

indexing_threshold_kb: 20000

Speaking of it, we might want to increase this number to more closely match the default indexing threshold. Note that the threshold is per segment.

Wdyt?

[coszio]

I don't see why kilobytes is a better metric here, since the problem we want to spot is that the user has a dynamic number of collections. E.g. having many named vectors and/or high-dimensional ones does not relate to the root of the issue, IMO.

Would you elaborate why you think it is a better metric in this case?

Regarding the threshold, I am happy to move it around. The choice was completely arbitrary.

[timvisee]

My thought was this: we count collections that are below some point threshold. I thought that it would make sense to match that up with the indexing threshold, in which case kilobytes should be used. I thought about it that way because it's usually one of the problems we mention when a lot of collections are created, many of them will be too small to make use of indexing.

But, if you designed it a bit different or if we just trigger from n collections onward, the specific metric probably doesn't really matter.

[timvisee]

we want to spot is that the user has a dynamic number of collections.

Could you elaborate on what this means, a dynamic number of collections?

Wouldn't we just trigger this issue above 30 (?) collections, no matter their contents?

[coszio]

For me, a dynamic number of collections means that creation or deletion of collections come from user interaction, not from admin decisions.

Valid cases for creating new collections might be:

• using new embedding models
• refactoring, new features, etc.

I think it might still be possible to reach the fixed limit while having "correct" usage, that's why I included the density threshold too.