refactor: count number of documents using hnswlib

[jupyterjazz]

Data storage in HnswDocumentIndex works in the following way:

1. Vectors are stored on disk using hnswlib.
2. All other types of data are saved in an SQLITE database.

One of the operations we frequently perform is determining the total number of documents (num_docs()). However, the only way to get number of documents from SQLITE is by scanning the entire table. Even though we've made efforts to reduce the number of times we use this functionality (#1729), it's still a time-consuming process.

For better performance, let's do the following: instead of scanning the SQLITE table, we can use hnswlib's get_current_count function to quickly get the number of documents in the index.

But there's a potential issue with this approach. What if documents don't have associated vectors? get_current_count would return 0.

We have two potential solutions:

1. Notify/Warn users about this behavior and return 0.
2. Use to the older method of counting using the SQL table if vector-less documents are detected.

[shobhit9957]

/attempt #1759

I would like to solve this issue. Can I get some more help.

[JoanFM]

Hey @shobhit9957 ,

Sure, the main refactoring to be done is to change the code here:

    def num_docs(self) -> int:
        """
        Get the number of documents.
        """
        if self._num_docs == 0:
            self._num_docs = self._get_num_docs_sqlite()
        return self._num_docs

to something that does not rely on the _get_num_docs_sqlite but from a new private method _get_num_docs_hnsw

[shobhit9957]

so I should replace the _get_num_docs_sqlite to _get_num_docs_hnsw is that correct?
I'm just a beginner, to this open-source-community. Just trying my hands out there...please help me out.

[JoanFM]

Yes, this would be the right approach

[shobhit9957]

Thanks! will do the PR Soon.

[shobhit9957]

Hey Joan. Submitted the PR, please check if there are any other issues or mistakes I've done in the PR, I would be happy to solve my mistakes and submit the PR again Joan. Thanks!

[shobhit9957]

def num_docs(self) -> int:
"""
Get the number of documents.
"""
if self._num_docs == 0:
# Replace the old method with the new one
self._num_docs = self._get_num_docs_hnsw()
return self._num_docs

def _get_num_docs_hnsw(self) -> int:
"""
Get the number of documents using the HNSW method.
This method should return the count of documents using the new technique.
"""
# Implement your logic here to get the count using the HNSW method
# For example, you might access some data or perform calculations
# that help you quickly determine the number of documents.
# Then return the count.
return calculated_num_docs
is this correct?

[munish0838]

Hi, is this issue fixed or needs contribution yet.
Would like to contribute

[JoanFM]

Hey @munish0838, there is still work to be done

[munish0838]

I would like to start working on this issue

[JoanFM]

be my guest. I believe there are parts of the plan that were not applied yet.