Optimization error: Service internal error: IO Error: No such file or directory (os error 2)

[shuxueslpi]

Optimization error: Service internal error: IO Error: No such file or directory (os error 2)

Current Behavior

Use docker on windows, and log error:

2024-04-24 14:33:11 qdrant  | 2024-04-24T06:33:11.087478Z ERROR qdrant::startup: Panic backtrace: 
2024-04-24 14:33:11 qdrant  |    0: qdrant::startup::setup_panic_hook::{{closure}}
2024-04-24 14:33:11 qdrant  |    1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
2024-04-24 14:33:11 qdrant  |    2: std::panicking::rust_panic_with_hook
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
2024-04-24 14:33:11 qdrant  |    3: std::panicking::begin_panic_handler::{{closure}}
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
2024-04-24 14:33:11 qdrant  |    4: std::sys_common::backtrace::__rust_end_short_backtrace
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
2024-04-24 14:33:11 qdrant  |    5: rust_begin_unwind
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
2024-04-24 14:33:11 qdrant  |    6: core::panicking::panic_fmt
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
2024-04-24 14:33:11 qdrant  |    7: tokio::runtime::task::raw::poll
2024-04-24 14:33:11 qdrant  |    8: std::sys_common::backtrace::__rust_begin_short_backtrace
2024-04-24 14:33:11 qdrant  |    9: core::ops::function::FnOnce::call_once{{vtable.shim}}
2024-04-24 14:33:11 qdrant  |   10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
2024-04-24 14:33:11 qdrant  |   11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
2024-04-24 14:33:11 qdrant  |   12: std::sys::unix::thread::Thread::new::thread_start
2024-04-24 14:33:11 qdrant  |              at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
2024-04-24 14:33:11 qdrant  |   13: <unknown>
2024-04-24 14:33:11 qdrant  |   14: <unknown>
2024-04-24 14:33:11 qdrant  |     
2024-04-24 14:33:11 qdrant  | 2024-04-24T06:33:11.087503Z ERROR qdrant::startup: Panic occurred in file lib/collection/src/update_handler.rs at line 345: Optimization error: Service internal error: IO Error: No such file or directory (os error 2)    
2024-04-24 14:33:12 qdrant  | 2024-04-24T06:33:12.772613Z  WARN collection::update_handler: Optimization task panicked, collection may be in unstable state: Optimization error: Service internal error: IO Error: No such file or directory (os error 2)  

Steps to Reproduce

I use docker on windows:

1. docker pull qdrant/qdrant:v1.8.4
2. use docker-compose like this:

  qdrant:
    image: qdrant/qdrant:v1.8.4
    hostname: qdrant
    restart: always
    container_name: qdrant
    volumes:
      - ./qdrant_storage:/qdrant/storage
    ports:
      - 6333:6333
      - 6334:6334
    environment:
      - QDRANT__LOG_LEVEL=INFO
      - QDRANT__OPTIMIZERS__MEMMAP_THRESHOLD_KB=20000
      - QDRANT__HNSW_INDEX__ON_DISK=true
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2048M

3. docker-compose up, already uploaded data, use docker stats to check:

CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O        PIDS
424eb0eebd7f   qdrant                 0.02%     1.52GiB / 2GiB      53.01%    47MB / 122kB      326MB / 858MB    15 

4. continue add data to qdrant, I found when memory usage suddenly drops, the error log occurs, check docker stats, looks like this:

CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O        PIDS
424eb0eebd7f   qdrant                 0.02%     1.02GiB / 2GiB      53.01%    47MB / 122kB      326MB / 858MB    15

The way I use qdrant client is like this:

from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.vector_stores.qdrant.utils import default_sparse_encoder
from llama_index.embeddings.azure_openai import AzureOpenAIEmbedding
from qdrant_client import QdrantClient

embedding_model = AzureOpenAIEmbedding(
        model=os.environ["AZURE_OPENAI_API_EMBEDDING_DEPLOYMENT"],
        api_key=os.environ["OPENAI_API_KEY"],
        azure_endpoint=os.environ["OPENAI_API_BASE"],
        api_version=os.environ["AZURE_OPENAI_API_VERSION"])

qdrant_client = QdrantClient(url=qdrant_url, api_key=qdrant_key)

vector_store = QdrantVectorStore(collection_name=...,
			         client=qdrant_client,
				 enable_hybrid=True,
				 batch_size=16,
				 sparse_doc_fn=default_sparse_encoder('./efficient-splade-VI-BT-large-doc'),
				 sparse_query_fn=default_sparse_encoder('./efficient-splade-VI-BT-large-query'))

index = VectorStoreIndex.from_vector_store(vector_store=vector_store,
                                           embed_model=embedding_model)

Expected Behavior

My guess is that the program is optimizing its memory usage to lower it. However, it seems that a problem was encountered during the optimization process, resulting in an error message. The ideal state should be that the program completes every optimization in the background without reporting any error messages.
After the error message appears, you can still continue to use it, but I'm not sure if there will be other adverse effects.

How can I configure it to avoid this problem?

[generall]

hey @shuxueslpi, thanks for reporting this!

How reliably can this issue be reproduced? Meaning, does this happen each time you try to index?

[shuxueslpi]

hey @shuxueslpi, thanks for reporting this!

How reliably can this issue be reproduced? Meaning, does this happen each time you try to index?

Hi @generall，I am currently in the development stage, doing self-tests, and will restart the program frequently. After every restart, this error will appear when adding data for the first time.（docker-compose up and docker-compose down...）
I am not sure whether this error will appear after the program has been running for a period of time, because the service is still under development and has not been run for a long time.

[shuxueslpi]

hey @shuxueslpi, thanks for reporting this!

How reliably can this issue be reproduced? Meaning, does this happen each time you try to index?

Hi @generall , I did some more testing, and in my scenario I turned on hybrid search, using the default sparse embedding function.
I found that the above problem only occurs when hybrid search is turned on. This problem does not occur if only dense embedding is used.
What's even worse is that when using hybrid search, after this problem occurs, the qdrant container cannot be accessed after running for a period of time.