-
-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When I attempt to insert new data into a table indexed with pg_search, I encounter failure. #1047
Comments
There are two separate errors here that suggest that something is going wrong on the file system.
I haven't seen this before. Could you try again on our latest release? ( |
Hi @neilyio , I'll update the version and give it a try. I'll let you know if there's any progress. Thank you. |
Going to close this in the meantime. Please feel free to re-open the issue if you still face problems on |
Hi @neilyio @philippemnoel ,
book-test=> CALL paradedb.create_bm25(
index_name => 'index_book_local_embedding_text',
schema_name => 'public',
table_name => 'book_local_embedding',
...
);
book=> CALL paradedb.create_bm25(
index_name => 'index_book_local_embedding_text',
schema_name => 'public',
table_name => 'book_local_embedding',
...
);
# Success
book-test=> INSERT INTO "book_local_embedding" (...) VALUES ...;
# Error: failed to create index reader while retrieving index.
# The error log is in the subject.
book=> INSERT INTO "book_local_embedding" (...) VALUES ...;
|
After observing for a while, I've noticed that the error persists. The successful insertion after renaming was just a coincidence. The issue is indeed related to the inserted data. This is the insert SQL. |
Thank you for reporting, sorry we missed this. We'll take a look and get back to you |
I'm experiencing the identical problem even after updating to version v0.6.1. |
Can you provide a reproduction? Are you able to share the data you're inserting? |
Thanks for response. Description: I'm encountering an error while performing bulk insertions into a ParadeDB table with HNSW and BM25 indexes.The error [215] ERROR: XX000: failed to create index reader while retrieving index Notably, the error consistently occurs after approximately 5-6 insertions. Steps to Reproduce:
CREATE TABLE IF NOT EXISTS chunks
(
id BIGSERIAL PRIMARY KEY,
content TEXT,
is_deleted BOOLEAN,
index_bert Vector(768),
index_text TEXT
);
CREATE INDEX IF NOT EXISTS semantic_index_bert ON chunks USING hnsw (index_bert vector_cosine_ops);
CALL paradedb.create_bm25(
index_name => 'literal_index_text',
schema_name => 'public',
table_name => 'chunks',
key_field => 'id',
text_fields => '{index_text: {tokenizer: {type: "default"}, fast: true}}'
);
import random
import string
import psycopg
from tqdm import tqdm
def generate_random_word(length=5):
return ''.join(random.choice(string.ascii_lowercase) for _ in range(length))
if __name__ == '__main__':
connection_string = "postgresql://test:test@localhost:5432/test_db"
sql_contexts = []
for i in tqdm(range(100)):
sql_context = "insert into chunks (content, is_deleted) values"
for j in range(1000):
sql_context += f"""('{generate_random_word(100)}', False)"""
if j != 999:
sql_context += ",\n"
sql_context += ";"
sql_contexts.append(sql_context)
with psycopg.connect(connection_string, autocommit=True) as conn:
for sql_context in tqdm(sql_contexts):
cursor = conn.cursor()
with conn.transaction():
cursor.execute(sql_context) Expected Behavior: The Python code should successfully insert 100,000 rows (100 insertions with 1000 rows each) into the chunks table Paradedb version: v0.7.3 |
Bug Description
When I attempt to insert new data into a table indexed with pg_bm25, I encounter failure.
How To Reproduce
logs
The text was updated successfully, but these errors were encountered: