I put my PDFs on Reor

[Arcovoltaire]

I'm not a programmer, but someone who really wants to use the program. To really use the full capacity of the program I needed to transform my database which is also in PDF (1500 files) to md with file names without accents and special characters. After searching the internet for something that could do this massively, I decided to try a script with CHATGPT. He told me to install the PyMuPDF library.

pip install PyMuPDF

And he created this script for me that worked, it converted the 1500 PDFs at once, so I put it in the Reor folder and it imported it.

import fitz # PyMuPDF
import os
import unicodedata
import re

def normalize_filename(filename):
# Normaliza para a forma NFD (Normalização de Decomposição de Forma Canônica)
# e então remove os caracteres de combinação.
normalized = unicodedata.normalize('NFD', filename)
# Mantém apenas caracteres válidos (letras, números, espaços e alguns símbolos)
return re.sub(r'[^\w\s-]', '', normalized).replace(' ', '_')

def pdf_to_markdown(pdf_path, markdown_path):
doc = fitz.open(pdf_path)
markdown_content = ""

for page in doc:
    markdown_content += page.get_text()

with open(markdown_path, "w", encoding="utf-8") as markdown_file:
    markdown_file.write(markdown_content)

Caminho para o diretório com os PDFs

input_directory = "./pdf"
output_directory = "./md"

for pdf_file in os.listdir(input_directory):
if pdf_file.endswith(".pdf"):

    pdf_path = os.path.join(input_directory, pdf_file)
    # Extrai o nome do arquivo sem a extensão .pdf
    file_name_without_extension = pdf_file[:-4]
    # Normaliza o nome do arquivo
    normalized_filename = normalize_filename(file_name_without_extension)
    # Adiciona a extensão .md manualmente
    markdown_filename = normalized_filename + ".md"
    markdown_path = os.path.join(output_directory, markdown_filename)
    pdf_to_markdown(pdf_path, markdown_path)

so it has been indexing the files for 1 hour

[Arcovoltaire]

After importing the files, they all appear with the correct name.

[samlhuillier]

Great @Arcovoltaire! So everything worked as expected except the slowness of indexing?

[Arcovoltaire]

Result: From what I studied, Xenova uses browser processing, and from tests, the larger the note size, and the greater the quantity of them, the slower it is to process everything. Some Xenovas process more than others, and understand and deliver context to the LLM in completely different ways depending on how they understand and process the language. For Portuguese, for my type of material, the perfect Xenova was Xenova/multilingual-e5-large - I analyzed about 10 to get to it, and observed what it processed both in the LMStudio terminal and in the left bar that shows which pieces of notes will be sent. Well, all this to say that since RAG is working well, I put 12 in RAG, but as a notes program, it was too heavy, I can't even touch it. But for my purpose, which is extracting information from notes, I'm loving it.

[samlhuillier]

So the editor (writing notes inside Reor) is too slow for you to do anything productive?

[Arcovoltaire]

Yes, with this xenova, anything I do the processor explodes at 100% and stops everything for a while. But with other xenovas it's lighter, but it doesn't associate the content well.

[Arcovoltaire]

But again: My case is specific to large notes, and many notes in the file.