Replies: 3 comments
-
🤖 Hello, Based on the similar issues you've found, it seems you can use a custom model with Here's an example of how you might do this: from llama_index import VectorStoreIndex
from transformers import AutoTokenizer, AutoModel
# Load your custom model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("path/to/your/tokenizer")
model = AutoModel.from_pretrained("path/to/your/model")
# Use them with VectorStoreIndex.from_documents
index = VectorStoreIndex.from_documents(
documents=["doc1", "doc2", "doc3"],
tokenizer=tokenizer,
model=model
) In this example, replace If you're unable to load your model and tokenizer outside of the huggingface LLM, you might need to implement a custom LLM class. You can find more information on how to do this in the LlamaIndex documentation. I hope this helps! Let me know if you have any other questions. Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. |
Beta Was this translation helpful? Give feedback.
-
What if I want to use the LlamaCPP object as model for VectorStoreIndex.from_documents? |
Beta Was this translation helpful? Give feedback.
-
@barvhaim Once you have the llm object, just throw it into the service context (NOTE: make sure you have llama-cpp-python <= 0.1.78 installed if using GGML files, after that version they switched to GGUF)
Also important is those mesages_to_prompt and completion_to_prompt function. Most models need a specific input format, and those functions convert it. Right now, the ones in that example are specific to llama-2-chat |
Beta Was this translation helpful? Give feedback.
-
Question Validation
Question
When I'm trying to use llama-index
VectorStoreIndex.from_documents
without OpenAI it will download and use a default llama-2 model (llama-2-13b-chat.ggmlv3.q4_0.bin) as mentioned inhttps://gpt-index.readthedocs.io/en/stable/examples/llm/llama_2_llama_cpp.html
.I managed to load different model and use it with constructing
LlamaCPP
object as mentioned in doc, but how can I use it withVectorStoreIndex.from_documents
? was not clear from the documentationBeta Was this translation helpful? Give feedback.
All reactions