[SOLVED]My chatbot always responds the same

[insoen]

I create a chatbot with llama_index and chromadb that works well locally (the responses are in Spanish, sorry. But they are valid responses).

My code is like this:

`from llama_index.llms.openai import OpenAI
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader, StorageContext, load_index_from_storage
import os
from llama_index.core.chat_engine import SimpleChatEngine
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
import dotenv
import time

vienen del .env

dotenv.load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")
os.environ["OPENAI_API_KEY"] = api_key

def cargar_indice():
# load from disk
db2 = chromadb.PersistentClient(path="./crhoma_db")
chroma_collection = db2.get_collection("cummins")
print(chroma_collection.count())
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_vector_store(
vector_store
)
return index`

`

Crear o cargar el índice

index = cargar_indice()

chat_engine = index.as_chat_engine(chat_mode='react', system_prompt=(
# "Eres un experto en motores cummins y te llamas Makinebot.Sólo chateas sobre motores Cummins en base a la documentación suministrada"
"Eres un experto en motores cummins y respondes siempre en español sin usar conocimientos externos a los documentos almacenados."
))

Ejecutar inferencia con Rag

while True:
consulta = input("Usuario: ")
if consulta.lower() == "salir":
print("Chatbot: ¡Hasta luego! 👋")
break
respuesta = chat_engine.chat(consulta)
print(f"Chatbot: {respuesta}")`

I copied that chroma_db database onto a server and created a form with Flask, and based on this code, I created a Chatbot class. But if I ask it the same things as in the first test, it always responds with 'what do you want to know about Cummins?' or similar phrases.
In both cases, the context is the same: 'you are an expert in Cummins engines'.

My class Chatbot:

class Chatbot:
def init(self, contexto, coleccion):
# vienen del .env
dotenv.load_dotenv()
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")
self.contexto = contexto
self.coleccion = coleccion
self.embed_model = OpenAI(model_name="text-embedding-3-small")

    #cargar el índice:
    self.index = self.cargar_indice()

    #crear el motor del chat:
    self.chat_engine = self.index.as_chat_engine(
        chat_mode='openai',
        system_prompt=(
            self.contexto
        ),
        llm = self.embed_model,
        verbose = True,
        similarity_top_k=5,
    )

def cargar_indice(self):
    # load from disk
    db2 = chromadb.PersistentClient(path="./crhoma_db")
    chroma_collection = db2.get_collection(self.coleccion)
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    index = VectorStoreIndex.from_vector_store(
        vector_store,
        embed_model=self.embed_model
    )
    return index

def responder(self, consulta):
    # Obtener respuesta del chatbot
    respuesta = self.chat_engine.chat(consulta)
    return respuesta`

Update: Solved. The issue was my own error unrelated to Llama_index. Thanks for the assistance.

[dosubot[bot]]

¡Hola de nuevo, @insoen! Qué alegría verte por aquí una vez más, llevando la ingeniería de chatbots a nuevos horizontes. Espero que todo lo demás vaya tan interesante como tu proyecto. 😊🔧🚀

Based on the information you've provided, there are a few potential areas that could be causing the issue you're experiencing.

Firstly, I noticed that you're using different chat modes in your local and server setups. In your local setup, you're using chat_mode='react', while in your server setup, you're using chat_mode='openai'. The chat mode can significantly influence the behavior of the chat engine. The 'react' mode is designed to interact with the chat engine in a reactive manner, focusing on responding to user inputs by leveraging a set of predefined tools or functions that can query an engine or database. On the other hand, the 'openai' mode is specifically designed to work with OpenAI models that support the function calling API, which allows the chatbot to execute functions or actions directly through the OpenAI model's API. This difference could be causing the change in behavior you're observing. I would recommend using the same chat mode in both environments to ensure consistent behavior.

Secondly, it's worth noting that moving from a local environment to a server environment can introduce a variety of factors that could influence the behavior of your application. These include differences in the Python versions, library versions, system resources, database connectivity, and latency. It's important to ensure that your server environment mirrors your local development environment as closely as possible.

Additionally, Flask applications can behave differently under load, especially regarding concurrency and how state is managed. If your Flask application is not correctly handling concurrent requests or managing state in a thread-safe manner, it could lead to repetitive or generic responses.

Lastly, the server might be encountering errors or exceptions that are not present or visible in the local environment. Ensure that your Flask application has comprehensive error handling and logging in place to capture any issues that might be affecting the chatbot's behavior.

Without more specific information about your server setup and the exact behavior of your chatbot, it's difficult to provide a more precise diagnosis. However, I hope these suggestions provide a good starting point for you to investigate further. If you have more information or if there's anything else you'd like to discuss, please don't hesitate to provide more details.

Sources

• llama-index-packs/llama-index-packs-cohere-citation-chat/llama_index/packs/cohere_citation_chat/citations_context_chat_engine.py
• llama-index-legacy/llama_index/legacy/chat_engine/types.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[insoen]

hank you for your response. In Flask, I am also using react mode, but the result is the same. I simply changed it because I wanted to see if there was a difference using that mode. In Flask, using Verbose mode, I see that the chatbot is adding my messages to memory. I have also asked the code to print a count of documents and the collection it is using, and I understand that it is indeed reading my collection. Many thanks, I am going to run a test that I just came up with to see if anything happens.

[insoen]

I did a test and it was interesting. I put the while loop from my original code into my Chatbot class, and while in my Flask view it just repeats the same nonsense, in the terminal where I run the server I asked questions and received more coherent answers. I think the problem might be that with each question, I am starting the chat again or something like that. I'm not sure, but I have a feeling that the problem is me xD.