配置模型时，可以配置在别的服务器部署的chatglm6b吗

[carllisicau]

大佬们 请问下
目前看到文档中，可以配置本地大模型和在线大模型api
但是本地大模型似乎要在一台机器上
langchain chatchat 可以配置部署在另外一台机器上的大模型chatglm6b吗

[dosubot]

Yes, you can configure Langchain Chatchat to use a ChatGLM6B model deployed on a different server. Here's how to do it:

1. Update model_config.py: Add a new entry for your remote model with details like api_base_url and api_key. For example:

ONLINE_LLM_MODEL = {
    "chatglm3-6b-remote": {
        "api_base_url": "https://your.remote.server/api",
        "api_key": "your_api_key_here",
        "provider": "YourProviderWorker",
    },
}

2. Modify get_ChatOpenAI function: Adjust this function to make HTTP requests to your remote model's API. Use libraries like httpx for asynchronous requests. Here's a basic structure:

import httpx

async def get_ChatOpenAI(model_name: str, temperature: float, max_tokens: Optional[int], callbacks: list):
    api_endpoint = "http://localhost:5000/generate"
    data = {
        "model_name": model_name,
        "temperature": temperature,
        "max_tokens": max_tokens,
    }
    async with httpx.AsyncClient() as client:
        response = await client.post(api_endpoint, json=data)
        generated_text = response.json().get("text", "")
        return generated_text

3. Configure the application to use the remote model: Ensure your application is set up to use the new remote model identifier you've added.

Remember, network latency and bandwidth are crucial factors to consider for maintaining responsive interactions with the remote model.

For more detailed examples and discussions on similar setups, you might find these threads helpful:

• 使用模型chatglm3-6b，我的chatglm3-6b模型部署在服务器上而非本地，只对外提供了api_server，如何在langchain-chatchat交互中调用到chatglm3-6b自定义的工具（function calling）
• langchain调用自己部署的模型的api
• [BUG] 使用ChatGLM模型本地部署，运行时还需访问ChatGPT

Details

Sources

• 现在是不支持使用的时候切换模型了吗？
• 使用模型chatglm3-6b，我的chatglm3-6b模型部署在服务器上而非本地，只对外提供了api_server，如何在langchain-chatchat交互中调用到chatglm3-6b自定义的工具（function calling）
• 命令行运行python startup.py -a 报错
• [BUG] 简洁阐述问题 /怎么加载ChatGLM-6B微调出来的模型
• langchain调用自己部署的模型的api
• [BUG] 使用ChatGLM模型本地部署，运行时还需访问ChatGPT
• knowledge_base/samples/content/test_files/langchain-ChatGLM_open.jsonl
• server/chat/chat.py
• configs/model_config.py.example

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.