How to install lmdeploy.serve.openai?

[realhaik]

I have followed the documentation and did: pip install lmdeploy
Unfortunately I am unable to use lmdeploy.serve.openai, as I get ModuleNotFoundError: No module named 'lmdeploy.serve.openai'
This is running in Ubuntu 22.04 docker container image.

root@3b1c8b744e04:/hostfs# pip show lmdeploy
Name: lmdeploy
Version: 0.0.5
Summary: A toolset for compressing, deploying and serving LLM
Home-page: None
Author: OpenMMLab
Author-email: openmmlab@gmail.com
License: None
Location: /usr/local/lib/python3.8/dist-packages
Requires: gradio, mmengine, pybind11, numpy, nvidia-nccl-cu11, torch, datasets, tritonclient, setuptools, transformers, sentencepiece, fire, safetensors
Required-by:

root@3b1c8b744e04:/hostfs# pip show lmdeploy.serve.openai
WARNING: Package(s) not found: lmdeploy.serve.openai

[lvhan028]

It will be released in v0.0.6.

[realhaik]

Hi @lvhan028 ,

I have merged the bump-version(0.0.6) into the main, and reinstalled lmdeploy.
Unfortunately it is still does not work when I try to run it.

Successfully built lmdeploy
Installing collected packages: lmdeploy
Successfully installed lmdeploy-0.0.6
root@3b1c8b744e04:/hostfs/lmdeploy# python3 -m lmdeploy.serve.openai.api_server ./workspace 0.0.0.0 8081 --instance_num 1 --tp 1
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/hostfs/lmdeploy/lmdeploy/serve/openai/api_server.py", line 13, in
from lmdeploy.serve.async_engine import AsyncEngine
File "/hostfs/lmdeploy/lmdeploy/serve/async_engine.py", line 9, in
from lmdeploy import turbomind as tm
File "/hostfs/lmdeploy/lmdeploy/turbomind/init.py", line 3, in
from .turbomind import TurboMind
File "/hostfs/lmdeploy/lmdeploy/turbomind/turbomind.py", line 22, in
import _turbomind as _tm # noqa: E402
ModuleNotFoundError: No module named '_turbomind'

I am following this documentation as you advised: https://github.com/InternLM/lmdeploy/blob/main/docs/en/restful_api.md

This is the second week that I am trying to to get lmdeploy to work. Is it possible to get a clear answer if it is possible to install this software and then use it? Because if I only can install it without the option to use, then there is no point in installing it.

[lvhan028]

Hi, @realhaik Sorry for the inconvenience.
LMDeploy needs to build turbomind from source. And I am going to release v0.0.6 today and publish it to pypi. After that, you can do pip install lmdeploy==0.0.6. I'll let you know once it is done.

Besides, when you install lmdeploy via pip install lmdeploy, please don't run python3 -m lmdeploy.serve in the root dir of lmdeploy source code, since it will look for the python package in the directory of the source code, where _turbomind doesn't exist. Instead, please run the command anywhere else.

[lvhan028]

@realhaik You may try to pip install lmdeploy==0.0.6 now

[realhaik]

@lvhan028
Hi,
I was able to install and run it, but the request to the /v1/chat/completions does not return.

# python3 -m lmdeploy.serve.openai.api_server ./workspace 0.0.0.0 8099 --instance_num 32 --tp 1
[WARNING] gemm_config.in is not found; using default GEMM algo
INFO:     Started server process [15]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8099 (Press CTRL+C to quit)

When I run:
GET http://localhost:8099/v1/models
I get response:

{
   "object":"list",
   "data":[
      {
         "id":"llama2",
         "object":"model",
         "created":1692996310,
         "owned_by":"lmdeploy",
         "root":"llama2",
         "parent":null,
         "permission":[
            {
               "id":"modelperm-zbXYShbcuhNdSvcfimoxqh",
               "object":"model_permission",
               "created":1692996310,
               "allow_create_engine":false,
               "allow_sampling":true,
               "allow_logprobs":true,
               "allow_search_indices":true,
               "allow_view":true,
               "allow_fine_tuning":false,
               "organization":"*",
               "group":null,
               "is_blocking":false
            }
         ]
      }
   ]
}

Unfortunately the following request gets stuck during the response, and never completes/returns even after an hour.

POST http://localhost:8099/v1/chat/completions HTTP/1.1
Content-Type: application/json
Host: localhost:8099
Content-Length: 99

{
"model": "llama2",
"messages": [{"role": "user", "content": "Hello! Ho are you?"}]
}

# nvidia-smi
Fri Aug 25 22:25:42 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.01              Driver Version: 536.67       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        On  | 00000000:01:00.0 Off |                    0 |
|  0%   29C    P8              13W / 450W |  16629MiB / 23028MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        15      C   /python3.10                               N/A      |
|    0   N/A  N/A        32      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        33      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+
#
# free -m
               total        used        free      shared  buff/cache   available
Mem:           48060        4669       25728          86       17661       42727
Swap:          12288           0       12288
#
# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
#

These are my installation steps:

docker pull nvidia/cuda:11.8.0-base-ubuntu22.04

docker run -p 8099:8099 --cap-add=SYS_PTRACE -v "C:/AI/LLaMA2_Docker_FileSystem":/hostfs/ --security-opt seccomp=unconfined --name LLaMA2 --cap-add=SYS_ADMIN --shm-size 22g --gpus all -it --env NCCL_LAUNCH_MODE=PARALLEL nvidia/cuda:11.8.0-base-ubuntu22.04 bash

apt-get update
apt-get install python3-pip

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118

apt update
apt install wget

wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
apt-get update
apt install libxml2

sh cuda_11.8.0_520.61.05_linux.run --silent --driver

pip install lmdeploy==0.0.6

python3 -m lmdeploy.serve.turbomind.deploy llama2 llama-2-7b-chat --model-format llama --tokenizer-path llama-2-7b-chat/tokenizer.model

export PATH=$PATH:/usr/local/cuda-11/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11/lib64

python3 -m lmdeploy.serve.openai.api_server ./workspace 0.0.0.0 8099 --instance_num 32 --tp 1

[lvhan028]

@AllentDan could you help investigate it?

[AllentDan]

@realhaik Can you run python3 -m lmdeploy.turbomind.chat ./workspace and chat successifully? Besides, could you give it a test on API generate with instance_id = 1?

[realhaik]

not working:

# cd hostfs
# python3 -m lmdeploy.turbomind.chat ./workspace
WARNING: Can not find tokenizer.json. It may take long time to initialize the tokenizer.
[WARNING] gemm_config.in is not found; using default GEMM algo

double enter to end input >>> Who is bill gates?

session 1
<BOS><|User|>:Who is bill gates?<eoh>
<|Bot|>:

instance_id = 1 where? I don't see instance_id anywhere, neither I see "generate".
If you mean the instance_num, then yes I have tried it with 1, and it is the same thing.

[AllentDan]

I see. Seems the architectures with compute capability is lower than 7.0. Is it right? We primarily did not support this yet. Check this out.

[realhaik]

Are you saying that you don't support Nvidia RTX 4090 Graphics Cards?

[AllentDan]

No. I just suspect that. But 4090 is in the support list. Could you set log level to info to chat.py by replace this line with os.environ['TM_LOG_LEVEL'] = 'INFO'. Or I can not get any clue from the situation.

[realhaik]

Unfortunately I am unable to compile the code, I can do only pip install

[AllentDan]

It is okay actually. Just pip show lmdeploy to get the codes root. Then find the lmdeploy/turbomind/chat.py file inside the root dir to modify it.

[realhaik]

cd hostfs

python3 -m lmdeploy.turbomind.chat ./workspace

WARNING: Can not find tokenizer.json. It may take long time to initialize the tokenizer.
[TM][INFO] Set logger level by INFO
[TM][INFO] Barrier(1)
[TM][INFO] Set logger level by INFO
[TM][INFO] Barrier(1)
[TM][INFO] Set logger level by INFO
[WARNING] gemm_config.in is not found; using default GEMM algo
[TM][INFO] NCCL group_id = 0
[TM][INFO] [LlamaCacheManager] max_entry_count = 48
[TM][INFO] [LlamaCacheManager] chunk_size = 1
[TM][INFO] [LlamaCacheManager][allocate]
[TM][INFO] [LlamaCacheManager][allocate] malloc 1
[TM][INFO] [LlamaCacheManager][allocate] count = 1
[TM][INFO] [LlamaCacheManager][allocate] free = 1
[TM][INFO] Set logger level by INFO
[TM][INFO] [internalThreadEntry] 0

double enter to end input >>> who is bill gates?

session 1
<|User|>:who is bill gates?
<|Bot|>: [TM][INFO] Set logger level by INFO
[TM][INFO] [forward] Enqueue requests
[TM][INFO] [forward] Wait for requests to complete ...
[TM][INFO] [synchronize] batch_size = 0
[TM][INFO] [LlamaCacheManager][create] 1
[TM][INFO] [LlamaCacheManager][allocate]
[TM][INFO] [LlamaCacheManager][allocate] free = 0
[TM][INFO] [init] infer_request_count = 1
[TM][INFO] [init] batch_size = 1
[TM][INFO] [init] session_len = 2056
[TM][INFO] [init] max_input_length = 18
[TM][INFO] [init] max_context_len = 18
[TM][INFO] [init] slot sequence_id history_len input_len context_len tmp_input_len token_ids.size cache_len
[TM][INFO] [init] 0 1 0 18 18 18 0 0
[TM][INFO] [decodeContext] base = 0, count = 1
[TM][INFO] [decodeContext] offset = 0, batch_size = 1, token_num = 17, max_input_len = 17, max_context_len = 17
[TM][INFO] context decoding start