Getting error: KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'

[manishparanjape]

Followed these instructions: https://docs.dbgpt.site/docs/installation/model_service/stand_alone
MAc details : Apple M2 Max (64 GB)

dbgpt model list
+-----------------+------------+---------------+------+---------+---------+-----------------+----------------------------+
|    Model Name   | Model Type |      Host     | Port | Healthy | Enabled | Prompt Template |       Last Heartbeat       |
+-----------------+------------+---------------+------+---------+---------+-----------------+----------------------------+
| vicuna-13b-v1.5 |    llm     | 192.168.86.29 | 6006 |   True  |   True  |                 | 2024-01-04T10:13:26.627204 |
|  WorkerManager  |  service   | 192.168.86.29 | 6006 |   True  |   True  |                 | 2024-01-04T10:13:26.627320 |
+-----------------+------------+---------------+------+---------+---------+-----------------+----------------------------+

dbgpt model chat --model_name vicuna-13b-v1.5
Chatbot started with model vicuna-13b-v1.5. Type 'exit' to leave the chat.


You: Hi
Bot: **LLMServer Generate Error, Please CheckErrorInfo.**: 'Cache only has 0 layers, attempted to access layer with index 0'

ERROR:
dbgpt.model.cluster.worker.default_worker[3654] ERROR Model inference error, detail: Traceback (most recent call last):
  File "/Users/mpla/Projects/text2sql/DB-GPT/dbgpt/model/cluster/worker/default_worker.py", line 154, in generate_stream
    for output in generate_stream_func(
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
    response = gen.send(None)
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/fastchat/serve/inference.py", line 132, in generate_stream
    out = model(input_ids=start_ids, use_cache=True)
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1181, in forward
    outputs = self.model(
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1068, in forward
    layer_outputs = decoder_layer(
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 796, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/mpla/Projects/text2sql/DB-GPT/dbgpt/model/llm/monkey_patch.py", line 58, in forward
    kv_seq_len += past_key_value[0].shape[-2]
  File "/Users/mpla/anaconda3/envs/dbgpt_env/lib/python3.10/site-packages/transformers/cache_utils.py", line 78, in __getitem__
    raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'