if tokenizer.eos_token_id not in stop_token_ids: AttributeError: 'NoneType' object has no attribute 'eos_token_id'

[yuxiyan]

Search before asking

• I had searched in the issues and found no similar issues.

Operating system information

Linux

Python version information

3.10

DB-GPT version

main

Related scenes

• Chat Data
• Chat Excel
• Chat DB
• Chat Knowledge
• Model Management
• Dashboard
• Plugins

Installation Information

• Installation From Source

• Docker Installation

• Docker Compose Installation

• Cluster Installation

• AutoDL Image

• Other

Device information

i used docker in window10 platform.

Models information

llM:vicuna-13b-v1.5
embeding:text2vec

What happened

see the pic showed:

What you expected to happen

i hope the issue will be resolved ,then i can use the db-gpt.

How to reproduce

i use it in docker,
docker run -itd --name dbgpt --shm-size="32g" --network=host -v /home/data/:/data python:3.10.7
1、apt-get update # apt
2、apt-get install git # apt
3、cd /data/zhl/dbgpt/
4、git clone [https://github.com/eosphoros-ai/DB-GPT.git
5、cd DB-GPT
6、pip install -r requirements.txt or pip install -e .
7、python -m spacy download zh_core_web_sm
8、mkdir models and cd models
9、git clone [https://huggingface.co/lmsys/vicuna-13b-v1.5
10、git clone https://huggingface.co/GanymedeNil/text2vec-large-chinese
11、cd ..
12、cp .env.template .env
13、apt-get install vim
14、vim .env
15、python dbgpt_server.py
16、CUDA_VISIBLE_DEVICES=0,1,2,3 python dbgpt_server.py

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[fangyinc]

Please provide some startup logs.

[bluescv]

Please provide some startup logs.

I have encoutered the same problem. My startup logs:

==========
== CUDA ==
==========

CUDA Version 12.1.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

*************************
** DEPRECATION NOTICE! **
*************************
THIS IMAGE IS DEPRECATED and is scheduled for DELETION.
    https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md

2024-05-11 02:54:07 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_unified_metadata_db_manager_factory and instance: <dbgpt.storage.metadata.db_factory.UnifiedDBManagerFactory object at 0x7fdab6cc08e0>
2024-05-11 02:54:08 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_thread_pool_default and instance: <dbgpt.util.executor_utils.DefaultExecutorFactory object at 0x7fdaa0b93190>
2024-05-11 02:54:08 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_default_scheduler and instance: <dbgpt.app.initialization.scheduler.DefaultScheduler object at 0x7fdaa0b91f60>
2024-05-11 02:54:08 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_model_controller and instance: <dbgpt.model.cluster.controller.controller.ModelControllerAdapter object at 0x7fdb904f5b10>
2024-05-11 02:54:08 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_connector_manager and instance: <dbgpt.datasource.manages.connector_manager.ConnectorManager object at 0x7fdaa0b92da0>
2024-05-11 02:54:08 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_plugin_hub and instance: <dbgpt.serve.agent.hub.controller.ModulePlugin object at 0x7fdb8c292890>
2024-05-11 02:54:08 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_multi_agents and instance: <dbgpt.serve.agent.agents.controller.MultiAgents object at 0x7fdaa14d05b0>
2024-05-11 02:54:08 6f5262b010bb dbgpt.app.initialization.embedding_component[1] INFO Register local LocalEmbeddingFactory
2024-05-11 02:54:08 6f5262b010bb dbgpt.model.adapter.embeddings_loader[1] INFO [EmbeddingsModelWorker] Parameters of device is None, use cuda
2024-05-11 02:54:08 6f5262b010bb dbgpt.app.initialization.embedding_component[1] INFO

=========================== EmbeddingModelParameters ===========================

model_name: text2vec
model_path: /app/models/text2vec-large-chinese
device: cuda
normalize_embeddings: None

======================================================================


2024-05-11 02:54:09 6f5262b010bb sentence_transformers.SentenceTransformer[1] INFO Load pretrained SentenceTransformer: /app/models/text2vec-large-chinese
2024-05-11 02:54:09 6f5262b010bb sentence_transformers.SentenceTransformer[1] WARNING No sentence-transformers model found with name /app/models/text2vec-large-chinese. Creating a new one with MEAN pooling.
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name embedding_factory and instance: <dbgpt.app.initialization.embedding_component.LocalEmbeddingFactory object at 0x7fdaa0b93700>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_model_cache_manager and instance: <dbgpt.storage.cache.manager.LocalCacheManager object at 0x7fda5f22b430>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_awel_trigger_manager and instance: <dbgpt.core.awel.trigger.trigger_manager.DefaultTriggerManager object at 0x7fda5f0b7e80>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_awel_dag_manager and instance: <dbgpt.core.awel.dag.dag_manager.DAGManager object at 0x7fda59dab2e0>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_app_editor_service and instance: <dbgpt.app.openapi.api_v1.editor.service.EditorService object at 0x7fda59dab460>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_serve_prompt and instance: <dbgpt.serve.prompt.serve.Serve object at 0x7fda59dab4c0>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_serve_prompt_service and instance: <dbgpt.serve.prompt.service.service.Service object at 0x7fda59dab4f0>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_serve_conversation and instance: <dbgpt.serve.conversation.serve.Serve object at 0x7fda59da9cc0>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_serve_conversation_service and instance: <dbgpt.serve.conversation.service.service.Service object at 0x7fda59da9b70>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_serve_flow and instance: <dbgpt.serve.flow.serve.Serve object at 0x7fda5f22ec80>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_dbgpts_loader and instance: <dbgpt.util.dbgpts.loader.DBGPTsLoader object at 0x7fda59d14bb0>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_serve_flow_service and instance: <dbgpt.serve.flow.service.service.Service object at 0x7fda5f22d330>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_rag and instance: <dbgpt.serve.rag.serve.Serve object at 0x7fda59d49660>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_rag_service and instance: <dbgpt.serve.rag.service.service.Service object at 0x7fda59d49c00>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_datasource and instance: <dbgpt.serve.datasource.serve.Serve object at 0x7fda5647b5e0>
2024-05-11 02:54:12 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_datasource_service and instance: <dbgpt.serve.datasource.service.service.Service object at 0x7fda5647b5b0>
2024-05-11 02:54:13 6f5262b010bb dbgpt.util._db_migration_utils[1] WARNING Initialize and upgrade database metadata with alembic, just run this in your development environment, if you deploy this in production environment, please run webserver with --disable_alembic_upgrade(`python dbgpt/app/dbgpt_server.py --disable_alembic_upgrade`).
we suggest you to use `dbgpt db migration` to initialize and upgrade database metadata with alembic, your can run `dbgpt db migration --help` to get more information.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Context impl SQLiteImpl.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Will assume non-transactional DDL.
2024-05-11 02:54:13 6f5262b010bb dbgpt.util._db_migration_utils[1] INFO Migration versions and their file paths:
========================================Migration versions========================================

==========================================================================================
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Context impl SQLiteImpl.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Will assume non-transactional DDL.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Context impl SQLiteImpl.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Will assume non-transactional DDL.
2024-05-11 02:54:13 6f5262b010bb dbgpt.util._db_migration_utils[1] INFO alembic migration current revision: None, latest revision: None
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Context impl SQLiteImpl.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Will assume non-transactional DDL.


=========================== WebServerParameters ===========================

host: 0.0.0.0
port: 5670
daemon: False
controller_addr: None
model_name: chatglm3-6b
share: False
remote_embedding: False
log_level: INFO
light: False
log_file: dbgpt_webserver.log
tracer_file: dbgpt_webserver_tracer.jsonl
tracer_storage_cls: None
disable_alembic_upgrade: False
awel_dirs: None
default_thread_pool_size: None

======================================================================


2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Context impl SQLiteImpl.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Will assume non-transactional DDL.
2024-05-11 02:54:13 6f5262b010bb alembic.runtime.migration[1] INFO Running upgrade  -> 48fb9cd4d02f, New migration
2024-05-11 02:54:13 6f5262b010bb dbgpt.model.cluster.worker.manager[1] INFO Worker params:

=========================== ModelWorkerParameters ===========================

model_name: chatglm3-6b
model_path: /app/models/chatglm3-6b
worker_type: None
model_alias: None
worker_class: None
model_type: huggingface
host: 0.0.0.0
port: 5670
daemon: False
limit_model_concurrency: 5
standalone: True
register: True
worker_register_host: None
controller_addr: None
send_heartbeat: True
heartbeat_interval: 20
log_level: None
log_file: dbgpt_model_worker_manager.log
tracer_file: dbgpt_model_worker_manager_tracer.jsonl
tracer_storage_cls: None

======================================================================


2024-05-11 02:54:13 6f5262b010bb dbgpt.model.cluster.worker.manager[1] INFO Run WorkerManager with standalone mode, controller_addr: http://127.0.0.1:5670
2024-05-11 02:54:13 6f5262b010bb dbgpt.model.adapter.model_adapter[1] INFO Use fastcat adapter
2024-05-11 02:54:13 6f5262b010bb datasets[1] INFO PyTorch version 2.2.1 available.
2024-05-11 02:54:13 6f5262b010bb dbgpt.model.adapter.fschat_adapter[1] INFO Found llm model adapter with model name: chatglm3-6b, <fastchat.model.model_adapter.ChatGLMAdapter object at 0x7fda5d2570d0>
2024-05-11 02:54:13 6f5262b010bb dbgpt.model.cluster.worker.default_worker[1] INFO model_name: chatglm3-6b, model_path: /app/models/chatglm3-6b, model_param_class: <class 'dbgpt.model.parameter.ModelParameters'>
2024-05-11 02:54:13 6f5262b010bb dbgpt.model.cluster.worker.default_worker[1] INFO [DefaultModelWorker] Parameters of device is None, use cuda
2024-05-11 02:54:13 6f5262b010bb dbgpt.model.cluster.worker.manager[1] INFO Init empty instances list for chatglm3-6b@llm
2024-05-11 02:54:13 6f5262b010bb dbgpt.component[1] INFO Register component with name dbgpt_worker_manager_factory and instance: <dbgpt.model.cluster.worker.manager._DefaultWorkerManagerFactory object at 0x7fda5d3d1120>
2024-05-11 02:54:13 6f5262b010bb dbgpt.util.dbgpts.loader[1] INFO Found 0 dbgpts packages from /root/.dbgpts/packages/c1bc8c726812c791aab6cee0f17e157b
INFO:     Started server process [1]
INFO:     Waiting for application startup.
2024-05-11 02:54:13 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/simple_rag_summary_example.py
2024-05-11 02:54:13 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=dbgpt_awel_simple_rag_summary_example) from mod <module 'unusual_prefix_c8c5279ecb1d92b0fccc6a7371cb40387f616003_simple_rag_summary_example' from '/app/examples/awel/simple_rag_summary_example.py'> and model file /app/examples/awel/simple_rag_summary_example.py
2024-05-11 02:54:13 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/simple_chat_dag_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=dbgpt_awel_simple_dag_example) from mod <module 'unusual_prefix_63bbcb004427ba8dc7524807850d9bab727ccecf_simple_chat_dag_example' from '/app/examples/awel/simple_chat_dag_example.py'> and model file /app/examples/awel/simple_chat_dag_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/simple_dag_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=simple_dag_example) from mod <module 'unusual_prefix_9a5094f6509fa0069a76e97e88a6e48a0f78eed9_simple_dag_example' from '/app/examples/awel/simple_dag_example.py'> and model file /app/examples/awel/simple_dag_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/simple_rag_rewrite_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=dbgpt_awel_simple_rag_rewrite_example) from mod <module 'unusual_prefix_2f1504465f3e0b3dba7c69fc4a0c580a660b2056_simple_rag_rewrite_example' from '/app/examples/awel/simple_rag_rewrite_example.py'> and model file /app/examples/awel/simple_rag_rewrite_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/simple_nl_schema_sql_chart_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=simple_nl_schema_sql_chart_example) from mod <module 'unusual_prefix_7485301b58339cade91ea4fb8dd48f6099db388e_simple_nl_schema_sql_chart_example' from '/app/examples/awel/simple_nl_schema_sql_chart_example.py'> and model file /app/examples/awel/simple_nl_schema_sql_chart_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/simple_chat_history_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=dbgpt_awel_simple_chat_history) from mod <module 'unusual_prefix_a39637140c50dde6644b930c25547e238915aab0_simple_chat_history_example' from '/app/examples/awel/simple_chat_history_example.py'> and model file /app/examples/awel/simple_chat_history_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/simple_llm_client_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=dbgpt_awel_simple_llm_client_generate) from mod <module 'unusual_prefix_78a9e6ceb7c743b16f6bcaf923066ab214c0b6d4_simple_llm_client_example' from '/app/examples/awel/simple_llm_client_example.py'> and model file /app/examples/awel/simple_llm_client_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=dbgpt_awel_simple_llm_client_count_token) from mod <module 'unusual_prefix_78a9e6ceb7c743b16f6bcaf923066ab214c0b6d4_simple_llm_client_example' from '/app/examples/awel/simple_llm_client_example.py'> and model file /app/examples/awel/simple_llm_client_example.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Importing /app/examples/awel/data_analyst_assistant.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.dag.loader[1] INFO Found dag DAG(dag_id=dbgpt_awel_data_analyst_assistant) from mod <module 'unusual_prefix_054741f85836deef91cc721d98fe11ddd3f36aa6_data_analyst_assistant' from '/app/examples/awel/data_analyst_assistant.py'> and model file /app/examples/awel/data_analyst_assistant.py
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=952529b9-36e1-4365-b8e5-e6cea992a52f)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5b8cfd00>(AWEL_trigger_route__examples_rag_summary), endpoint: /examples/rag/summary, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/rag/summary
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=f4b036a9-d999-4a4a-9570-e19b2e5319da)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5a754160>(AWEL_trigger_route__examples_simple_chat), endpoint: /examples/simple_chat, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/simple_chat
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=8bac95af-3428-4796-8c94-b23077fb3439)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function_get at 0x7fda5a754700>(AWEL_trigger_route__examples_hello), endpoint: /examples/hello, methods: ['GET']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/hello
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=1d85c8ea-1e98-4750-a76e-9ab971bcd9ca)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5a754dc0>(AWEL_trigger_route__examples_rag_rewrite), endpoint: /examples/rag/rewrite, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/rag/rewrite
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=d841d94f-0f90-45e2-ac83-4dfb1c636f5b)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5a755630>(AWEL_trigger_route__examples_rag_schema_linking), endpoint: /examples/rag/schema_linking, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/rag/schema_linking
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=53a1036d-829d-4285-8474-66615c14ff63)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5a756050>(AWEL_trigger_route__examples_simple_history_multi_round_chat_completions), endpoint: /examples/simple_history/multi_round/chat/completions, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/simple_history/multi_round/chat/completions
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=0a9f2ed2-3b11-4445-a41b-78de830e4727)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5a756b00>(AWEL_trigger_route__examples_simple_client_chat_completions), endpoint: /examples/simple_client/chat/completions, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/simple_client/chat/completions
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=638d4687-8f56-4a76-8f59-041dc762f60e)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5a7576d0>(AWEL_trigger_route__examples_simple_client_count_token), endpoint: /examples/simple_client/count_token, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/simple_client/count_token
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Register trigger HttpTrigger(node_id=bb863c0d-6b53-49bf-bbbe-e6a5d5c9de0f)
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO mount router function <function HttpTrigger._create_route_func.<locals>.create_route_function.<locals>.route_function at 0x7fda5a7904c0>(AWEL_trigger_route__examples_data_analyst_copilot), endpoint: /examples/data_analyst/copilot, methods: ['POST']
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.http_trigger[1] INFO Mount http trigger success, path: /api/v1/awel/trigger/examples/data_analyst/copilot
2024-05-11 02:54:14 6f5262b010bb dbgpt.core.awel.trigger.trigger_manager[1] INFO Include router <fastapi.routing.APIRouter object at 0x7fda59dab010> to prefix path /api/v1/awel/trigger
2024-05-11 02:54:14 6f5262b010bb dbgpt.model.cluster.worker.manager[1] INFO Begin start all worker, apply_req: None
2024-05-11 02:54:14 6f5262b010bb dbgpt.model.cluster.worker.manager[1] INFO Apply req: None, apply_func: <function LocalWorkerManager._start_all_worker.<locals>._start_worker at 0x7fda5a792050>
2024-05-11 02:54:14 6f5262b010bb dbgpt.model.cluster.worker.manager[1] INFO Apply to all workers
INFO:     Application startup complete.
2024-05-11 02:54:14 6f5262b010bb dbgpt.model.cluster.worker.default_worker[1] INFO Begin load model, model params:

=========================== ModelParameters ===========================

model_name: chatglm3-6b
model_path: /app/models/chatglm3-6b
device: cuda
model_type: huggingface
prompt_template: None
max_context_size: 4096
num_gpus: None
max_gpu_memory: None
cpu_offloading: False
load_8bit: True
load_4bit: False
quant_type: nf4
use_double_quant: True
compute_dtype: None
trust_remote_code: True
verbose: False

======================================================================


INFO:     Uvicorn running on http://0.0.0.0:5670 (Press CTRL+C to quit)
2024-05-11 02:54:14 6f5262b010bb dbgpt.model.adapter.loader[1] WARNING chatglm or chatglm2 not support quantization now, see: https://github.com/huggingface/transformers/issues/25228
2024-05-11 02:54:14 6f5262b010bb transformers_modules.chatglm3-6b.tokenization_chatglm[1] WARNING Setting eos_token is not supported, use the default one.
2024-05-11 02:54:14 6f5262b010bb transformers_modules.chatglm3-6b.tokenization_chatglm[1] WARNING Setting pad_token is not supported, use the default one.
2024-05-11 02:54:14 6f5262b010bb transformers_modules.chatglm3-6b.tokenization_chatglm[1] WARNING Setting unk_token is not supported, use the default one.
Generating /app/pilot/meta_data/alembic/versions/48fb9cd4d02f_new_migration.py ...  done
Model Unified Deployment Mode!
Warning: Loading base MPT model with `zero_shot` conversation configuration.  If this is not desired, inspect model configurations and names.
INFO:     172.23.128.1:63489 - "GET /chat/?scene=chat_normal&id=ee6ba486-0f3e-11ef-973e-0242ac110002 HTTP/1.1" 200 OK
Loading checkpoint shards:  71%|███████▏  | 5/7 [00:18<00:07,  3.65s/it]2024-05-11 02:54:34 6f5262b010bb dbgpt.app.openapi.api_v1.api_v1[1] INFO /controller/model/types
2024-05-11 02:54:34 6f5262b010bb dbgpt.model.cluster.controller.controller[1] INFO Get all instances with None, healthy_only: True
INFO:     172.23.128.1:63489 - "GET /api/v1/chat/dialogue/list HTTP/1.1" 200 OK
INFO:     172.23.128.1:63490 - "GET /api/v1/model/types HTTP/1.1" 200 OK
INFO:     172.23.128.1:63490 - "POST /prompt/list HTTP/1.1" 200 OK
INFO:     172.23.128.1:63489 - "POST /api/v1/chat/mode/params/list?chat_mode=chat_normal HTTP/1.1" 200 OK
INFO:     172.23.128.1:63495 - "GET /api/v1/feedback/select HTTP/1.1" 200 OK
INFO:     172.23.128.1:63494 - "GET /api/v1/chat/dialogue/messages/history?con_uid=ee6ba486-0f3e-11ef-973e-0242ac110002 HTTP/1.1" 200 OK
INFO:     172.23.128.1:63494 - "POST /prompt/list HTTP/1.1" 200 OK
INFO:     172.23.128.1:63495 - "POST /api/v1/chat/dialogue/scenes HTTP/1.1" 200 OK
Loading checkpoint shards: 100%|██████████| 7/7 [00:23<00:00,  3.42s/it]
/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:4193: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead
  warnings.warn(
INFO:     172.23.128.1:63495 - "POST /api/v1/chat/dialogue/new HTTP/1.1" 200 OK
2024-05-11 02:54:40 6f5262b010bb dbgpt.app.openapi.api_v1.api_v1[1] INFO get_chat_instance:conv_uid='cf99d444-0f41-11ef-a6fb-0242ac110002' user_input='Hi' user_name=None chat_mode='chat_normal' select_param='' model_name=None incremental=False sys_code=None
chat_completions:chat_normal,,None
INFO:     172.23.128.1:63495 - "POST /prompt/list HTTP/1.1" 200 OK
INFO:     172.23.128.1:63499 - "POST /api/v1/chat/mode/params/list?chat_mode=chat_normal HTTP/1.1" 200 OK
INFO:     172.23.128.1:63500 - "GET /api/v1/feedback/select HTTP/1.1" 200 OK
2024-05-11 02:54:40 6f5262b010bb dbgpt.core.awel.runner.local_runner[1] INFO Begin run workflow from end operator, id: 0bece5af-e252-4b59-8155-ec286ba69ac4, runner: <dbgpt.core.awel.runner.local_runner.DefaultWorkflowRunner object at 0x7fda59dab340>
Get prompt template of scene_name: chat_normal with model_name: chatglm3-6b, proxyllm_backend: None, language: zh
INFO:     172.23.128.1:63494 - "POST /api/v1/chat/completions HTTP/1.1" 200 OK
2024-05-11 02:54:40 6f5262b010bb dbgpt.core.awel.runner.local_runner[1] INFO Begin run workflow from end operator, id: 26a310f9-04e9-48df-9f42-3ef33ca6385a, runner: <dbgpt.core.awel.runner.local_runner.DefaultWorkflowRunner object at 0x7fda59dab340>
2024-05-11 02:54:40 6f5262b010bb dbgpt.app.scene.base_chat[1] INFO payload request:
ModelRequest(model='chatglm3-6b', messages=[ModelMessage(role='system', content='你是一个有用的 AI 助手。', round_index=0), ModelMessage(role='human', content='Hi', round_index=0)], temperature=0.6, max_new_tokens=1024, stop=None, stop_token_ids=None, context_len=None, echo=False, span_id='c94dfe8b-4d91-4ff0-b74b-66bd1b6a39d1:0e0d636d-6790-4ea2-93c0-63a91bd9d5fc', context=ModelRequestContext(stream=True, cache_enable=False, user_name=None, sys_code=None, conv_uid=None, span_id='c94dfe8b-4d91-4ff0-b74b-66bd1b6a39d1:0e0d636d-6790-4ea2-93c0-63a91bd9d5fc', chat_mode='chat_normal', chat_param=None, extra={}, request_id=None))
2024-05-11 02:54:40 6f5262b010bb dbgpt.core.awel.runner.local_runner[1] INFO Begin run workflow from end operator, id: 5df5c2db-dc6c-440d-aa04-2b61bc50771c, runner: <dbgpt.core.awel.runner.local_runner.DefaultWorkflowRunner object at 0x7fda59dab340>
2024-05-11 02:54:40 6f5262b010bb dbgpt.core.awel.operators.common_operator[1] INFO branch_input_ctxs 0 result None, is_empty: False
2024-05-11 02:54:40 6f5262b010bb dbgpt.core.awel.operators.common_operator[1] INFO Skip node name llm_model_cache_node
2024-05-11 02:54:40 6f5262b010bb dbgpt.core.awel.operators.common_operator[1] INFO branch_input_ctxs 1 result True, is_empty: False
2024-05-11 02:54:40 6f5262b010bb dbgpt.core.awel.runner.local_runner[1] INFO Skip node name llm_model_cache_node, node id 49486efc-6cdb-412b-b8a3-ed3d358c8da1
2024-05-11 02:54:40 6f5262b010bb dbgpt.model.adapter.base[1] INFO Message version is v2
2024-05-11 02:54:40 6f5262b010bb dbgpt.model.cluster.worker.default_worker[1] ERROR Model inference error, detail: Traceback (most recent call last):
  File "/app/dbgpt/model/cluster/worker/default_worker.py", line 158, in generate_stream
    for output in generate_stream_func(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 35, in generator_context
    response = gen.send(None)
  File "/usr/local/lib/python3.10/dist-packages/fastchat/serve/inference.py", line 86, in generate_stream
    if tokenizer.eos_token_id not in stop_token_ids:
AttributeError: 'NoneType' object has no attribute 'eos_token_id'

llm_adapter: FastChatLLMModelAdapterWrapper(fastchat.model.model_adapter.ChatGLMAdapter)