如何调整参数，让知识库问答接口支持更多的并发

[xqxls]

如何调整参数，让知识库问答接口支持更多的并发。普通模式和使用vllm加速模式下，分别如何调整。

vllm主要参数：

    "host": DEFAULT_BIND_HOST,
    "port": 30002,
    "device": LLM_DEVICE,
    # False,'vllm',使用的推理加速框架,使用vllm如果出现HuggingFace通信问题，参见doc/FAQ
    # vllm对一些模型支持还不成熟，暂时默认关闭
    "infer_turbo": 'vllm',
    "max_parallel_loading_workers":1,
   "enforce_eager":False,
   "max_context_len_to_capture":2048,
   "max_model_len":2048,

    # model_worker多卡加载需要配置的参数

    # 以下为model_worker非常用参数，可根据需要配置
    # "load_8bit": False, # 开启8bit量化
    # "cpu_offloading": None,
    # "gptq_ckpt": None,
    # "gptq_wbits": 16,
    # "gptq_groupsize": -1,
    # "gptq_act_order": False,
    # "awq_ckpt": None,
    # "awq_wbits": 16,
    # "awq_groupsize": -1,
    # "model_names": LLM_MODELS,
    # "conv_template": None,
    # "limit_worker_concurrency": 5,
    # "stream_interval": 2,
    # "no_register": False,
    # "embed_in_truncate": False,

    # 以下为vllm_worker配置参数,注意使用vllm必须有gpu，仅在Linux测试通过

    # tokenizer = model_path # 如果tokenizer与model_path不一致在此处添加
    'tokenizer':'/opt3/models/models/Qwen-7B-Chat',
    'tokenizer_mode':'auto',
    'trust_remote_code':True,
    'download_dir':None,
    'load_format':'auto',
    'dtype':'auto',
    'seed':0,
    'worker_use_ray':False,
    'pipeline_parallel_size':1,
    'tensor_parallel_size':1,
    'block_size':16,
    'swap_space':4 , # GiB
    'gpu_memory_utilization':0.90,
    'max_num_batched_tokens':5120,
    'max_num_seqs':512,
    'disable_log_stats':False,
    'conv_template':None,
    'limit_worker_concurrency':5,
    'no_register':False,
    'num_gpus': 7,
    'engine_use_ray': False,
    'disable_log_requests': False

[dosubot[bot]]

To enhance the concurrency support of your knowledge base Q&A interface, especially under VLLM acceleration, consider the following parameter adjustments:

1. max_parallel_loading_workers: Increase this value to allow more data to be loaded in parallel, improving concurrent request handling. For example, changing it from 1 to a higher value suitable for your hardware.

   "max_parallel_loading_workers": 2  # Adjust based on hardware capabilities

2. num_gpus for VLLM Workers: If additional GPUs are available, incrementing num_gpus enables the VLLM worker to leverage more GPU resources, enhancing concurrent processing capabilities.

   "num_gpus": 8  # Based on available GPUs

3. gpu_memory_utilization: Increasing this allows the worker to utilize more GPU memory, potentially improving concurrency handling. Adjust cautiously to avoid resource contention.

   "gpu_memory_utilization": 0.95  # From 0.90 to a higher value, as per hardware

4. max_context_len_to_capture and max_model_len: Raising these limits allows for handling larger requests but requires more memory.

   "max_context_len_to_capture": 4096,
   "max_model_len": 4096,

5. load_8bit, cpu_offloading, gptq_ckpt: Adjusting these can further enhance concurrency handling. For instance, enabling 8-bit model loading if supported by your hardware, offloading computations to the CPU, and specifying a GPTQ checkpoint path.

   "load_8bit": True,
   "cpu_offloading": True,
   "gptq_ckpt": "/path/to/your/gptq/checkpoint",

These adjustments should be made with consideration to your hardware's capabilities and the specific requirements of your application. Testing changes in a controlled environment before production deployment is crucial to ensure system stability and performance.

Details

Sources

• [BUG] 多并发调用，偶尔出现因为embedding没加载而调用失败
• 你好，问一下各位，后端api部署的时候，支持多用户同时问答吗？？？
• startup.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.