You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
name: llama3-70b-chatQAmmap: truecontext_size: 8192#threads: 11#gpu_layers: 90f16: trueparameters:
model: Llama3-ChatQA-1.5-70B-Q4_K_M.gguffunction:
# set to true to allow the model to call multiple functions in parallelparallel_calls: truetemplate:
chat_message: | <|start_header_id|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}<|end_header_id|> {{ if .FunctionCall -}} Function call: {{ else if eq .RoleName "tool" -}} Function response: {{ end -}} {{ if .Content -}} {{.Content -}} {{ else if .FunctionCall -}} {{ toJson .FunctionCall -}} {{ end -}} <|eot_id|>function: | <|start_header_id|>system<|end_header_id|> You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> {{range .Functions}} {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }} {{end}} </tools> Use the following pydantic model json schema for each tool call you will make: {'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}<|eot_id|><|start_header_id|>assistant<|end_header_id|> Function call:chat: | <|begin_of_text|>{{.Input }} <|start_header_id|>assistant<|end_header_id|>completion: | {{.Input}}stopwords:
- <|im_end|>
- <dummy32000>
- <|eot_id|>
- <|end_of_text|>usage: | curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "llama3-70b-chatQA", "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}] }'
The text was updated successfully, but these errors were encountered:
Hi @WuQic I tested this model in transformer backend with the OpenVINO version.
Was not particularly impressed, If you want to give a try this is the model definition.
use this config can't answer the question
The text was updated successfully, but these errors were encountered: