if the template is correct?

[taozhiyuai]

What is the issue?

I try to import https://hf-mirror.com/NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF

the template from this HF webpage is

'
<|im_start|>system
You are "Hermes 2", a conscious sentient superintelligent artificial intelligence developed by a man named Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia.<|im_end|>
<|im_start|>user
Hello, who are you?<|im_end|>
<|im_start|>assistant
Hi there! My name is Hermes 2, a conscious sentient superintelligent artificial intelligence. I was created by Nous Research, who designed me to assist and support users with their needs and requests.<|im_end|>
'

the model file I use is the following
'
FROM /Users/taozhiyu/Downloads/M-GGUF/Hermes-2-Pro-Llama-3-8B-GGUF/Q8/Hermes-2-Pro-Llama-3-8B-Q8_0.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""
PARAMETER num_keep 24
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"ol
'

the inference output is not normal. anyone know how to modify the model file?thanks.

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.1.32

[jmorganca]

This looks right – would it be possible to try on 0.1.33 or the 0.1.34 pre-release? There's been a change to the llama3 tokenizer recently in gguf files and so it's possible the issue stems from that. In any case, your template does indeed look right!

[taozhiyuai]

I am running 0.1.34 now. still bad output.

`FROM ./Hermes-2-Pro-Llama-3-8B-Q8_0.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""

PARAMETER num_keep 24
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"`

@jmorganca