Replies: 1 comment
-
Try this one main.exe --model Meta-Llama-3-8B-Instruct.Q8_0.gguf --color --threads 30 --keep -1 --n-predict -1 --repeat-penalty 1.1 --ctx-size 0 --interactive -ins -ngl 99 --simple-io --in-prefix "<|start_header_id|>user<|end_header_id|>\n\n" --in-suffix "<|start_header_id|>assistant<|end_header_id|>\n\n" -p "<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability." -e --multiline-input --no-display-prompt --conversation You do not have to adding start / stop tokens as they are built in into ggml. |
Beta Was this translation helpful? Give feedback.
-
I can't force llama 3 (any GGUF) to work correctly with llama cpp. Either there is a bug in all the models or I can't realize how to make a correct prompt.
Here is example with Llama 3 8B Q4
./main -m ~/ai/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf -p "<s>You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.</s>[INST]Write me a detailed markdown article in Dutch on a subject: 5 Legit Reasons Divorced Men Are Great To Marry[/INST]" --in-prefix "<|start_header_id|>user<|end_header_id|> " --in-suffix " <|eot_id|><|start_header_id|>assistant<|end_header_id|> "
Here is another quant of a model i require
alex@M1 llama.cpp % ./main -m ~/ai/suzume-llama-3-8B-multilingual--Q4_K_M.gguf -p "<s>You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.</s>[INST]Write me a detailed markdown article in Dutch on a subject: 5 Legit Reasons Divorced Men Are Great To Marry[/INST]" --in-prefix "<|start_header_id|>user<|end_header_id|> " --in-suffix " <|eot_id|><|start_header_id|>assistant<|end_header_id|> "
But when I use the model in the instruct mode, It works appropriate:
I've been also trying the simple prompt right from the official Facebook page with the same result:
./main -m ~/ai/suzume-llama-3-8B-multilingual--Q4_K_M.gguf -p "<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Write me a detailed markdown article in Dutch on a subject: 5 Legit Reasons Divorced Men Are Great To Marry<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
Beta Was this translation helpful? Give feedback.
All reactions