-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Llama 3 Preferred RAG Prompting Format (xml tags vs. markdown vs. something else) #450
Comments
There's no mention of a preferred format for Llama 3. According to the Llama 3 model card prompt format, you just need to follow the new Llama 3 format there (also specified in HF's blog here), but if you use a framework LangChain or service provider like Groq/Replicate or run Llama 3 locally using Ollama for your RAG apps, most likely you won't need to deal with the new prompt format directly as it's been hardcoded by them under the hood. Just use an appropriate RAG prompt (e.g. rag-prompt) with your question, context and possibly chat history for Llama 3 to answer. |
Thank you, @jeffxtang! I am aware of the new prompt format. I was asking more about model preferences regarding RAG type of prompts and longer input prompts. Example 1 - XML tags (aka the way Anthropic recommends their models to be prompted):
Example 2 (formatted as markdown):
Example 3 (special tokens, like for example - https://huggingface.co/jondurbin/airoboros-l2-c70b-3.1.2):
Is there a format that Llama-3 Instruct models prefer? |
I'm not aware of such preference for Llama 3, but it should be easy, with some automated RAG evaluation frameworks (there're quite a few nice open source frameworks), to compare the results of the example with different formats and see if there's any quality difference. @krumeto |
Hi @krumeto, were you able to find out what works best with Llama-3 ? |
Hey @trivikramak, no, not yet (apologies). |
Ask it and see what it says. Try some different stuff. Interesting question. |
馃殌 The feature, motivation and pitch
Anthropic directly states that their models prefer context for longer prompts (like the usual RAG applications) to be inserted in XML tags. Some claim OpenAI's models prefer markdown-style (their docs mention both markdown and XML tags).
Does Llama 3 have a preferred format for longer prompts?
Thank you in advance!
Alternatives
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: