Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Template import/export and defaults #395

Open
bannsec opened this issue May 3, 2024 · 1 comment
Open

Template import/export and defaults #395

bannsec opened this issue May 3, 2024 · 1 comment

Comments

@bannsec
Copy link

bannsec commented May 3, 2024

I think it would be helpful to be able to import/export templates. So when I load up to run a new model with llamafile, i can simply point it to some template file definition that contains all the required template fields for this specific model.

To take it a step further, it would be very helpful to have baked in templates. For instance, I was just playing with jart's Phi-3 llamafile. I'm still trying to convert what was mentioned as the template into the correct fields and stuff on the web interface. Instead, in the name of simplicity, what if the llamafile itself would have the appropriate template baked into it, so you literally just ./ run it and go. No need to configure basically anything.

Thanks for the great work folks!

@Skarian
Copy link

Skarian commented May 7, 2024

I agree the most challenging aspect now to run LLMs locally is to track which model has which settings and prompt template. There is a ton of confusion specifically right now with Llama 3 (ex: https://www.reddit.com/r/LocalLLaMA/comments/1ca06g6/what_sampler_settings_are_you_using_for_llama_3/)

A community sourced set of configurations distributed with the models via llamafile would be amazing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants