feat: openai api compatible model presets (profiles)

[keriati]

Is your feature request related to a problem? Please describe.
As the modelfiles from the hub are pretty good, it would be nice if we could use also them with external models from openAI api compatible services.

Describe the solution you'd like
Option A:
When adding a modelfile, we could check that it should use a model from the external provider and select it.
Option B:
When selecting a modelfile, we should be able to exchange then the actual model to one from the external service.

Describe alternatives you've considered
Setting the default prompt according to the modelfile works for now, just the profile pick and name is wrong and the extra settings.

Additional context
My ollama instance has limited vram available. External services can provide bigger models. I think it would be good if the same characters would be usable when using an external service.

[explorigin]

Ollama.ai is not directly associated with this project. Your feature request belongs with them.

[keriati]

Thanks for taking a look. I already implemented this and it works for me. See the referenced PR.

[explorigin]

OK first, thanks for adding features. It's awesome to have a community that's adding to a project.

Per the PR, I have some suggestions that I'll post over there.

[explorigin]

Changed my mind, I'll post here:

Modelfiles are an Ollama.ai-specific feature to describe settings that belong to a specific model. OpenAI doesn't have such a thing because most of the provided functionality it is wrapped up in the product offering. Of course we live in a world where the OpenAI API is becoming the standard even for interacting with custom models and local LLM servers. That leaves a functionality hole that you're trying to address here.

However, I think the over-arching feature request (added metadata for non-local, non-ollama.ai models) needs to be thought of a bit more holistically. For example, in the UI, we set an Ollama server and optionally and OpenAI key and endpoint. the WebUI then gives you access to both.

I think the right way to solve this issue is to unify the idea of external LLM servers much like how LiteLLM handles it. (That and even Ollama.ai is working on adopting OpenAI's API.) Your PR provides metadata to all in an agnostic way but maintains the type-difference segregation.

The whole idea of multiple models/servers/configurations is tied up in other feature requests that should be considered since they all would (should?) share the same solution: #278 #619

[explorigin]

I also feel like the Modelfile format is the wrong format for its purpose. It's patterned after Dockerfile (possibly since the ollama.ai author previously worked on docker). But Dockerfile is the way it is because each line represents a progression. This is not a thing in Modelfile. I feel like we should have our own custom file to manage this metadata. TOML or YAML (but I haven't put too much thought into it).

[justinh-rahb]

@explorigin, you've precisely captured the essence of the thoughts I've been struggling to express. Modelfile is designed with Ollama's specific needs and objectives in mind. Merely suggesting to incorporate OpenAI into it without a thorough strategy might not be the most effective approach to address the underlying issue.

[keriati]

Thanks guys for thinking about this.
Stepping back a bit, here is what I actually wanted to achieve (with my patch I actually also achieved it):

To get a character from ollamahub, but use it with an openAI API (in my case together.ai)
The main thing I wanted to have was the system prompt, I was not so much interested in the specific parameters.
And obviously the character name and profile picture.

So maybe the request is this:

Make ollamahub models also usable with openAI API compatible services, not just ollama.

What do you think?

[explorigin]

Ollamahub is supposed to provide access to anything that the user can download via Ollama.ai...so allowing it to reference any custom model will likely lead to a bad user-experience if that model in question is not available. We certainly don't want modelfiles pointing to together.ai endpoints.

LMStudio references models by huggingface path (e.g. TheBloke/CapybaraHermes-2.5-Mistral-7B-GGUF). This feels like the right way to go w.r.t. applying specific settings to specific models. It's quantization-agnostic while remaining model-specific. Ollama.ai tries a little too hard to be docker with it's :latest tag that you have to chase hashes to find what that even means. But I digress:

So what we want is a sort of configuration file that:

• applies to model regardless of source
• sets custom prompts
• provides a character name and profile pic
• provides sensible defaults for given models

Anything else?

Definitely think we should consider what's in LMStudio's Model configuration dialog.

[justinh-rahb]

Absolutely, I concur with everything above. However, I believe it's crucial that we extend our exploration to the broader community to ensure we're not overlooking any existing generalized solutions to this problem that we could potentially endorse. Let's avoid bike-shedding another standard if possible...

(Obligatory xkcd reference: https://xkcd.com/927/)

[explorigin]

It's not a standard if it's not applied everywhere. 😁 # 15 here we come!

[tjbck]

TODO

• Decouple modelfiles from Ollama
• Save model preset to webui db directly
• (optional) unify model list endpoint to /models

[lee-b]

I feel like trying to find/create a new general model format for everything is boiling the ocean here. Essentially there are two problems, that I can see:

1. Despite the shift from Ollama-webui to open-webui, Ollama is still treated first-class model provider (with more features implemented for it), whereas others like OpenAI are second-class (you can't seem to ONLY use an OpenAI API for example; Ollama seems to be required even if you want to use an OpenAI API).

2. Model files assume that the model that you're referring to with the FROM line are provided by ollama.

I think that:

• changing the UI to have a list of named providers and a default provider would solve # 1.

• changing the interpretation of the FROM line to resolve the model using the default provider, but use a named provider if FROM named_provider:model_name_or_path is used (rather than simply model_name_or_path) would solve # 2.