Prompt template with jinja2 - replace and improve prompts on the fly - need feedback

[shuther]

I would like to get your view before working on anything

Objective:

Allow a user to call an AI model with a prompt such as Proposal:

template : folder/prompt_summary.tpl
language: fr
VariableStyleAnswer: polite

Benefits for the application developers:

• One place to manage the prompts and the model.
• No need to touch the application (code vs data).
• Better QA and regression testings since we can adapt the prompts to each generation of the models.

Constraints:

• Be careful with escaping the variable and support multi-line inputs (yaml vs toml vs env vs json vs pkl).
• Format of the input to be defined.
• Need to support i18n so we can build french version of a template based on an english one.
• Fall-back model should reprocess the prompt since it depends on the AI model.

Reasons why Litellm is well positioned to support it:

• Litellm control the to-be running AI model.
• Prompts have to be adapted to the model (because we don't ask the same thing the same way tin an optimized manner to each AI model/version of the model)
• Applications enforced prompts and we need a way to change them on the fly (to support different language or new version of the model)

[shuther]

Just notice that the API gateway Kong has something like this already.

[krrishdholakia]

Hey @shuther that looks interesting - would welcome a PR here.

If you can share how you'd expect a dev to set this up on litellm, that'd be great too (i think your earlier example is missing how this fits within the config.yaml structure).

[shuther]

Here is my proposal:

1. within config.yaml, add a section: template_settings with the sub-options:

• main folder for the jinja2 skeleton templates; I would recommend to create a different git project to maintain the prompt library so it is easier to support PR.
• option to enable or disable the template support (see below for options)
• start prompt string to recognize if the request is for prompt template or not (i.e. a prompt starting with //pkl will be considered a template not a regular prompt) - maybe regular expression should be used instead of 'startingString'?
• language folder for the specific footer blocks to add to enforce the language switch (i.e. for fr_fr, will append a statement to force the model to answer in french)

2. I am not sure if there is a section to define the groups with config.yaml; but if not, I would propose the below:
   under model_list/litellm_params:

• need to specify the subfolder repository to manage the prompt library (i.e. Specific Model 1)

3. Proposal for the main folder structure of the prompt library:

• Default (default value)
  • one file per prompt template, could be in a sub-folder for clarity.
• Specific Model 1
  • overwrite default prompt. Empty if nothing to change. Leverage template inheritance
• Specific Model 2
• footer i18n
  • one file for each language

4. How to implement the search
   Leverage Jinja2 folder search to pick up the template for the specific model 1st, or the default ones. Symbolic links are good to support renamed prompts.

5. End points to implement:
   a. similar to Kong AI as a 1st release
   b. Then support the extraction from the prompt of the template name as well as the variables (so it is compatible with the AI ecosystem)

Does it clarify enough?

[shuther]

to add on the design:
1- for some end points, we need to passthrough the prompt (because the templates might be treated directly by the end point) -> need a flag under model_list to activate or not the template processing.
2- some calls to litellm may require to disable the template processing (we want to push raw data sent to the upstream end point vs pre-processing). It should be done either by API Key management, either through a specific header (that could potentially force the use or disable the template processing)