[Feature]: Support passing in api keys as an array (Router)

[krrishdholakia]

The Feature

support passing in api keys as an array to the router

Motivation, pitch

Twitter / LinkedIn details

No response

[paneru-rajan]

For the above use case, how we are thinking about using the api_keys from the array?

Random: This could use the one just has been rate limited

How we are thinking about solving it? Or we will leave as it is because we retry anyway and random will pick the another key eventually?

@krrishdholakia I am interested on working on it, can you assign it to me.

My understanding of the problem so that i can have a proper fix:

{
  "model_name": "....",
  "litellm_params": {
      "model": "....",
      "api_key": <your-api-key>, # this can be either array or string, if array a random value will be chosen 
  }
}

[paneru-rajan]

Changing the api_key's dataType fromstr to list[str] might have long lasting implications. It is not usually good idea to support like this (based on my experience).

I think if user catch and handle RateLimitError, this can be done without affecting the litellm codebase.

This is just me suggesting, i am still learning and trying litellm, so my answer might not make sense, guidance is appreciated.

Sample Code:

import asyncio
import random

from litellm import RateLimitError, acompletion

API_KEYS = ["key-1", "key-2", "key-3", "key-4"]
LAST_KEY = ""


def get_next_key():
    global LAST_KEY

    new_key = ''
    while new_key == LAST_KEY:
        new_key = random.choice(API_KEYS)
    LAST_KEY = new_key
    return new_key


async def chat():
    api_key = get_next_key()
    try:
        return await acompletion(
            model="ollama_chat/llama3:instruct",
            api_key=api_key,
            messages=[
                {"role": "user", "content": "Hello"},
            ],
            base_url='http://localhost:11434'
        )
    except RateLimitError:
        return await chat()


if __name__ == '__main__':
    resp = asyncio.run(chat())
    print(resp["choices"][0]["message"])