Send batch of requests ASAP #3596

anterart · 2024-05-12T14:13:57Z

anterart
May 12, 2024

Hello,

I'm trying to figure out the correct way to use LiteLLM in my use case.
I work with many different LLM APIs supported by LiteLLM.
I want to send prompts to all of these APIs and get the results ASAP.
I'm using LiteLLM SDK to do it, more specifically the Router class, however I get RateLimitError or RateLimitError: Deployment over defined rpm limit on every API I try.
I'm aware of the rpm & tpm constraints for every API I use has and I define them in the configuration.
Also I'm setting the following arguments when creating the router:
router = Router(model_list=model_list, num_retries=3, cooldown_time=60, retry_after=10, routing_strategy="usage-based-routing", allowed_fails=5)
I also tried using "usage-based-routing"

Then I'm using the following code to execute the requests:

answers_dict = {}
model_names = ['gpt4-turbo']
for model_name in model_names:
    print(model_name)
    tasks = [router.acompletion(model=model_name, messages=message) for message in final]
    results = await tqdm.gather(*tasks)
    answers = [results[i].choices[0].message.content for i in range(len(results))]
    answers_dict[f'{model_name}'] = answers

Ideally I want to run the requests successfully as fast as possible and if there are any errors on the way, retry, cooldown a number of times.

Currently it doesn't look to me like it's obeying any of the keywords I set like cooldown, retry_after, num_retries & allowed_fails.
Am I doing something wrong? I will be happy for any advice.