What's a cheap way to deploy my LiteLLM proxy in a U.S. based location?

[twardoch]

What's a cheap (or free) way to deploy my LiteLLM proxy in a U.S. based location? I'm planning to use it for just a few hundred or a couple of thousand accesses a day for now, but my tasks are pretty complex & my orchestration requires use of multiple providers.

[Manouchehri]

We're a big fan of Google Cloud Run.

[ishaan-jaff]

Link to deploying on platforms for anyone reading: https://docs.litellm.ai/docs/proxy/deploy#platform-specific-guide

[whalygood]

@Manouchehri Almost certain I'm making an obvious mistake here, but can't get the litellm container to start in Cloud Run for the life of me. Trying to integrate with the simplest AI provider (OpenAI) but eventually plan to use other AIs such as Anthropic, which I suppose I'll just have to pass a few more env variables for. Am using the following command:

 gcloud run deploy litellm\
          --project=project-name-123456\
          --platform=managed\
          --region=us-east1\
          --set-env-vars="OPENAI_API_KEY=sk-123456"\
          --image=gcr.io/repository-name-123456/litellm\
          --allow-unauthenticated

The error in the Cloud Shell is:

ERROR: (gcloud.run.deploy) Revision 'litellm-00013-j7j' is not ready and cannot serve traffic. The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.

The error in the Log Explorer is:

openai.OpenAIError: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, or the AZURE_OPENAI_API_KEYorAZURE_OPENAI_AD_TOKEN environment variables.

[ishaan-jaff]

@whalygood this is because the default litell config requires the following keys in the environment. Can you add the following variables to your environment and try ? Evan example / fake values would work
OPENAI_API_KEY
AZURE_API_KEY
AZURE_API_BASE

This something we can improve on litellm in order to get started. Can you let me know if this works @whalygood ?

[krrishdholakia]

@whalygood try mounting your own config. That should fix it

[whalygood]

After some tinkering I was able to get this working with Azure OpenAI.

I had to pass the --port=4000 and --memory=2048M arguments to the gcloud run deploy command, as otherwise the container would exceed the default 512MiB memory limit of Cloud Run when spinning up (with the Virtual Keys feature) and the ingress would assume a port of 8080 when in fact LiteLLM listens on port 4000 by default.

Also, I think it's a good idea to recommend users to use the Virtual Keys feature to protect others from running up charges against their external AI endpoint. Because by default the one click Cloud Run deploy passes the --allow-unauthenticated flag.

gcloud run deploy litellm\
          --project=project-id-123456\
          --platform=managed\
          --region=us-east1\
          --set-env-vars="AZURE_API_KEY=foo,AZURE_API_BASE=https://resource-name.openai.azure.com,AZURE_API_VERSION=2024-02-15-preview,DATABASE_URL=postgresql://postgres:foo@db-resource-name.postgres.database.azure.com/postgres,LITELLM_MASTER_KEY=sk-foo"\
          --args="--api_base=https://resource-name.openai.azure.com/openai/deployments/gpt-35-turbo-16k,--model=azure/gpt-35-turbo-16k,--debug"\
          --image=gcr.io/project-id-123456/litellm:main-latest\
          --memory=2048M\
          --port=4000\
          --allow-unauthenticated

[ishaan-jaff]

@twardoch We use Railway https://railway.app/dashboard, it's been about $100/mo on our endpoints with low overhead. I'd love top hop on a call and learn how we can improve litellm for you @twardoch. Sharing a link to my cal for your convenience https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat