VertexAIException - Expecting value: line 1 column 1 (char 0) on Cloud Run

[whalygood]

Deploying a LiteLLM proxy on Cloud Run with the following gcloud run command and calling it with curl using a Virtual Key results in a 500 error with Claude 3 Opus on the Vertex AI provider (may or may not be related to #3141).

The service account has the Vertex AI role properly assigned.

Deployment command:

gcloud run deploy litellm\
          --project=project-id-123456\
          --platform=managed\
          --service-account=vertex-ai@project-id-123456.iam.gserviceaccount.com\
          --args="--model=vertex_ai/claude-3-opus@20240229,--drop_params,--debug"\
          --memory=2048M\
          --port=4000\
          --set-env-vars="VERTEXAI_PROJECT=project-id-123456,VERTEXAI_LOCATION=us-central1,DATABASE_URL=postgresql://postgres:postgresPass@resourceName.postgres.database.azure.com/postgres,LITELLM_MASTER_KEY=sk-1a2b3c4"\
          --image=gcr.io/project-id-123456/litellm:main-latest\
          --allow-unauthenticated\
          --region=us-central1

cURL command and output:

curl --location 'https://litellm-foo.a.run.app/chat/completions' \
  --header 'Authorization: Bearer sk-2b3c4d5' \
  --header 'Content-Type: application/json' \
  -d '{
     "model": "vertex_ai/claude-3-opus@20240229",
     "messages": [
         {
         "role": "user",
         "content": "what llm are you"
         }
     ],
   }'
{"error":{"message":"VertexAIException - VertexAIException - Expecting value: line 1 column 1 (char 0)\nmodel: vertex_ai/claude-3-opus@20240229\n\nmodel: vertex_ai/claude-3-opus@20240229\n","type":null,"param":null,"code":500}}

Here are the tracebacks from the Log Explorer for the Cloud Run container:

ERROR 2024-04-18T22:43:13.329448Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py", line 240, in completion json_obj = json.loads(vertex_credentials)
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py\", line 240, in completion\n    json_obj = json.loads(vertex_credentials)",
    "insertId": "6621a201000506e85094c2f4",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "litellm",
        "location": "us-central1",
        "project_id": "project-id-123456",
        "revision_name": "litellm-00022-j8b",
        "configuration_name": "litellm"
      }
    },
    "timestamp": "2024-04-18T22:43:13.329448Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CJr7pZHzgp3dnwE"
      }
    ]
  }
ERROR 2024-04-18T22:43:13.329560Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 1686, in completion model_response = vertex_ai_anthropic.completion(
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 1686, in completion\n    model_response = vertex_ai_anthropic.completion(",
    "insertId": "6621a20100050758f80626f2",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "project-id-123456",
        "revision_name": "litellm-00022-j8b",
        "location": "us-central1",
        "configuration_name": "litellm",
        "service_name": "litellm"
      }
    },
    "timestamp": "2024-04-18T22:43:13.329560Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CPX1jNCzkNjP4AE"
      }
    ]
  }
ERROR 2024-04-18T22:43:13.329755Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 313, in acompletion init_response = await loop.run_in_executor(None, func_with_context)
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 313, in acompletion\n    init_response = await loop.run_in_executor(None, func_with_context)",
    "insertId": "6621a2010005081b752446b3",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "location": "us-central1",
        "revision_name": "litellm-00022-j8b",
        "service_name": "litellm",
        "configuration_name": "litellm",
        "project_id": "project-id-123456"
      }
    },
    "timestamp": "2024-04-18T22:43:13.329755Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CLLXm_CVkNCQfw"
      }
    ]
  }
ERROR 2024-04-18T22:43:13.329912Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 3655, in chat_completion responses = await asyncio.gather(
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py\", line 3655, in chat_completion\n    responses = await asyncio.gather(",
    "insertId": "6621a201000508b8e80a1ba9",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "litellm",
        "revision_name": "litellm-00022-j8b",
        "project_id": "project-id-123456",
        "service_name": "litellm",
        "location": "us-central1"
      }
    },
    "timestamp": "2024-04-18T22:43:13.329912Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CKrC8bbjqYbsWQ"
      }
    ]
  }
ERROR 2024-04-18T22:43:13.337492Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py", line 240, in completion json_obj = json.loads(vertex_credentials)
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py\", line 240, in completion\n    json_obj = json.loads(vertex_credentials)",
    "insertId": "6621a20100052654517cdb44",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "project-id-123456",
        "service_name": "litellm",
        "location": "us-central1",
        "revision_name": "litellm-00022-j8b",
        "configuration_name": "litellm"
      }
    },
    "timestamp": "2024-04-18T22:43:13.337492Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CJr7pZHzgp3dnwE"
      }
    ]
  }
ERROR 2024-04-18T22:43:13.337601Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 1686, in completion model_response = vertex_ai_anthropic.completion(
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 1686, in completion\n    model_response = vertex_ai_anthropic.completion(",
    "insertId": "6621a201000526c1e337abfa",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "project-id-123456",
        "configuration_name": "litellm",
        "revision_name": "litellm-00022-j8b",
        "location": "us-central1",
        "service_name": "litellm"
      }
    },
    "timestamp": "2024-04-18T22:43:13.337601Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CPX1jNCzkNjP4AE"
      }
    ]
  }
ERROR 2024-04-18T22:43:13.337787Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 313, in acompletion init_response = await loop.run_in_executor(None, func_with_context)
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 313, in acompletion\n    init_response = await loop.run_in_executor(None, func_with_context)",
    "insertId": "6621a2010005277b5e22d6aa",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "project-id-123456",
        "configuration_name": "litellm",
        "revision_name": "litellm-00022-j8b",
        "service_name": "litellm",
        "location": "us-central1"
      }
    },
    "timestamp": "2024-04-18T22:43:13.337787Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CLLXm_CVkNCQfw"
      }
    ]
  }
ERROR 2024-04-18T22:43:13.338105Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 3655, in chat_completion responses = await asyncio.gather(
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py\", line 3655, in chat_completion\n    responses = await asyncio.gather(",
    "insertId": "6621a201000528b9cd6d72bd",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "litellm",
        "location": "us-central1",
        "project_id": "project-id-123456",
        "revision_name": "litellm-00022-j8b",
        "service_name": "litellm"
      }
    },
    "timestamp": "2024-04-18T22:43:13.338105Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00a22404dcf014e357743dffc2cb8f56dd41dc3a6e5fbe68d49fbb68169f73620656f5fc3daa51d1b0f4aa83509fc954528495826d52b320a7721132d92cdd11504404"
    },
    "logName": "projects/project-id-123456/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2024-04-18T22:43:13.527933683Z",
    "errorGroups": [
      {
        "id": "CKrC8bbjqYbsWQ"
      }
    ]
  }

INFO:     169.254.1.1:64079 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error