Tiktoken 0.7.0 read-only system error in AWS Lamda

[maurobender]

Using latest version from main with model gpt-4o throws the following error when running on AWS Lamda:

Failed to clip tokens: [Errno 30] Read-only file system: '/var/lang/lib/python3.10/site-packages/litellm/llms/tokenizers/fb374d419588a4632f3f557e76b4b70aebbca790.3633445e-4ab0-4767-8f6a-0cd5fd32eb79.tmp'\n

Stack trace

[ERROR]	2024-05-17T13:42:14.431Z	9c2023d9-ecea-430c-bb8c-b115540613c0	An error occurred running the application.
--
Traceback (most recent call last):
File "/var/lang/lib/python3.10/site-packages/mangum/protocols/http.py", line 58, in run
await app(self.scope, self.receive, self.send)
File "/var/lang/lib/python3.10/site-packages/fastapi/applications.py", line 290, in __call__
await super().__call__(scope, receive, send)
File "/var/lang/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "/var/lang/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/var/lang/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/var/lang/lib/python3.10/site-packages/starlette_context/middleware/raw_middleware.py", line 92, in __call__
await self.app(scope, receive, send_wrapper)
File "/var/lang/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/var/lang/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "/var/lang/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
raise e
File "/var/lang/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
await self.app(scope, receive, send)
File "/var/lang/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "/var/lang/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/var/lang/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/var/lang/lib/python3.10/site-packages/fastapi/routing.py", line 241, in app
raw_response = await run_endpoint_function(
File "/var/lang/lib/python3.10/site-packages/fastapi/routing.py", line 167, in run_endpoint_function
return await dependant.call(**values)
File "/var/task/pr_agent/servers/github_app.py", line 51, in handle_github_webhooks
response = await handle_request(body, event=request.headers.get("X-GitHub-Event", None))
File "/var/task/pr_agent/servers/github_app.py", line 273, in handle_request
await handle_comments_on_pr(body, event, sender, sender_id, action, log_context, agent)
File "/var/task/pr_agent/servers/github_app.py", line 117, in handle_comments_on_pr
await agent.handle_request(api_url, comment_body,
File "/var/task/pr_agent/agent/pr_agent.py", line 93, in handle_request
await command2class[action](pr_url, ai_handler=self.ai_handler, args=args).run()
File "/var/task/pr_agent/tools/pr_reviewer.py", line 75, in __init__
self.token_handler = TokenHandler(
File "/var/task/pr_agent/algo/token_handler.py", line 47, in __init__
self.encoder = TokenEncoder.get_token_encoder()
File "/var/task/pr_agent/algo/token_handler.py", line 19, in get_token_encoder
cls._encoder_instance = encoding_for_model(cls._model) if "gpt" in cls._model else get_encoding(
File "/var/lang/lib/python3.10/site-packages/tiktoken/model.py", line 103, in encoding_for_model
return get_encoding(encoding_name_for_model(model_name))
File "/var/lang/lib/python3.10/site-packages/tiktoken/registry.py", line 73, in get_encoding
enc = Encoding(**constructor())
File "/var/lang/lib/python3.10/site-packages/tiktoken_ext/openai_public.py", line 92, in o200k_base
mergeable_ranks = load_tiktoken_bpe(
File "/var/lang/lib/python3.10/site-packages/tiktoken/load.py", line 147, in load_tiktoken_bpe
contents = read_file_cached(tiktoken_bpe_file, expected_hash)
File "/var/lang/lib/python3.10/site-packages/tiktoken/load.py", line 74, in read_file_cached
with open(tmp_filename, "wb") as f:
OSError: [Errno 30] Read-only file system: '/var/lang/lib/python3.10/site-packages/litellm/llms/tokenizers/fb374d419588a4632f3f557e76b4b70aebbca790.e359bd26-b7da-4e24-9920-26bb937d278e.tmp'

requirements.txt

aiohttp==3.9.1
atlassian-python-api==3.41.4
azure-devops==7.1.0b3
azure-identity==1.15.0
boto3==1.33.6
dynaconf==3.2.4
fastapi==0.99.0
GitPython==3.1.32
google-cloud-aiplatform==1.35.0
google-cloud-storage==2.10.0
Jinja2==3.1.2
litellm==1.31.10
loguru==0.7.2
msrest==0.7.1
openai==1.13.3
pytest==7.4.0
PyGithub==1.59.*
PyYAML==6.0.1
python-gitlab==3.15.0
retry==0.9.2
starlette-context==0.3.6
tiktoken==0.7.0
ujson==5.8.0
uvicorn==0.22.0
tenacity==8.2.3

[maurobender]

I tried using the env variable TIKTOKEN_CACHE_DIR pointing to /tmp since it seems it can be used to setup the
temporary cache dir in here with no success, it still creates the temp file in the read only filesystem.

[mrT23]

We need the new Tiktoken for GPT-4o model, so this new requirement will stay.

You can always use a previous release of PR-Agent without the new Tiktoken, but this seems to be a problem of AWS lambda that should be solved, one way or another.
Maybe contact them, or try other workarounds.

If you do find a workaround, share and we will add it to the docs.

[maurobender]

I did some debugging trying to find a workaround and the only one I found was setting TIKTOKEN_CACHE_DIR but that didn't work because litellm is overwriting that environment variable (see this PR BerriAI/litellm#1947) so I'm unable to pass a writable directory for AWS Lamda.

This issue is clearly related to this other issue in litellm: BerriAI/litellm#2607.

I'll also report it in that issue to see if there is any fix coming. Until this is fixed I guess AWS Lamda users will not be able to use the latest gpt-4o model with the pr agent.