Issues: openai/tiktoken
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Tiktoken educational BPE trainer takes long time to train with vocab size 30k
#299
opened May 19, 2024 by
sagorbrur
Use a custom exception ValueError subclass for the special tokens warning
#290
opened May 2, 2024 by
simonw
Custom tokenizer fails to encode despite characters being in mergeable_ranks
#289
opened May 2, 2024 by
afang-story
Exception has occurred: ConnectionError HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001F4D42B0EE0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
#287
opened Apr 25, 2024 by
anithamudigoudar
How to find the token count of a prompt using meta/llama2-70b model
#274
opened Mar 29, 2024 by
pradeepdev-1995
ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken'
#273
opened Mar 28, 2024 by
sazirod
pinging tiktoken URL always fails, pinging api.openai.com always works
#269
opened Mar 14, 2024 by
distributev
Enhancement: Add convenience token-counting functions to this package
#250
opened Feb 7, 2024 by
pamelafox
Is there a way for tiktoken to interoperate better with offline AI software?
#232
opened Dec 27, 2023 by
ParetoOptimalDev
F-REQ: If the pip installer doesn't find Rust, it should install the pure python version of the tokenizer
#227
opened Dec 10, 2023 by
Emasoft
Previous Next
ProTip!
Mix and match filters to narrow down what you’re looking for.