agustinbrusco / tokens-thorugh-lang Public

Notifications You must be signed in to change notification settings
Fork 0
Star 2

Analysis of LLM token representation of texts in different languages

2 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
flores200		flores200
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
flores_gpt-4_tokens.tsv		flores_gpt-4_tokens.tsv
flores_llama_tokens.tsv		flores_llama_tokens.tsv
token_by_lang.ipynb		token_by_lang.ipynb

Repository files navigation

tokens-thorugh-lang

Analysis of the token representation of texts in different languages

About

Analysis of LLM token representation of texts in different languages

multilingual machine-translation research-project tokenization inequality low-resource-languages large-language-models

Report repository

Releases

No releases published

Packages

No packages published

Languages

Jupyter Notebook 100.0%