FIELD NOTES TOPIC
language processing
1 note, newest first.
The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages
The research quantifies the disadvantage faced by many non-English languages in large language models (LLMs) due to subword tokenizers that are primarily trained on English corpora. The study focuses on Indian…
