Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

Source

arXiv

arxiv.org/abs/2607.24276v1

Read the source

Source

Plain-Language Brief

The research quantifies the disadvantage faced by many non-English languages in large language models (LLMs) due to subword tokenizers that are primarily trained on English corpora. The study focuses on Indian languages and uses the FLORES-200 parallel corpus to measure the tokenization fertility across six popular tokenizers and fourteen languages. It identifies the main cause of this disparity: failed byte-pair merges leading to text fragmentation into single-byte tokens. Multilingual tokenizers such as XLM-R and OpenAI's o200k_base reduce the average Indic tokenizer tax by 73%. The research also shows a practical consequence, that under fixed context budgets, Indian-language documents preserve less original content than equivalent English documents.

What the Document Says

  • The study focuses on the disadvantage faced by non-English languages in large language models due to subword tokenizers trained predominantly on English corpora.
  • It uses the FLORES-200 parallel corpus and measures tokenization fertility across six widely used tokenizers and fourteen Indian languages.
  • The main cause of the disparity is identified as failed byte-pair merges leading to text fragmentation into single-byte tokens.
  • Multilingual tokenizers such as XLM-R and OpenAI's o200k_base reduce the average Indic tokenizer tax by 73%.
  • It shows that under fixed context budgets, Indian-language documents preserve less original content than equivalent English documents.

Key Concepts Mentioned

  • large language models (LLMs)
  • subword tokenizers
  • Indian languages
  • tokenization fertility
  • byte-pair merges
  • multilingual tokenizers

Claims Made by the Document

  • Subword tokenizers introduce a systematic disadvantage for many non-English languages.
  • Indian languages experience an average 8.0x tokenization tax relative to English under cl100k_base.
  • Multilingual tokenizers can reduce the average Indic tokenizer tax by 73%.

Evidence Used

  • The FLORES-200 parallel corpus is used to measure tokenization fertility across various languages and tokenizers.
  • The study shows that failed byte-pair merges lead to text fragmentation into single-byte tokens.

Methods / Approach

  • Using the FLORES-200 parallel corpus to measure tokenization fertility
  • Analyzing the relationship between tokenizer fertility and reading comprehension performance

Limitations Stated by the Authors

The study does not provide specific details on how the fixed context budgets are defined.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study only considers Indian languages and six popular tokenizers. It does not explore other non-English languages or tokenizers.

Open Questions

  • What is the tokenization tax for other non-English languages?
  • Are there other causes of the disadvantage faced by non-English languages in LLMs?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The research quantifies and explains the cross-lingual cost of subword tokenization for Indian languages in the context of large language models (LLMs). It focuses on the FLORES-200 parallel corpus, identifies failed byte-pair merges as a primary cause of the disparity, and demonstrates that multilingual tokenizers can reduce the average Indic tokenizer tax by 73%. The research also shows a practical consequence of this disparity under fixed context budgets.

Raw Abstract / Excerpt

Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages. In this work, we quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus, measuring tokenization fertility across six widely used tokenizers and fourteen languages. Under cl100k_base (used by GPT-3.5 and GPT-4), Indian languages experience an average 8.0x tokenization tax relative to English, reaching 13.0x for Malayalam, reducing the effective context window to as little as 12% of that available to English users for equivalent semantic content. We identify the primary mechanism behind this disparity: failed byte-pair merges that leave text fragmented into single-byte tokens, with merge failure strongly correlating with tokenizer tax (Pearson r = 0.89). We further show that this phenomenon is not an inherent property of Indic scripts but a consequence of tokenizer design. Multilingual tokenizers such as XLM-R and OpenAI's o200k_base reduce the average Indic tokenizer tax by 73%, demonstrating that the disparity is largely remediable. Beyond token statistics, we quantify a practical consequence by showing that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents. Finally, we examine the relationship between tokenizer fertility and reading comprehension performance on the Belebele benchmark, finding that the apparent correlation is largely explained by language resource availability rather than tokenizer behavior alone.