A new study from the Coalition for Language Equity in AI Research reveals that large language models require up to 17 times more tokens to process non-English text, forcing users of underrepresented languages to pay significantly higher API costs for the same computational output, as reported by Rest of World.
How Tokenizers Disadvantage Underrepresented Languages
Every large language model relies on a tokenizer to break incoming raw text down into manageable numerical blocks known as tokens. For languages backed by massive online datasets, this segmentation process runs with high efficiency. The research shows that among the top 20 languages by web presence, less common tongues demand 1.5 to 5 times as many tokens as the exact same prompt written in English. When the scope widens to 200 languages, that multiplier climbs as high as 17 times for certain tested models.
Because commercial AI infrastructure bills customers per token processed, this technical bottleneck creates what researchers designate as a hidden tokenizer tax. More tokens mean steeper bills for end users. They also trigger longer generation latencies and restrict the volume of data that can safely reside within a model’s active context window.
Populations Impacted by the Tokenizer Tax
The financial penalty does not target only obscure dialects with minimal speakers. When sorting populations by speaker counts rather than sheer web presence, the tax falls squarely on massive communities. Major languages absorb the economic blow directly.
- Bengali, with approximately 266 million speakers
- Urdu, spoken by roughly 288 million people
- Punjabi, representing about 200 million speakers
- Burmese, encompassing 36 million speakers
- Amharic, covering 36 million speakers
Hundreds of millions of individuals find themselves billed at multiples of the standard English rate while utilizing identical underlying model intelligence. Writing systems rarely encountered in primary training corpora bear the heaviest burden. Burmese, Tibetan, Khmer, and Lao suffer severe token expansion rates. Conversely, languages utilizing the Latin script retain structural advantages even when represented by tiny speaker populations.
Broader Software Ecosystem Consequences
This structural disparity directly complicates global enterprise deployments. Chief Information Officers rolling out generative AI initiatives across international markets must grapple with subpar performance and unpredictable cost scaling outside of Western languages.
