Speakers of less common languages may face higher costs when using artificial intelligence compared to those who use dominant languages like English. A recent study by the Coalition for Language Equity in AI Research, released last month, revealed that AI models demand significantly more tokens for certain languages. Tokens are the basic units of text or data processed by these models.
Researchers describe this disparity as a hidden “tokenizer tax,” which disproportionately affects languages with a smaller online footprint. A tokenizer is a tool that breaks down text, images, or audio into these tokens. Even among the top 20 most widely used internet languages, the study found variations in token usage.
The language requiring the most tokens used 1.5 to 5 times more than the least token-intensive language. This disparity grew more pronounced when the study expanded to 200 languages, with one AI model showing a 17-fold increase in the token cost gap.
Language Inequity in AI
The study highlights that a company in Bengaluru might incur extra charges for each AI prompt and response, unlike a competitor in Austin. This tokenizer tax has consequences beyond financial costs.
Higher token usage leads to increased latency and reduced context capacity, impacting performance, especially in regions with limited computing power and connectivity. This inequity limits participation in the AI economy, particularly for non-English languages, which leave less room for documents or conversation history and require more effort to generate responses.
Read Also: Chinese AI Models Gain Global Traction Amid Security Concerns
Pratyush Choudhury, co-founder of Activate AI, India’s first AI-focused venture capital fund, explained, “Think about a tutor: Retaining the student’s earlier mistakes and explanations is part of delivering a good lesson.
If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service,” Choudhury added.
Improving Language Processing
Enhancing language processing requires better underlying data. Initiatives like RESPIN, a Gates Foundation-funded collaboration with the Indian Institute of Science, are key. RESPIN gathered text and spoken data from native speakers of 38 dialects.
Raoul Nanavati, CEO of Navana.ai, a Mumbai-based AI voice startup, emphasized the importance of such efforts in local reports.
The Coalition’s study found that a language’s rarity isn’t the sole factor in token usage; the script also plays a significant role. Asturian, despite having few speakers and no Common Crawl data, benefits from its Latin script and ties to languages like Italian.
Read Also: Vivo X500 Ultra camera details leaked
Conversely, languages with less common scripts—such as Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki—faced consistent disadvantages across most models analyzed.
Dheemanth Reddy, CEO of Maya Research, noted that many tokenizers are still heavily optimized for English and other well-resourced languages. Maya Research is developing conversational AI models for the next 5 billion internet users, aiming for native-like speech and responses. Reddy pointed out that the core issue is the lack of high-quality digital data.
He cautioned that better training data alone won’t solve the tokenizer problem.
Addressing the Tokenization Gap
Reddy noted that falling token prices should reduce direct costs over time, and improved multilingual and synthetic data should enhance model quality. However, closing the tokenization gap requires labs to design future tokenizers that better handle diverse languages and scripts.