AI Model Costs Vary by Language: Tokenisation Gap Study

What happened

A new study from the Coalition for Language Equity in AI Research has found that large language models (LLMs) need considerably more tokens to process some languages than others, which can make the same request more expensive for users of less widely digitised languages. Computerworld reported the findings, drawing on coverage by Rest of World.

An LLM's tokeniser splits text into units called tokens. According to the researchers, this process is far more efficient for languages with abundant digitised data. Among the 20 languages with the largest online presence, less common languages used between 1.5 and 5 times as many tokens as an equivalent English prompt. When the researchers widened the analysis to 200 languages, the gap reached as much as 17 times in one of the models they examined.

Because most LLM providers bill according to tokens consumed, the extra tokens translate directly into higher charges. The study also points to longer response times and less room within a model's context window, which limits how much information a single interaction can carry.

Why it matters

For organisations deploying AI assistants across multilingual markets, the findings suggest that a single pricing model can produce very different unit costs depending on the language a customer speaks. A service that appears uniform on a dashboard may be delivering a materially different experience, in both cost and speed, to users in different linguistic communities.

The study also shows that the limits of today's models are not evenly distributed. Users of underrepresented languages may receive slower answers and less contextual depth from the same tool, which bears directly on the fairness of AI-enabled services. Leaders planning AI roll-outs in the region and beyond will need to account for language as a variable in both cost modelling and service quality.

By the numbers

• 1.5 to 5 times the tokens of an English prompt for less common languages among the 20 with the largest web presence, according to the study.
• 20 languages with the largest web presence were included in the initial comparison.
• 200 languages were covered when the analysis was expanded.
• Up to 17 times the English token count in one of the models examined in the expanded analysis.

The Renascence take

Most coverage will frame this as a pricing story for AI providers. We see it as a service-design problem that surfaces as a cost line: when the interface looks identical across languages but the underlying effort and price are not, the customer experience is unequal before a single word is read.

The headline is about cost, but the real issue is that fairness is being decided by tokenisers that most users never see. Behavioural economists would call this a hidden default, and hidden defaults rarely stay hidden once customers compare bills. Operators should audit cost, latency and context limits by language segment now, rather than waiting for a provider to adjust its pricing. Where the gap cannot be closed, be transparent about it, because an honest service boundary builds more trust than a uniform promise that quietly disadvantages some customers.