Why does a non-English prompt cost more?
Large language models (LLMs) read and write text as tokens, and tokens are what you pay for. We measured how many tokens the same sentence becomes in English, Swedish, Norwegian, Danish and Finnish.
We sent the same 3,000 sentences, in English, Swedish, Norwegian, Danish and Finnish, through twelve of the newest AI models from nine labs, and counted what each one billed us.
Every single model charged more for every one of the four languages than for English. For Swedish, between 32% and 68% more. For Finnish, up to twice as much.
In this post we explain where the extra tokens come from.
What is a token?
A language model does not read letters or words. It reads tokens: pieces of text from a fixed list, its vocabulary. GPT-6.1's vocabulary has about 200,000 entries. Every text you send is first cut into pieces that are on that list, and you pay per piece.
The list is built once, from a huge pile of training text, before the model itself is trained. The method is simple. Start with single letters. Find the two pieces that most often stand next to each other, and add them, glued together, as a new entry. Then do it again, and again, until the list is full.
Common words get glued all the way into one entry. Rare words stop halfway and have to be spelled out. Take the word meeting. In GPT-6.1's vocabulary it is a single token. The Swedish translation mötet is not on the list, so it becomes two tokens: mö and tet.
The result is that you use more tokens, and consequently are billed more for the same query. Models write one token at a time, so more tokens also means a slower response.
What we measured
To compare languages fairly you need the same content in every language. We used two public research datasets: FLORES-200, about a thousand sentences from Wikinews, Wikivoyage and Wikibooks, and NTREX-128, about two thousand sentences of news.
We sent every text through the newest flagship model of each major lab, as of 1 November 2026: GPT-6.1, Claude Opus 5.5, Gemini 4, Grok 4.7, Mistral Large 4, DeepSeek V4, Kimi K3.1, GLM-5.3 and Qwen 3.8.
Where a model's tokenizer is public, we also counted the texts ourselves.
We report the result as a premium: the number of tokens for a text in one language, divided by the number for the same text in English. A premium of 1.35 means 35% more tokens, and 35% higher cost, for the same content.
Every model charges more
Every model had a premium of 30% or more compared to English, for all four languages. Across the frontier models, the same content cost:
| Language | Encyclopedic text (FLORES) | News (NTREX) |
|---|---|---|
| Swedish | +32% to +68% | +29% to +64% |
| Norwegian | +35% to +66% | +32% to +62% |
| Danish | +38% to +73% | +34% to +64% |
| Finnish | +56% to +100% | +54% to +94% |
News text gives slightly lower numbers, but the same picture.
Mistral, Europe's own frontier lab, is no cheaper for Scandinavian languages. Mistral Large 4 uses its own tokenizer, Tekken, and charges 41% extra for Swedish, slightly more than the American models. A look inside its vocabulary suggests why. It has about 50% more entries with French letters like é and ç than OpenAI's, but about as many with å, ä or ö. Viking, built for Nordic text, has almost ten times as many.
It is not the language
The easy explanation would be that Swedish is simply a wordier language. It is not. The Swedish and Norwegian translations are exactly as long as the English original in characters, to within 0.2%. Danish is 3% longer, Finnish 6%. For the Scandinavian languages the whole premium comes from the tokenizer.
We tested this by counting the same texts with a few public tokenizers that were built for specific languages. Viking, an open Nordic model from 2024, needs only about 5% more tokens for Swedish than for English.
Bigger vocabularies have helped. GPT-3's tokenizer from 2020 charged 91% extra for Swedish, GPT-4's from 2023 58%, and GPT-4o's from 2024 35%. But OpenAI's newest models still use the tokenizer from 2024.
This is not unique to the Nordic languages. Any language that was not a priority when the vocabulary was created tends to pay a premium. We also measured Japanese, where the premium ranges from 13% to 66%, and earlier research has found languages that need up to 15 times as many tokens as English.
A case for local AI labs
At Klang we experience this first-hand, since a large part of our users are in the Nordics.
The fix is not a secret. Give a model a vocabulary built with Nordic text, and most of the premium disappears. And given how strong open-weight models have become in the last year, there is real potential in local models that catch up with the frontier, both in quality and in cost. We think this highlights the case for local AI labs, and is a potentially important opportunity for the Nordics.
References
- Petrov et al. (2023): Language Model Tokenizers Introduce Unfairness Between Languages
- Ahia et al. (2023): Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
- Viking model card (LumiOpen)
- NLLB Team (2022): No Language Left Behind (FLORES-200)
- Federmann et al. (2022): NTREX-128