Which LLM actually speaks Hungarian? A measured comparison

What did we measure, and what came out of it?

We ran the same business text through twelve current models in Hungarian and in English, counting input tokens with each model’s native tokenizer. The Hungarian version consumes 1.50 to 2.23 times more input tokens for identical content, median 1.63. Model choice alone moves the Hungarian token count by 49 percent. We measured cost, not language quality. Measurement date: 14 August 2026.

Hungarian is the example here, but the finding is not about Hungarian. Every language that is thin in tokenizer training data splits into more pieces than English does, and every one of them pays for that on the invoice, in latency and in usable context. The mechanism generalises. The numbers do not: Finnish, Turkish, Greek or Thai would each need their own measurement, and the ratios would land somewhere else.

We picked Hungarian for two reasons. It is agglutinative and heavily accented, so it sits near the difficult end of the spectrum, which makes the effect easy to see. And the local AI vendors keep promising native Hungarian processing while publishing no measurement of what that costs. If you build in a small language, that second part will sound familiar.

Why tokens and not something richer? Because the token is the only thing the invoice is about, and because the measurement needs neither an annotator team nor a corpus. Anyone can repeat it in an afternoon. We print the method and the raw numbers in full for exactly that reason.

This article rests on a second measurement round. The first one, run the same day, covered the previous model generation: Llama 3.3, Llama 4 Maverick, Gemma 4, Mistral Small 3.2, Gemini 2.5 Flash-Lite, Claude Haiku 4.5 and DeepSeek V3.2. We re-ran it against the current generation with an unchanged text pair and an unchanged harness. Comparing the two rounds produced a finding of its own, which has its own section below.

Why does tokenization make a non-English chatbot more expensive?

Language models process tokens, not characters or words, and the vocabulary that defines those tokens is trained on a corpus dominated by English. Billing follows the same units. Across eleven of the twelve models, one token carried 5.30 to 5.49 characters in English and 2.36 to 3.52 in Hungarian. The twelfth, Claude Opus 5, sits outside that range in both languages at 3.63 and 2.18. More pieces means higher cost, slower prefill, and less content per context window.

On the measured pair the Hungarian text is 1,183 Unicode characters and the English 1,213, so they are effectively the same length. The Hungarian version uses 151 words against 194, because agglutination packs into suffixes what English spells out with prepositions and auxiliaries. In UTF-8 the Hungarian is 11 percent more bytes, since accented characters take two. For the tokenizer, that second property is the one that hurts.

One concrete chatbot, costed out

Take a customer service chatbot with a RAG knowledge base and 500 conversations a month. Six turns per conversation, accumulating history, with the system prompt and the retrieved document snippets resent every turn. In English that is roughly 8,000 input and 900 output tokens per conversation, so 4,000,000 input and 450,000 output tokens a month.

Gemini 3.7 Flash lists at $0.75 per million input tokens and $3.75 per million output tokens. The English version therefore costs $3.00 plus $1.69, or $4.69 a month. The Hungarian version, scaled by the measured 1.64 ratio, comes to 6,542,000 input and 736,000 output tokens, or $7.67. The difference is $2.98 a month, which is €2.43.

The same scenario on Claude Opus 5, listed at $5 input and $25 output per million tokens (platform.claude.com): $31.25 in English, $50.80 in Hungarian, a difference of $19.55, or €15.96 a month.

Four measured models are missing from that table because we found no official public per-token price list for them: Qwen3.8-Max, Muse Spark 1.2, Nemotron 3.5 Lightning and Seed 2.1 Turbo. Their tokenizer numbers stand; only the currency conversion would be unsourced. Reseller prices exist for all four, which is not the same thing.

Honestly, at this size the language surcharge is not interesting. A line item of €0.98 to €20.06 a month is less than an hour or two of developer time. Any vendor justifying a monthly fee with the cost of processing a small language is arguing badly.

The number turns serious where volume grows, or where an expensive model runs underneath.

The real effect is not on the invoice

The token surcharge shows up in three places, and the bill is the least painful of them.

• A 32k context window holds 1.5 to 2.2 times less content in Hungarian. A longer contract or a forty-turn conversation still fits in English and no longer fits in Hungarian. The system starts forgetting, or you move to a larger, pricier model. That is a functional ceiling rather than a cost line.
• Input processing is roughly linear, so 1.5 to 2.2 times more input means about that much longer time to first token. In a chat interface the user feels it.
• Token-based per-minute quotas drain faster, so the same rate limit serves fewer users in the non-English language.

For a product built primarily for one small-language market, those three items make tokenization an architecture decision, not an accounting footnote. It shapes which model you can afford to run, how much history you can keep, and how aggressively you have to compress retrieved context.

How did we measure it? The full method

The OpenRouter API returns the value computed by the model’s native tokenizer in the usage.prompt_tokens field, and billing runs off the same number. The documentation states this explicitly (openrouter.ai/docs). So what we measure is exactly what gets paid. We called twelve models with identical parameters and the same two texts, then subtracted the fixed chat-template overhead per model. The sections below describe those three steps in enough detail to repeat them.

Call parameters

Two models rejected max_tokens: 1. Gemini 3.7 Flash returned an empty response and a generation record with no usage data, and Muse Spark 1.2 answered HTTP 400 with a minimum of 16. Both ran all three measurements (baseline, Hungarian, English) at max_tokens: 16, so each stays internally consistent. max_tokens caps the output and by definition does not touch the input count.

Subtracting the prompt overhead

This is the correction the whole measurement stands on. The calls do not send raw text: every model wraps it in a chat template, and the harness adds its own system preamble. That wrapper differs by model and it is not small. In this round we found a fixed overhead of 222 to 647 tokens, which at the top end is half again the size of the English text itself.

Every model got a baseline run with identical parameters and a payload of a single x character. Clean text length follows from:

net_tokens = total_tokens − baseline_tokens + 1

The plus one is there because the baseline itself contains one token of payload. We checked the correction three independent ways, because with a 647-token Grok baseline plain subtraction would not have been credible on its own.

• Linearity. With a payload of ten x characters, Gemini went from 222 to 231, Claude Opus 5 from 319 to 328, Grok from 647 to 656 and Kimi K3 from 311 to 320. All four landed on exactly the expected plus nine, large overhead included.
• Marginal cost. Claude Opus 5 returned a conspicuously high English figure, so we ran a check that ignores the baseline entirely: the English text sent twice in the same message. Once gives 652 tokens, twice gives 987, so one extra copy costs 335. Baseline subtraction had produced 334. The one-token gap is the paragraph separator.
• Control from the earlier round. We re-measured Claude Haiku 4.5 the same day on the same harness: baseline 233, English 465. Bit-identical to the earlier round, so the two rounds are directly comparable.

The two measured texts, verbatim

Four paragraphs, same content, business and technical register. The Hungarian is the original and the English is a faithful translation of it. The same two blocks ran in the earlier round, which is what makes the rounds comparable. Copy these if you want to repeat the run.

Az ügyfélszolgálati folyamatok automatizálása a legtöbb középvállalatnál a beérkező megkeresések osztályozásával kezdődik. A rendszer beolvassa az e-mailt, azonosítja a küldő ügyfelet a vállalatirányítási rendszerben, majd eldönti, hogy árajánlatkérésről, reklamációról vagy technikai hibabejelentésről van-e szó. Az osztályozás eredményét a munkatársak felülvizsgálhatják, és a javításokból a modell folyamatosan tanul.

A bevezetés első szakaszában érdemes szűk területen indulni. Válasszunk egyetlen ügytípust, amelyből havonta legalább néhány száz eset érkezik, és mérjük meg a jelenlegi átfutási időt, valamint az egy megkeresésre jutó munkaráfordítást. Ezek nélkül a későbbi megtérülés nem igazolható.

A második szakaszban következik az integráció: a vállalatirányítási rendszer, a számlázó program és a levelezés összekötése. Itt derül ki, hogy a meglévő adatok mennyire tiszták. Tapasztalatunk szerint a projektek csúszásának háromnegyede nem a modell pontosságán, hanem a hiányos törzsadatokon múlik.

Végül az üzemeltetés kérdése marad. Meg kell határozni, ki felel a hibás döntésekért, milyen naplózás készül, és hogyan lehet a rendszert visszavonni, ha a minőség romlik.

For most mid-sized companies, automating customer service processes begins with classifying incoming requests. The system reads the email, identifies the sending customer in the enterprise resource planning system, and then decides whether it is a quote request, a complaint, or a technical fault report. Staff can review the result of the classification, and the model learns continuously from the corrections.

In the first phase of the rollout it is worth starting on a narrow scope. Pick a single case type that generates at least a few hundred cases per month, and measure the current turnaround time as well as the work effort per request. Without these, the later return on investment cannot be proven.

The second phase brings integration: connecting the enterprise resource planning system, the invoicing software, and the mail server. This is where it becomes clear how clean the existing data is. In our experience, three quarters of project delays are not caused by model accuracy but by incomplete master data.

Finally, the question of operations remains. It must be defined who is accountable for wrong decisions, what logging is produced, and how the system can be rolled back if quality degrades.

The models under test

Twelve models from twelve tokenizer implementations. We checked every slug against the model listing endpoint first; none of them is a guessed name.

One model dropped out. openai/gpt-5.6-luna-pro returned 2,396 input tokens for the single-character baseline, an order of magnitude above the 222 to 647 range of the field, which suggests the pro reasoning mode folds internal passes into the prompt token count. That invalidates linear overhead subtraction. With a long payload it also returned no usage data on four consecutive calls. We substituted the flagship Sol from the same family and tokenizer, whose baseline is bit-identical to the Luna figure from the earlier round.

What are the raw results per model?

Every figure below is the input token value returned by OpenRouter, with the net columns carrying the overhead correction. Net Hungarian counts scatter between 336 and 543, English between 221 and 229 with one exception.

Two ratios that get confused with each other

The table holds two numbers that are easy to mix up. The HU/EN token ratio says how many times more tokens the same content costs in Hungarian; that is the cost effect, and it sits between 1.50 and 2.23. The fertility ratio says how many tokens a Hungarian word needs relative to an English one; that is the metric linguistics papers normally report, and it sits between 1.93 and 2.86.

They diverge because the same content needs 22 percent fewer words in Hungarian. Agglutination punishes and rewards at the same time: more tokens per word, fewer words per sentence. The net result is that the real cost surcharge is roughly 25 percent smaller than a bare fertility number would suggest. Quoting a 1.9 to 2.9 fertility figure as a cost argument overstates the disadvantage. That correction applies to any language where morphology carries information English puts in separate words, which is most of the agglutinative and richly inflected ones.

Model choice moves 49 percent

The two extremes are Muse Spark 1.2 at 336 tokens and Claude Opus 5 at 543, on the same Hungarian text, which is 62 percent apart. Opus 5 is a special case though: its English count degraded too, so the penalty there is not specific to Hungarian. Drop it and the remaining eleven models still span 336 to 501, a 49 percent difference in Hungarian input cost without changing anything in the application. In the earlier round the same figure was 46 percent.

Which gives the most concrete takeaway in this article: for an application running in an under-represented language, tokenizer efficiency is a model selection criterion in its own right, alongside capability and list price. In practice not a single model card carries this number for any language.

One small observation from the table: three models from entirely unrelated vendors, Qwen3.8-Max, Nemotron 3.5 Lightning and Mistral Medium 3.5, returned exactly the same Hungarian count of 366. That either points to shared vocabulary ancestry, or to large BPE vocabularies converging on a similar optimum for Hungarian. The measurement cannot settle it.

Across generations: are tokenizers getting better at Hungarian?

This section compares the two measurement rounds. Same text pair, same harness, same day, only the model generation differs. The control runs (Qwen3.8-Max in both rounds, Claude Haiku 4.5 re-measured) returned bit-identical values, so the numbers line up.

Tokenizers are not improving on Hungarian from one generation to the next. Five vendors out of six return an identical Hungarian token count down to the last token, despite full architecture changes, context window expansions and capability jumps in between. Vendors ship a new model and carry the old vocabulary forward unchanged. If your plan assumes the next generation will fix the token penalty in your language, that plan needs a different assumption.

The field as a whole did not tighten either. The spread grew from 46 to 62 percent, the median barely moved (1.64 to 1.63), and one of the five new tokenizer families, Kimi K3, became the weakest entry at 501 tokens. Kimi is unremarkable in English at 5.39 characters per token and drops to 2.36 in Hungarian, the most language-specific penalty in the set.

Anthropic changed its tokenizer, and Hungarian got worse

The only vendor that touched the vocabulary is Anthropic. Their own pricing documentation states it plainly: Claude 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same text. Our measurement puts numbers on that and adds a detail the vendor does not break out, namely that the increase is not spread evenly across languages.

The new Anthropic tokenizer penalises English harder. That sets a trap worth naming. Claude Opus 5 shows a HU/EN ratio of 1.63 against Claude Haiku 4.5’s 1.86, which reads as an improvement for Hungarian. It is not: the Hungarian text costs 25 percent more tokens on Opus. The ratio only got better because the English baseline got worse faster.

We verified the Opus 5 figure three independent ways, because a number that far out looks like measurement error first. The marginal test returned 335 tokens against the 334 from subtraction. Pinning the provider changed nothing: Anthropic first-party and Amazon Bedrock both returned the same 861 for the Hungarian text. And the Claude Haiku 4.5 control reproduced the earlier round exactly. The tokenizer change is real.

What does this number not tell you?

The measurement is a point estimate rather than a distribution, and it is fragile in several places: one text pair, one register, one day, input only. Those limits have to be stated or the article misleads. If you take one thing away, take this: tokenizer efficiency is not the same thing as language quality.

One text pair, one register

The measurement rests on a 1,183 character Hungarian text and a 1,213 character English one, in a business and technical register. We do not know how it would behave in legal or medical language, where rare long compounds are more frequent and the ratio probably gets worse. Short conversational chat messages would likely improve it. Prompts carrying code or structured data dilute the ratio, because those parts are language-neutral. A more serious study would use 10 to 20 text pairs across registers and report variance.

Provider variance, one bug we found

On Kimi K3 two identical calls returned different token counts: 349 then 311 for the single-character baseline. The generation records show OpenRouter routing the calls to different providers, Chutes returning 349 and Modal 311. With open-weight models this usually comes from a different chat template or a different Unicode normalisation, and accented characters are particularly sensitive to it. We pinned the model to Modal and ran every Kimi measurement that way. The other eleven models returned bit-identical results on repeat, Claude Opus 5 across two different providers.

Translation is not neutral

The English text is a translation. Another equally faithful translation could be 5 to 8 percent shorter or longer, which shifts the ratio directly. Objective translation does not exist, so the measurement carries roughly five percent of uncertainty before the tokenizer even sees it. Anyone repeating this for a different language inherits the same problem.

What we did not measure at all

• Output was not measured. The cost example assumes Hungarian output is more expensive by the same ratio as input. That is plausible, since the same tokenizer is doing the work, but it is not measured data.
• We say nothing about language quality. A model may tokenize Hungarian cheaply and speak it poorly.
• Prompt caching was not measured either. With a cached system prompt the real cost gap is smaller than the table shows.
• The overhead subtraction is an approximation resting on two assumptions: that the x payload is one token, and that the fixed overhead does not depend on payload size. We validated the first on four models and the second directly on Claude Opus 5 through the marginal test, but only indirectly elsewhere.
• Four models have no official price, so they sit outside the cost table. Alibaba Cloud lists Qwen without a public token price, Meta publishes no per-token API rate card, the NVIDIA model page timed out, and the Volcengine price list renders only behind a login.

What Hungarian LLM benchmarks exist today?

Three serious efforts: the HuLU language understanding benchmark, OpenHuEval for Hungarian-specific knowledge, and Racka, which describes a model adapted to Hungarian. All three measure accuracy. None of them addresses what the language costs on commercial models, and none touches the context window effect.

The same pattern holds for most small languages. There is usually one academic understanding benchmark, sometimes a cultural knowledge set, occasionally an adapted model, and nothing at all on the economics. Worth checking before you assume your language is better served.

HuLU

The Hungarian language understanding benchmark from the HUN-REN Research Centre for Linguistics, built on the GLUE and SuperGLUE pattern with six annotated corpora (hulu.nytud.hu).

This is the most thoroughly documented Hungarian understanding suite, with its own web-based evaluation service. Its weakness is visible in the table: five of six tasks are translations, and only HuCoLA is originally Hungarian. Translated benchmarks tend to measure English-shaped reasoning wearing a Hungarian surface, rather than phenomena specific to the language. If you evaluate a small-language model, check the provenance of every task before trusting the leaderboard.

We could not extract the public results table: the page renders client-side and the fetch returned only navigation. The one concrete HuLU score quoted below therefore comes indirectly, from the Racka paper.

OpenHuEval

A benchmark for Hungarian-specific and cultural knowledge, with five tasks and 3,953 questions across eight dimensions (arXiv:2503.21500, ACL 2025 Findings).

The most interesting result for me is not the ranking but a side finding: for 70 percent of the evaluated models the ordering changed relative to English benchmarks. The English leaderboard does not predict Hungarian performance. That is a strong argument for testing in the target language, and it is the single most transferable finding in this section, whichever language you work in. The authors call their own work an initial step.

Racka

Work from the digital humanities and artificial intelligence departments at ELTE, MSZNY 2026 Best Paper (arXiv:2601.01244, the model: huggingface.co/elte-nlp/Racka-4B). Qwen3-4B backbone, LoRA-based continued pre-training on 160 billion subword tokens, on the Komondor HPC in Debrecen, 64 A100 GPUs for 287 hours. They replaced the tokenizer: a new BPE tokenizer trained on a Hungarian corpus, then 32,000 Hungarian-optimised tokens merged into the original Qwen3 vocabulary.

This is the only one of the three that measures tokenizer efficiency directly and publishes a number for it.

With the original Qwen3-4B tokenizer, Hungarian fertility is 1.99 times the English figure. With the Hungarian-optimised vocabulary Hungarian drops below English, 1.66 against 1.94, while English degrades by 23 percent. The model scored 0.751 overall on HuLU against Qwen3-4B’s 0.711, and 33.93 on OpenHuEval against 33.44. Notice the German row as well: adding Hungarian tokens cost German 12.62 percent. Vocabulary space is zero-sum, which is the whole reason under-represented languages are expensive in the first place.

What the paper does not give you: the fertility measurement covers one model family, and the goal is to justify the authors’ own adaptation rather than compare models available on the market. There is no GPT, Claude, Gemini or Grok tokenizer comparison in it, and no cost. Fertility there is a runtime argument, not an invoice.

Where the gap is

The gap is easy to describe: nobody measures what a small language costs on commercial models, across tokenizer families, in a comparable way. This article fills that gap narrowly, with a single text pair, but with a real measurement.

What should you do if you build in an under-represented language?

Four decisions are directly affected: model choice, prompt size, caching, and the rarer question of whether to run a language-adapted open-source model. The order is deliberate. The first two are free, the third is cheap, and the fourth is expensive.

Put tokenizer efficiency on the model selection list

Do not read the list price on its own. Read it multiplied by the absolute token count for your language. Our measurement shows a 49 percent gap on Hungarian input across models that look comparably priced on paper, and 62 percent once Claude Opus 5 is in the picture. If your system mostly processes text in one non-English language, that multiplier is the real price. The comparison is simple: send your own typical prompt through two or three models and read the usage.prompt_tokens value.

One caveat matters here. This criterion is the third one, not the first. A model that tokenizes cheaply but performs worse on your task leaves the saving unfunded. Measure capability separately, on your own dataset. We wrote about the tooling for that in the article on monitoring and debugging.

Compress the prompt, because every word costs more

A Hungarian system prompt costs one and a half to two times what its English equivalent does in tokens, and you resend it every turn. Two practical moves help. Write the system prompt in English and request the response language by instruction; models follow that reliably, and the fixed cost drops. Cut the retrieved RAG snippets harder: four chunks instead of eight, with a narrower window.

The second move is not free. Reducing retrieved context hurts accuracy if the chunking is poor. There is a detailed treatment in the article on RAG systems.

Cache the fixed part

The system prompt and the standing examples are the same on every turn. Prompt caching at the major providers bills that portion at a fraction of the price, and with non-English text the absolute saving is larger precisely because the repeated part is the expensive one. We did not measure the cache effect, so no number goes here, but the mechanism is clear.

When is a language-adapted open-source model worth it?

Based on the Racka data, a language-optimised vocabulary can roughly halve the token count, which buys a proportional prefill speedup and more usable context. You also lose frontier model capability, English gets worse, and you take on GPU operations.

In our experience this pays off when the task is narrow (one classification, one extraction, not a general assistant), traffic is high and dominated by the one language, and there is a data residency constraint that argues for own hardware anyway. If only one of the three holds, stay on an API. The practical side of local inference is covered in the article on running AI locally, and the cost comparison of the three implementation layers sits in the AI chatbot, n8n and custom agent comparison.

Summary and frequently asked questions

Sources

• Own measurement, AppForge, 2026-08-14, through the OpenRouter API. On token values computed with the native tokenizer: openrouter.ai/docs/api_reference/overview.
• Anthropic pricing, which also confirms the tokenizer change: platform.claude.com/docs/en/about-claude/pricing. Other official rate cards: OpenAI, Google, xAI, DeepSeek, Moonshot, Mistral, Upstage.
• Exchange rates, 2026-08-14: 314.36 HUF/USD mid-rate from the Hungarian National Bank, mnb.hu/en/arfolyamok, converted to euro at 385 HUF/EUR, giving 0.8165 EUR/USD.
• HuLU benchmark and task descriptions: hulu.nytud.hu and hulu.nytud.hu/tasks.
• OpenHuEval, ACL 2025 Findings: arxiv.org/html/2503.21500v1.
• Racka, MSZNY 2026 Best Paper: arxiv.org/abs/2601.01244; the model: huggingface.co/elte-nlp/Racka-4B.

If you are planning an AI system in a language that is not English and cost is part of the model decision, the guide to AI development costs and our process automation service are the next things to read. We kept the raw log and the generation IDs; if you reproduce this and get a different number, we would be glad to compare notes.