This is not a fairness problem. This is an engineering problem with hard, measurable numbers. Every paper published this month on low resource NLP arrives at the same quiet conclusion: we did not build universal models. We built English models that can pretend to speak other languages.

The tokenization tax is not a bug

Commercial LLMs bill, scale latency, and budget context per token. Tokenizers assign more subword tokens to the same meaning in some languages than others. Speakers of languages with high token fertility pay a structural penalty before a model is ever invoked.

This penalty was measured across 20 African languages spanning five language families and three scripts, using strictly parallel corpora to isolate language effect from content. Every tested language carried a tokenization premium above English.

Tokenizer	Mean African language premium	Best case	Worst case
GPT-5 o200k_base	3.31x	1.62x	8.92x
GPT-4o cl100k_base	3.47x	1.71x	9.14x
Gemma 4	2.38x	1.37x	6.08x
Llama 3.1	2.92x	1.54x	7.71x
Qwen 2.5	2.76x	1.49x	7.22x

For N'Ko, every prompt, every generation, every API call costs 9 times what it costs for the exact same meaning written in English. Users also only get 11% of the effective context window available to English speakers. If you are building a chatbot for Mali, you cannot even fit a standard 1000 word document before hitting the context limit that an English user would use for a short email.

Nobody designed this. Nobody sat down and decided to charge West African users 9x more. This is what happens when you train a tokenizer on 90% English text, then ship it globally. It is a structural penalty encoded into the very first step of every LLM pipeline, and it applies before any fine tuning, any prompt engineering, any alignment work.

Marathi POS: 83 million speakers, one good dataset

Marathi is the 17th most spoken language on Earth. More people speak Marathi than French. More than German. Until last week there was no standard gold POS dataset for the language.

POS tagging is the foundation for every other NLP task. Without good POS tagging you cannot build reliable translation, information extraction, or parsing.

L3Cube-MahaPOS fixes this gap. 32,354 manually annotated sentences, aligned to Universal Dependencies, annotated by native speakers, with full Unicode normalisation and Devanagari-aware preprocessing.

Model	Token accuracy	Macro F1
HMM	72.11%	60.03%
CRF	79.42%	69.18%
BiLSTM	82.79%	72.41%
MuRIL	86.02%	77.95%
MahaBERT-v2	88.67%	81.67%

Even the best Marathi specific model only hits 88.7% accuracy. For English POS tagging, state of the art crossed 97% in 2018. That is the gap. 9 years of progress, just for a single foundational task.

Marathi has no capitalisation. It has free word order. It has pervasive code mixing. None of these are flaws in the language. They are properties that every general purpose NLP system should handle. We just never bothered to build systems that do.

Parametric knowledge does not travel across languages

We all operate under the unstated assumption that once an LLM knows a fact, it knows that fact in every language it speaks. This is wrong.

Facts stored in model parameters are not language agnostic. If you ask GPT-4o the capital of Nigeria in English you will get the correct answer 99.8% of the time. If you ask exactly the same question in Ewe you will get the correct answer 62% of the time. If you ask it in N'Ko you will get it right 34% of the time.

This is not a translation failure. The model speaks all of these languages. It knows the fact. It just cannot retrieve it when prompted in the wrong language.

All knowledge is stored near English embeddings. To get it out you have to go through English. That is the secret architecture of every multilingual LLM shipped today.

Proverbs and the lie of semantic similarity

When we evaluate multilingual models we almost always measure semantic similarity. Did the model output the same core meaning? That is the only metric we use.

The proverb study shows this is a spectacularly incomplete measurement. Researchers took identical moral proverbs across 15 languages, asked models to expand them into short narratives. At the semantic level, output was almost identical across languages. 94% semantic similarity as measured by standard embedding scores.

But when you looked at the actual narrative, everything that matters was different. Agency was reassigned. Gender roles flipped. Social hierarchy was rewritten. All of the cultural context was stripped out and replaced with default western norms, while the core moral proposition remained intact.

If you only measure semantic similarity you will declare this a perfect success. If you are a speaker of that language you will immediately recognise that the model did not understand your proverb at all. It just extracted the moral lesson, threw away your culture, and gave you back the universal English version.

Turkish scam detection: safety does not translate

Scam detection is not a theoretical problem. 12 million Turkish people received fraudulent phone calls last year. There was no public dataset. There were no deployed detection systems.

Researchers built the first annotated dataset: 100 aligned audio and transcript pairs of real scam and benign calls. They tested every major frontier LLM. The best model achieved 78% accuracy on transcripts. That is 20 points below equivalent performance on English scam detection.

Raw audio input performed 27% worse than transcripts. All of the recent progress on audio LLMs you have seen advertised? Almost all of it was measured on English. It does not work for Turkish. It will not work for any other low resource language.

Safety is not a feature you add after you build the model. Safety is built into the distribution the model was trained on. If your language is not in that distribution, you do not get safety.

Cross lingual prompting beats scaling

There are two ways to get better performance for low resource languages.

1. Spend 10 million dollars training a larger model with more of your language.
2. Spend 10 seconds writing a better prompt.

The parametric knowledge paper found that cross lingual exploration prompting delivers better accuracy gains than doubling model size. It is strictly better on every axis: cost, latency, accuracy, consistency.

This is the most important result published this entire month. All of the arguments about needing more training data, bigger models, larger compute budgets? They are wrong. For most low resource use cases today you will get better results with good prompting than you will get with any amount of scaling.

That is not an argument against building better models. It is an argument that we are currently leaving 70% of the existing model's capability on the table, just because nobody looked for it.

What actually gets lost

None of these papers are making moral arguments. They are all measuring concrete, engineering failures.

• 9x cost penalty for some languages
• 2x lower baseline task accuracy
• 3x lower factual recall
• Complete erasure of cultural context
• No working safety systems

These are not tradeoffs. These are failures. Every one of these problems could be fixed tomorrow. We have just chosen not to fix them.

This is not an accident

Every single one of these gaps follows exactly the same pattern. We build systems for English first. Then we bolt on support for other languages. Then we declare them universal. Then we are surprised when everything works worse for everyone else.

Tokenizers are trained on English. Benchmarks are written in English. Model alignment is done in English. Knowledge is anchored to English.

This is not malice. It is default behaviour. If you do not actively, intentionally build for other languages, you will build an English system. There is no neutral default.

What comes next

All of these papers point in the same direction. We do not need bigger models. We need fairer models. We need models that do not charge people more for speaking the wrong language. We need models that store facts equally well in every language. We need models that understand that meaning is more than just semantic similarity.

The good news is every single one of these problems is solvable. We know exactly what is wrong. We have measurements. We have baseline benchmarks. We have working proof of concept fixes.

The only thing missing is the will to prioritise 3 billion people over incremental improvements to English performance.