If your AI works in English, it has not passed.

The handover that changes language

Picture a field colleague receiving an AI-generated case summary in English, then explaining it to somebody in the language they use at home. The facts may travel. The uncertainty, warning and route to correction may not.

This is an illustrative scene, not a client story. It points to a deceptively simple risk: we test the system in the language of head office, then ask another person to carry its meaning across the last mile.

A multilingual interface is not a multilingual service.

I want AI to help mission-led teams reach further, with more consistency and care. That ambition becomes real only when the people furthest from the product decision receive the same quality of reasoning, safeguards and recourse as the people closest to it.

English is not the neutral baseline

Alhanai and colleagues created roughly one million human-translated words across eight low-resource African languages, then evaluated leading models on reasoning, reading and medical-knowledge tasks. For GPT-4o, English accuracy was 83.9% on Winogrande; the average across eleven African languages was 64.8%. On clinical knowledge, the comparison was 89.8% and 70.6%.

Those gaps are percentage points, not a universal failure rate. The study covered selected benchmarks, model versions from 2023–24 and averages that hide substantial variation. Multiple-choice accuracy also says little about tone, dignity or the consequences of a mistake.

It does show why “supports this language” is too weak for governance. Availability tells us that a system produces text. It does not tell us whether the text supports a sound decision in the context where we will use it.

Translation is a route, not a waiver

A tempting workaround is translate into English, run the model, then translate the answer back. Sometimes that helps. IrokoBench evaluated 17 African languages across natural-language inference, mathematics and knowledge questions. Most models performed better on at least some tasks when the prompt was translated into English, yet GPT-4o and GPT-4 Turbo were notable exceptions in the study’s overall comparison. The best closed model still led the best open model by 21.9 points across the reported tasks.

The practical lesson is not “always translate” or “buy the largest model”. It is to test the route you intend to deploy. A translation-first workflow introduces two extra places where meaning can move: before the model and after it. Save all three versions, ask bilingual reviewers where the decision-relevant meaning changed, and include names, dates, local terms and negation in the test.

Test the test set too

Benchmarks can carry their own blind spots. Taguchi and colleagues re-evaluated the first 50 FLORES+ development pairs in four languages, using one native speaker fluent in English for each. Only 27 Asante Twi, 34 Japanese, one Jinghpaw and 12 South Azerbaijani pairs were judged correct. The authors also found critical errors in three of the four samples.

This is a narrow, non-random slice, not proof that every FLORES+ translation is poor. One annotator per language limits reliability. The result still changes the question we should ask of a reassuring benchmark score: what exactly was the model compared against?

Evaluation data needs local review just as model output does. If the gold answer is unnatural, culturally distant or simply wrong, a higher score may reward the wrong behaviour.

Build a language gate before rollout

Start with the decisions people actually need to make. For each priority language, assemble a protected sample of real-shaped but de-identified or synthetic cases. Have qualified bilingual reviewers agree the intended meaning before testing the AI. Score factual preservation, omitted conditions, harmful additions, local clarity and whether a person can challenge the result.

Here is illustrative arithmetic. Four languages, forty cases each, gives 160 reviewed cases. If reviewers find material meaning changes in 12%, that is about 19 cases: 160 × 0.12 = 19.2. Report the count as well as the percentage, because nineteen opportunities to misdirect support are easier to reason about than a polished average.

Do not merge every language into one pass rate. Set a minimum per language and a separate stop rule for severe errors. A system should not borrow confidence from English to cover a language it has not earned.

Design for the person at the edge

Language quality is not a finishing task for the communications team. It is part of service design, data governance and accountability. Put a named language owner beside the product owner. Pay community reviewers for their expertise. Preserve the source and translation trail without exposing sensitive content. Give staff a visible way to flag “technically fluent, operationally wrong”.

The real test of inclusive AI is not how many languages appear in a dropdown. It is whether meaning, responsibility and the right to question a decision survive the journey.

If reliable multilingual records are the first obstacle, explore our Data Validation solution alongside the AI readiness assessment below. Begin with one workflow and one language pair. Learn carefully, publish what the evidence can support, then earn the right to expand.