A model can be accurate and still miss the people who need you.

The successful pilot

Picture a team reviewing an AI pilot. Accuracy is strong. The workflow is faster. The dashboard is green. The natural next sentence is: “Let’s roll it out.”

This is an illustrative scene, not a client story. The difficult question comes one sentence later: roll it out to whom, where and under what conditions?

A model can be accurate and still miss the people who need you.

I am ambitious about what AI can do for mission-led organisations. But a successful pilot is evidence about one setting, not a passport to every setting. If the next country, language, office, season or beneficiary group looks different, the evidence must travel again.

An average is not coverage

Mena and colleagues studied satellite-based poverty maps using survey data from ten countries. They simulated programmes selecting the poorest 20% of regions. Nationwide, recall and precision ranged from 54% to 71%. When selection was restricted within rural areas, the range widened to 38%–73%; within urban areas it was 46%–65%.

These are simulated regional allocations, not individual eligibility decisions. The ranges combine countries and two metrics. Yet the operational lesson is sharp: a national score can look respectable while performance inside the group you most need to distinguish is weaker.

The same study found one of the mechanisms. Satellite features often separated urban from rural wealth better than they separated poverty within either setting. In eight countries where urbanisation and wealth were strongly related, simulated aid to rural areas was under-allocated by one to five percentage points compared with survey-based allocations. A model can learn a visible proxy and still blur the people behind it.

Context leaks into the prediction

Zech and colleagues trained pneumonia-screening models on 158,323 chest radiographs from three hospital systems. Their strongest pooled model reached AUC 0.931 on internal test data and 0.815 at a fully external hospital. The model could also identify which hospital produced an image with more than 99.9% accuracy for two sites.

This is medicine, not humanitarian delivery, and AUC is not impact. The external sample was smaller, creating a wide confidence interval. What transfers is the failure mode: the system found institutional clues that were highly predictive in development but less useful elsewhere.

Our equivalent clues may be form layout, referral source, office coding, device type, a dominant language or the way one programme records missing data. If a pilot only tests the environment that created those clues, we may be measuring familiarity rather than readiness.

Treat change as normal

The WILDS research team assembled ten datasets spanning hospitals, camera traps, poverty mapping, land use, molecular experiments, online comments and code. Across all ten, standard training produced substantially lower performance outside the training distribution. Existing robustness methods did not reliably close the gap.

WILDS is a benchmark suite, not proof that every deployed model will fail. Its value is conceptual: distribution shift is not an edge case. Real systems encounter new locations, populations, equipment, time periods and workflows.

That changes our governance language. Do not ask only, “Did the model pass UAT?” Ask, “Which contexts did it pass, which contexts remain untested, and what change would trigger a fresh check?” A deployment boundary should be written as clearly as a data-retention rule.

Count who is missed

Start with a deployment matrix: country or office, language, route into the service, beneficiary group, time period and device or channel. For every high-consequence decision, keep a protected holdout sample from each relevant cell. Report false negatives and false positives as counts as well as rates.

Here is illustrative arithmetic. Suppose a programme reviews 200 cases and 40 truly meet the priority rule. At 70% recall, the model identifies 28 and misses 12: 40 × (1 − 0.70) = 12. A 70% headline feels abstract; twelve missed cases creates a decision.

Then inspect the misses with local staff. Are they clustered around one language, data source or referral route? Do not “fix” the metric by changing the denominator after seeing the result. Record the issue, decide whether to narrow use, add review or pause, and test the revised workflow on a new sample.

Earn the next context

A responsible rollout is not slow by definition. It is staged. Name an evidence owner for each context. Define the minimum subgroup sample, stop rules and correction route before launch. Monitor data drift and outcome drift after launch, not just uptime. Revalidate after a material change in geography, language, policy, vendor model or release process.

The goal is not to demand perfect prediction. It is to stop an average from hiding a pattern that matters.

If the foundation is inconsistent data across programmes or offices, begin with AIHI’s Data Validation solution and the readiness assessment below. Build one transparent matrix, invite the people closest to the work into the review, and make expansion something the evidence earns. That is how AI grows without leaving the mission behind.