A confidence score is not a safety label.

The reassuring number

Picture a programme manager opening a recommendation marked “92% confident”. The number is crisp. The deadline is close. A colleague is waiting for a decision that will shape the support somebody receives.

This is an illustrative scene, not a client story. But the pull of that number is familiar: it feels like an answer to the question, “How much should I trust this?”

A confidence score is not a safety label. It is a claim that still needs evidence.

Useful AI can help good organisations move with greater clarity. Yet a percentage beside an answer can create authority faster than it creates understanding. Before we show it, we need to know what was measured, on which cases, and what a person is expected to do when the number is high, low or plainly misleading.

Start with the denominator

Calibration is a relationship across many comparable predictions. If a system says “80%” on 100 genuinely comparable cases, a well-calibrated system should be right on roughly 80 of them. That does not make any one case safe, and it says nothing by itself about the severity of the twenty mistakes.

For a language model, “confidence” might mean a model probability, a separate estimator, agreement between repeated runs, retrieval quality or simply assertive wording. Those are not interchangeable.

So the first governance question is wonderfully ordinary: 80% of what? Ask who chose the cases, how recent they are, whether they resemble your workflow, and what happened to missing or ambiguous examples. If nobody can explain the denominator, the display is decoration, not decision support.

Words change decisions

In a preregistered study of 404 people, Kim and colleagues tested a fictional search assistant on eight medical yes-or-no questions. The AI was deliberately correct only half the time. Participants who saw answers without uncertainty were correct on an estimated 63.9% of questions. First-person wording such as “I’m not sure” lifted that estimate to 72.8%; the no-AI group reached 74.2%. General statements about uncertainty reached 67.9%, a difference that was not statistically significant from the no-uncertainty condition.

The wording also reduced agreement with incorrect AI answers, but did not erase it. These were lay participants, a short factual task and an artificially unreliable system. We should not copy the percentages into programme risk registers. We can take the design lesson: the way uncertainty is expressed changes behaviour, so wording belongs in evaluation, not just in brand review.

Confidence can rub off

A 2025 study with 270 US participants asked people to make 120 income predictions: forty alone, forty with AI, then forty alone again. While using the system, participants’ confidence moved towards the AI’s confidence. Some of that alignment persisted after the AI was removed. Real-time correctness feedback reduced the effect.

The task was synthetic and the participants were not humanitarian professionals. Still, the result gives us a useful hypothesis: confidence does not merely report a system state; it can train the person looking at it.

The most important confidence signal may be the feedback your team receives after a decision. If people never learn whether a confident recommendation held up, they cannot calibrate their own reliance. A review process should return timely, specific outcomes—not only tell people to “use judgement”.

Updates move the floor

Calibration is not a permanent property of a product name. A 2026 study across 216 configurations found that supervised fine-tuning degraded calibration in 112 and improved it in 104. The direction could not be safely inferred from the pre-tuned model.

That near split is not a universal probability, and the work used translation, question-answering and mathematics benchmarks rather than live service decisions. It does make one operational point hard to ignore: a model update can change the meaning of yesterday’s confidence display.

If confidence changes after the model changes, your controls must change with it. Save the model and prompt version, re-run a protected test set, and compare calibration before release. A new capability score is not enough if people use confidence to decide when to review, escalate or act.

Make confidence testable

Begin with one consequential workflow and a sample that represents the cases your team actually sees. Define the outcome before examining the model’s score. Group predictions into useful confidence bands, then compare stated confidence with observed correctness inside each band. For 50 cases labelled around 80%, for example, 38 correct answers means 76% observed accuracy. That difference may be noise in a small sample; report the count and uncertainty rather than polishing it away.

Then study the decision around the model. Can reviewers open the source? Do they notice when evidence is missing? What happens when a low-confidence case is urgent? Record overrides, escalations, later corrections and harm severity. Protect personal data and use synthetic or de-identified cases before testing on live access to essential support.

Confidence should help a person ask a better question, not end the conversation. If reliable records are the obstacle, our Data Validation solution and AI readiness assessment offer two practical next steps below. The ambition is not a prettier percentage. It is a decision your team can explain, challenge and improve.