An assistant puts a number beside its answer: 80 percent. The sentence looks measured. You are no longer reading an opinion; you seem to be holding an instrument. But what, exactly, has been measured?
There are two different questions here. Does a scoring method behave sensibly across many answers? And is this particular answer supported by evidence? The first cannot quietly take the place of the second.
A hundred answers, not one verdict
Consider a deliberately simplified example—not an experiment or a result about any product. A system gives 100 comparable answers, assigning each a confidence of 80 percent. An independent check finds 80 correct answers. For that group, the stated confidence matches the observed frequency of success.
There are still 20 wrong answers on the table.
The group-level match does not tell you which answer is wrong. Nor does this single group establish that the scoring method will remain calibrated on other questions. A useful evaluation needs enough cases, clear scoring rules and an account of what kinds of questions were tested.
The standard group-level interpretation is set out in Guo and colleagues’ 2017 paper, section 2, in the context of classification models. It does not establish the calibration of a present-day conversational assistant.
Calibration is not a stamp placed on a sentence. It is a relationship assessed across predictions and outcomes.
A good average can hide a weak corner
Now split that same hypothetical group. Ninety answers concern familiar subjects; 78 are correct. Ten concern a less familiar subject; only two are correct. The total is still 80 correct out of 100. The overall match remains, but it hides a large difference between the two groups.
This is an arithmetic illustration, not a measured result. Its lesson is about scope: a broad evaluation can be informative without establishing that a score is calibrated for every subgroup. If your question belongs to the weak corner, the overall average is not the whole story.
The number needs a history
Before interpreting a confidence percentage, ask where it came from. A number written in the generated reply, a probability assigned to a token and a separately evaluated answer-confidence score are not interchangeable measurements.
Suppose the evaluated task involved short factual questions in English. Your question instead asks for a synthesis of several changing local regulations. The earlier evaluation may still offer useful information about the system, but its relevance to this task needs justification. A familiar percentage does not make the new setting familiar.
This is why the useful request is not simply “give me a confidence score.” Ask what was evaluated, how correctness was decided, and whether the present question resembles that evaluation.
Why confident mistakes deserve their own investigation
In a 2025 study by Adi Simhi and colleagues, the authors investigate cases in which models answer questions correctly under one formulation but produce confident errors after small prompt changes. They assess certainty using answer probabilities, probability differences and semantic entropy—not just confident wording. Their reported phenomenon concerns tested models and datasets; it is not a universal error rate for every assistant.
A separate 2025 paper by Adam Tauman Kalai and colleagues analyzes how training and evaluation can reward guessing rather than acknowledging uncertainty. These are research accounts with defined assumptions, not proof that every confident answer is false or that a modest answer is true.
The practical distinction is narrower: how an answer sounds cannot replace checking what supports it.
Keep the score and the source in different columns
When reviewing an answer, keep its confidence claim separate from the evidence for its content. One column describes the scoring method and its evaluation. Another records the source, date, relevant passage and any unresolved conflict. If you cannot fill the first column, the percentage remains unverified. If you cannot fill the second, a well-evaluated score does not conjure a missing source.
Our guide to uncertainty before acting addresses the next decision; the guide to checking AI-generated references addresses the source itself. Here the smaller question comes first: what would make this number meaningful?
You need not reject useful probability estimates. You can ask them to show their work.
A score can describe a track record. It cannot read the evidence for you.
Original AI-generated conceptual illustration: a group of stones and one stone on a balance. It is not a chart or a record of an experiment.
M.R. Orion
