What is softmax and why can its confidence mislead?

Softmax is the step that turns a model's raw scores, called logits, into positive numbers that add up to 1 and are read as probabilities. Each score goes through an exponential and is then divided by the sum of all those exponentials: the ranking stays the same, and gaps between scores become ratios between shares. When the gap is wide, the top score takes almost everything. The name gives it away: it is a "soft" version of picking the maximum, which, instead of handing everything to the winner, spreads the weight gradually, and that smoothness is what lets the model learn through it during training. It shows up in three key places in today's AI: in the output layer of a neural network that chooses among several classes, in the attention weights of Transformers, which set how much each word matters to the others, and in an LLM's next-token distribution, which sampling parameters such as temperature and top-p reshape.

See also: Confidence Calibration · How well an AI model's stated confidence actually matches its real probability of being right.

It looks like a probability, but it is no guarantee

The softmax number looks like the probability of being right, and dashboards and APIs often show it as "confidence". It comes with three caveats. The first is structural: softmax always splits 100% among the classes the model knows, so an input that belongs to none of them still gets a winner, and there is no built-in "none of these" answer. In 2015 Nguyen, Yosinski and Clune showed that an optimization algorithm can build images unrecognizable to a person, such as white noise, that state-of-the-art networks labeled as familiar objects with 99.99% confidence. The second is about calibration: in 2017 Guo and colleagues documented that modern networks, unlike those of a decade earlier, are poorly calibrated: the stated number does not match how often they are actually right, and they lean toward overconfidence. The third is about how much the signal is worth: Hendrycks and Gimpel observed that the maximum probability tends to be higher for correct answers than for wrong ones or for inputs unlike anything in the training data. The signal is real and works as a baseline, one that the same authors note other methods sometimes beat, so treat it as a hint that still needs checking.

An enterprise example

An insurer routes incoming paperwork with a document classifier: claim notice, invoice, claims adjuster report, complaint. Above 90% "confidence" the document goes automatically to the right queue; below that, to a clerk. Then in comes a mediation record, a document type the model never saw in training. Softmax splits 100% across the four known classes, one of them wins, perhaps at 95%, and the record is filed automatically with the adjuster reports. Raising the threshold is not enough: an unfamiliar input can still clear it, and Hendrycks and Gimpel measured an average of 91% on random-noise images fed to a digit classifier. What works is a combination: a check that recognizes unfamiliar inputs or an explicit "out of scope" class; abstention rules that hand uncertain cases to a person; and, for the known classes, probabilities recalibrated on a recent holdout, most cheaply with temperature scaling.

Why it matters for decision makers

When a vendor shows a "confidence" figure, the first question to ask is where it comes from. Is it raw softmax or a calibrated probability? If calibrated, on which data, and how close is that data to the company's real cases? What happens to an input that belongs to none of the expected classes: does the system recognize it, abstain or pick a winner anyway? An automation threshold built on an unchecked number moves risk from the process to the model without saying so.