The Logits article ended with raw, unconverted scores waiting to become something usable. This article covers what they actually turn into: a probability distribution.
The simple definition
A probability distribution is the complete set of probabilities across every possible outcome, where every value is between 0 and 1, and the whole set adds up to exactly 100%. For next-token prediction, that means one probability for every single token in the model’s entire vocabulary — 50,257 numbers, in GPT-3’s case, as confirmed in OpenAI’s published paper — all calculated together, all guaranteed to sum to 100%, no matter how confident or uncertain the model is about any individual token.
How logits actually become a probability distribution: softmax
Recall from the Prediction article’s earlier mention of softmax, and the Logits article’s promise of “the conversion process.” Softmax is the specific mathematical function that performs this conversion. It takes the raw logits — any positive or negative numbers, no fixed total — and transforms them into a proper probability distribution: every value becomes positive, and the whole set gets scaled so it sums to exactly 1 (100%). Crucially, softmax preserves relative ordering — whichever token had the highest logit still ends up with the highest probability — while also exaggerating the gap between high and low scores somewhat, making a clearly winning logit translate into an even more dominant probability.
flowchart LR
A["Logits: 'blue'=4.7, 'clear'=3.2, 'furniture'=-2.0"] --> B[Softmax]
B --> C["Probabilities: 'blue'=68%, 'clear'=15%, 'furniture'=0.01%"]
C --> D["All 50,257 token probabilities sum to exactly 100%"]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a pie chart representing how a group of 100 people voted on their favorite flavor of ice cream, out of every flavor a shop sells. Every single flavor gets a slice, even flavors nobody actually voted for (a razor-thin, nearly invisible sliver), but all the slices, added together, always make up the complete, whole pie — 100%, no more, no less.
Where this breaks down: A pie chart represents real, countable votes cast by real people. A model’s probability distribution represents calculated confidence, produced by the softmax conversion of logits — a purely mathematical transformation, with no actual “voting” happening; the “votes” are really just the accumulated result of the model’s learned parameters processing the input sequence.
Reading what the shape of a distribution actually tells you
This is genuinely useful to understand, since the shape of a probability distribution — not just its individual values — reveals something real about a model’s confidence at any given prediction step. A sharply peaked distribution, where one token holds, say, 95% of the probability and everything else is negligible, means the model is highly confident about what comes next — often the case after a very predictable phrase like “The capital of France is.” A flat, spread-out distribution, where probability is divided fairly evenly across many plausible tokens, means genuine uncertainty — several different words could reasonably continue the sentence, and the model doesn’t strongly favor any one of them.
flowchart LR
A["Sharply peaked distribution: 'Paris' = 92%, everything else tiny"] --> B[High confidence]
C["Flat distribution: many tokens each around 5-10%"] --> D[Genuine uncertainty]A concrete example, layered
For a simple beginner example: given “The capital of France is”, a well-trained model’s probability distribution would likely be sharply peaked, with “Paris” holding an overwhelming majority of the probability, since that fact is unambiguous and extremely well-represented in training data. For a production example: given the more open-ended prompt “My favorite type of weather is”, a model’s probability distribution across its full vocabulary would be far flatter — “sunny,” “rainy,” “cool,” “warm,” and many other genuinely reasonable continuations would each hold modest, comparable probability, reflecting the real, legitimate ambiguity of an open-ended, subjective continuation rather than a single, unambiguously correct answer.
Why this connects directly to hallucination
It’s worth naming a real, honest limitation here, tying back to a caution raised throughout this glossary. A model’s probability distribution reflects statistical likelihood based on training data patterns — not verified truth. A flat, uncertain distribution across several plausible-sounding but factually wrong tokens can still result in the model confidently picking one of them, with nothing in the distribution itself distinguishing “statistically common phrasing” from “factually correct information” — a mechanical root cause of the hallucination problem discussed in the Prediction and Generative AI articles.
Turn three logits into probabilities
Continue with logits [2, 1, 0]. Softmax first exponentiates them:
e² ≈ 7.389
e¹ ≈ 2.718
e⁰ = 1.000
sum ≈ 11.107Then divide each value by the sum:
P(blue) = 7.389 ÷ 11.107 ≈ 66.5%
P(clear) = 2.718 ÷ 11.107 ≈ 24.5%
P(dark) = 1.000 ÷ 11.107 ≈ 9.0%
Total = 100.0%flowchart LR
A[Logits: 2, 1, 0] --> B[Exponentiate]
B --> C[7.389, 2.718, 1]
C --> D[Divide by 11.107]
D --> E[66.5%, 24.5%, 9.0%]How GPT and Gemini use the distribution
The distribution is recalculated for every generated token. It is conditional on the entire usable context, so adding one selected token changes the next distribution.
The GPT-4 technical report describes multiple-choice evaluation by selecting the A–D continuation with the highest model probability. That is a real example of using token probabilities for evaluation rather than free-form sampling.
High probability means “favored by this model under this context,” not “factually true.” Post-training can also change calibration: the GPT-4 report shows that confidence and real correctness need not align perfectly.
Common misconception
A frequent beginner assumption: that a probability distribution being sharply peaked always means the model’s answer is correct, while a flat distribution always signals something is wrong. Neither is automatically true — a sharply peaked distribution simply means the training data consistently pointed toward one specific continuation, which is often but not always factually correct; a flat distribution often reflects entirely legitimate, healthy ambiguity (many reasonable answers genuinely exist), not necessarily a flaw or failure in the model.
Where this fits in what comes next
You now understand the full, standardized set of probabilities a model calculates at every prediction step. The final article in this phase, Sampling, covers what happens next — how the model actually picks one specific token from this distribution to generate, rather than always mechanically choosing the single highest-probability option.
In one sentence
A probability distribution is the complete, standardized set of probabilities across every possible next token — produced by converting raw logits through softmax — and its shape, sharply peaked or genuinely flat, reveals real, meaningful information about how confident or uncertain a model actually is at any given prediction step.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed