The Top-k article closed on a genuine limitation: a fixed cutoff that doesn’t adapt to how confident or uncertain a model actually is at any given moment. This article covers the fix: top-p sampling, also called nucleus sampling.
The simple definition
Top-p sampling includes just enough of the highest-probability tokens to reach a target cumulative probability — like 90% — rather than always including a fixed number of tokens. If p is set to 0.9, the model adds up probabilities starting from the most likely token downward, stopping the moment that running total crosses 90%, and only samples from that dynamically-sized group.
Unlike top-k’s fixed number, this group can be small or large depending entirely on how the probability distribution is actually shaped at that specific moment.
Why this directly solves top-k’s real weakness
Recall from the Top-k article’s closing limitation: a fixed k of 40 is wasteful when the model is highly confident (the real answer might be covered by the top 3 tokens alone) and potentially too narrow when the model is genuinely uncertain (40 tokens might not be enough to capture every reasonable option).
Top-p adapts automatically to exactly this situation — when the distribution is sharply peaked, as covered in the Probability Distribution article, it takes very few tokens to reach 90% cumulative probability, so the eligible group naturally shrinks to just those few strong candidates.
When the distribution is flat and uncertain, it takes many more tokens to reach that same 90% threshold, so the eligible group naturally grows to include more genuinely plausible options.
flowchart LR
A[Sharply peaked distribution] --> B["Few tokens needed to reach 90% - small, tight group"]
C[Flat, uncertain distribution] --> D["Many tokens needed to reach 90% - larger, broader group"]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a hiring manager who doesn’t commit to a fixed number of interview candidates in advance, but instead keeps adding candidates to the shortlist, starting from the strongest, until the shortlist collectively represents “the clearly strong pool” — sometimes that means just 3 candidates if a handful are obviously far ahead of everyone else, sometimes it means 15 if the whole applicant pool is closely, genuinely competitive.
Where this breaks down: A hiring manager’s sense of “clearly strong” involves genuine judgment about candidate quality. Top-p’s cutoff is a precise, mechanical running sum of calculated probabilities, stopping exactly when a numeric threshold is crossed — no judgment, no case-by-case discretion, just consistent arithmetic applied identically at every single token generation step.
Why this technique earned the name “nucleus sampling”
This is worth knowing, since you’ll see the term “nucleus sampling” used interchangeably with top-p in research and documentation.
The idea is that the small set of tokens accounting for the bulk of the real probability mass forms the genuine “nucleus” of plausible continuations — everything outside that nucleus is, by definition, a comparatively unlikely long tail that contributes little to genuine plausibility.
Sampling only from this nucleus, rather than the full distribution, is exactly the adaptive filtering this article has described.
A concrete example, layered
For a simple beginner example: given “The capital of France is” with top-p set to 0.9, if “Paris” alone holds 92% of the probability, the nucleus might genuinely consist of just that one token — the cumulative threshold is crossed almost immediately, and the model samples with near-certainty from that single dominant option.
Given the more open-ended “My favorite hobby is” with the same top-p of 0.9, reaching that same 90% threshold might require including a dozen or more genuinely plausible hobbies — reading, hiking, painting, cooking — since no single option dominates the way “Paris” did.
For a production example: OpenAI’s API documentation exposes a top_p parameter directly to developers, and it’s common practice to adjust either temperature or top-p, but generally not both aggressively at the same time, since they’re two different mechanisms for controlling roughly the same underlying trade-off between focus and variety.
Why top-p has become a common default choice
It’s worth being direct about why this technique is often preferred over plain top-k in modern practice, not just presenting it as an equally valid alternative.
Because top-p automatically adapts to the model’s actual confidence at each specific step, it tends to avoid both of top-k’s failure modes at once — it doesn’t waste consideration on irrelevant tokens when the model is confident, and it doesn’t artificially cut off reasonable options when the model is genuinely uncertain.
This adaptiveness is exactly why many production AI systems default to top-p (often set around 0.9 to 0.95) rather than a fixed top-k value.
Calculate the nucleus step by step
Use the same sorted distribution:
blue 0.40 cumulative 0.40
clear 0.25 cumulative 0.65
bright 0.15 cumulative 0.80
gray 0.10 cumulative 0.90
green 0.06 cumulative 0.96With top-p = 0.80, the nucleus contains the first three tokens. With top-p = 0.90, it contains the first four. If the model is extremely confident and one token has probability 0.92, that one token alone may form the nucleus at top-p = 0.90.
This adaptive candidate count is the key difference:
top-k → keep a fixed number of candidates
top-p → keep enough candidates to reach a probability massGoogle documents a typical filtering order in Gemini as top-k candidates followed by top-p filtering and temperature-based sampling. The exact implementation and allowed settings remain model-specific, so applications should test the combined behavior rather than reason about each slider in isolation.
Real-world example: an adaptive writing assistant
When completing “Paris is the capital of…”, one token may dominate, so a top-p nucleus can be very small. When completing “The mysterious box contained…”, several endings may be believable, so the same top-p value can retain a larger group.
Confident factual continuation → small nucleus
Open creative continuation → larger nucleusThis adaptability is why nucleus sampling is useful in conversational and creative generation. The API sends top-p as a numeric generation setting. At each decoding step the model sorts the current token probabilities, keeps enough to reach the selected probability mass, renormalizes them, and samples. It repeats the calculation after every newly generated token.
Common misconception
A frequent beginner assumption: that top-p is simply a more complicated version of top-k, doing the same essential job with more steps.
As this article has shown, the genuine difference is adaptiveness — top-k always considers a fixed number of tokens regardless of context, while top-p always considers just enough tokens to cover a fixed probability threshold, a number that changes naturally from one prediction step to the next based on how confident or uncertain the model actually is at that specific moment.
Where this fits in what comes next
You now understand all three of the major sampling controls covered across this glossary — temperature (reshaping probabilities), top-k (a fixed-size safety net), and top-p (an adaptive, probability-based safety net).
The next article, Reasoning, shifts focus away from token-selection mechanics entirely, toward the broader question of what it actually means for a model to work through a complex problem.
In one sentence
Top-p sampling adaptively includes just enough of the highest-probability tokens to reach a target cumulative threshold, automatically narrowing when a model is confident and widening when it’s uncertain — solving top-k’s fixed-cutoff limitation directly, and making it a common default choice in modern production AI systems.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed