The Temperature article covered a setting that reshapes an entire probability distribution. This article covers a different kind of control, one that limits which tokens are even eligible to be picked in the first place: top-k sampling.
The simple definition
Top-k sampling restricts token selection to only the k tokens with the highest probability, completely discarding every other option, however small its probability, before any random sampling happens. Recall from the Sampling article’s original mention: if k is set to 40, only the 40 most likely next tokens are even considered — everything outside that top-40 list, no matter how the probability distribution is shaped, has zero chance of being selected, full stop.
Why this matters even when temperature is already doing real work
Recall from the Temperature article that a higher temperature flattens the probability distribution, giving low-probability tokens a more meaningful chance of being picked.
This is genuinely useful for variety, but it comes with a real risk: at a high enough temperature, even a token with a tiny, near-zero original probability — something genuinely bizarre or nonsensical in context — gets some real chance of being selected.
Top-k solves this specific problem directly, by drawing a hard line: no matter how flat the distribution gets, only the k most plausible candidates are ever in the running at all.
flowchart LR
A[Full vocabulary: 50,000+ possible tokens] --> B[Top-k filter: keep only the k highest-probability tokens]
B --> C[Sampling happens only among those k tokens]
D[Everything outside the top k] --> E[Zero chance of selection, regardless of temperature]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a hiring manager reviewing 200 job applications, but only seriously considering the top 10 candidates by resume score for the actual interview process — everyone ranked 11th or lower is simply out of contention from the start, no matter how the final interview decision plays out among those top 10.
Where this breaks down: A hiring manager’s shortlist is based on holistic human judgment.
Top-k’s cutoff is a purely mechanical rule based only on the raw probability ranking calculated by the model, as covered in the Probability Distribution article — a fixed number, applied identically at every single token generation step, with no case-by-case judgment about whether 10, or 40, or 100 is the “right” number of candidates for this specific moment.
How top-k and temperature actually work together
This is worth being concrete about, since these two settings are frequently used in combination, not as alternatives to each other. Top-k first narrows the field down to the k most plausible candidates; temperature then reshapes the relative probabilities within that narrowed field, before the actual random sampling happens.
The two settings are addressing genuinely different concerns — top-k answers “which tokens are even allowed to be considered,” while temperature answers “how should the odds be weighted among the ones that are.”
flowchart LR
A[Full probability distribution] --> B[Top-k: narrow to k candidates]
B --> C[Temperature: reshape probabilities among those k]
C --> D[Sample one token]A concrete example, layered
For a simple beginner example: given the sequence “The weather today is” with top-k set to 5, only the 5 most probable next words — perhaps “sunny,” “cloudy,” “rainy,” “cold,” and “warm” — are even eligible for selection, no matter how high the temperature is turned up.
A wildly improbable word like “purple” or “elephant,” even with some small original probability, simply can’t be chosen, since it never made it into the top-5 shortlist at all.
For a production example: several major AI provider APIs, including systems built on the Hugging Face Transformers library widely used across the industry, expose a top_k parameter directly to developers, commonly used in combination with temperature specifically to allow controlled creative variety while still guaranteeing a hard floor of basic plausibility.
Why a fixed k has a real, honest limitation
It’s worth being direct about a genuine weakness in this approach, one that motivates the very next article. A single fixed number of candidates doesn’t adapt to how confident or uncertain the model actually is at any given moment.
When a model is highly confident — like predicting the word after “The capital of France is” — the top 2 or 3 tokens might already account for nearly all the real probability, making a top-k of 40 pointlessly generous, including dozens of essentially irrelevant options.
When a model is genuinely uncertain, a fixed top-k of 40 might actually be too narrow, cutting off reasonable candidates that happened to rank just outside the cutoff.
This exact mismatch — a fixed number not adapting to the shape of the distribution — is precisely the problem the next article, Top-p, is built to solve.
Filter a real probability table
Possible next token Probability
blue 0.40
clear 0.25
bright 0.15
gray 0.10
green 0.06
sings 0.04With top-k = 3, only blue, clear, and bright remain. Their original probabilities total 0.80, so sampling renormalizes them:
blue 0.40 / 0.80 = 0.5000
clear 0.25 / 0.80 = 0.3125
bright 0.15 / 0.80 = 0.1875Every other token receives zero sampling probability for this step. At the next generated position, the model calculates an entirely new distribution and applies top-k again.
Not every API exposes top-k. Google’s Gemini API supports it for models whose model metadata permits it; OpenAI APIs commonly expose temperature and top-p but may not expose a top-k parameter. Generation controls are API features, not guarantees shared by every model provider.
Real-world example: keeping suggestions sensible
Imagine a child asks a story assistant to finish “The astronaut opened the door and saw…” The model may assign high probability to Earth, stars, and a, while thousands of unrelated tokens receive tiny probabilities. With top-k = 3, only the three strongest candidates can be sampled at that step.
At the next step, the candidate list is rebuilt because the context now includes the chosen token. Top-k therefore acts like a teacher who selects a fresh shortlist for each blank, not one permanent shortlist for the entire story.
Gemini exposes top-k only for models that support that setting. Some GPT APIs do not expose top-k directly. A production application must inspect the exact model’s API rather than assuming every sampling control exists everywhere.
Common misconception
A frequent beginner assumption: that top-k and temperature do essentially the same job, just described differently, and that using both together is redundant.
As the “how they work together” section explained, this isn’t accurate — they solve genuinely different problems (which tokens are eligible, versus how the odds are weighted among them), and combining them thoughtfully is a common, deliberate practice, not a redundant overlap.
Where this fits in what comes next
You now understand how top-k provides a hard, fixed-size safety net against implausible tokens. The next article, Top-p, covers a related, more adaptive technique — one that adjusts how many tokens are considered based on the actual shape of the probability distribution at each specific moment, rather than always using the same fixed number.
In one sentence
Top-k sampling restricts a model to choosing only among its k most probable next tokens, providing a hard, reliable floor against implausible output regardless of temperature — though its one real limitation, a fixed cutoff that doesn’t adapt to how confident or uncertain the model actually is, is exactly what the next article’s more flexible technique addresses.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed