Understanding temperature, top-p and top-k in LLM inference

LLMs predict one token at a time. To do that, they output logits for all the words (actually, tokens) in the vocabulary. These logits are then used to calculate probabilities for each of them.

Logits are just values (ranging from $-\infty$ to $\infty$) which are calculated based on corresponding weights & hidden states from the previous layer.

Usually, in the absence of temperature parameter, the probabilities are calculated using the following function, called softmax

$$\sigma(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K}e^{z_j}}$$

Where $K$ is the vocab size.

Or if you prefer it as a Python function:

import math

def softmax(logits):
	exp_logits = [math.exp(l) for l in logits]
	sum_exp_logits = sum(exp_logits)
	return [el/sum_exp_logits for el in exp_logits]

logits = [5, 1, 0.4, -6.7]

print(softmax(logits))
# rounded to [0.97, 0.02, 0.01, 0.0]

Softmax is a way to normalize the logit values into conditional probabilities. Larger logits mean larger probabilities.

What happens now is, in theory, we do a weighted random choice based on the probabilities. We randomly select a token using the calculated probabilities as weights.

Let’s look at an example. Assume there are only four tokens, [t1, t2, t3, t4] in the vocabulary. The logits corresponding to those four tokens, [5, 1, 0.4, -6.7], lead to probabilities approximately [0.97, 0.02, 0.01, 0.0]. This means if we sample 100 times, on average, token t1 is expected to be chosen 97 times, token t2 2 times, token t3 once and token t4 zero times.

So the vast majority of the time, token t1 is going to be chosen. This might lead to repetitive or less creative model outputs.

Temperature

This is where temperature comes into the picture. Temperature $T$ is usually set to a value between $0$ and $2$ (mathematically there is no upper bound). Before the softmax normalization, the logits are divided by $T$. The effect of temperature on the final probabilities depends on the value of temperature in the following ways:

$T=1$: The logits remain unchanged, so do the probabilities.

$T<1$: The gap between larger logits and smaller logits increases. This is because dividing any two numbers by a value between 0 and 1 widens the gap between those two numbers. When softmax is applied, because of this increased gap between the logits, higher probabilities will become disproportionately higher, while lower probabilities will become disproportionately lower.

More generally, with $T<1$, more likely tokens will become much more likely and less likely tokens will become much less likely. This will lead to even more repetitive and deterministic model outputs.

$T>1$: This will have the opposite effect. Dividing by $T>1$ will bring the logits closer to each other. This will have an effect of bringing the probabilities of tokens closer to each other. This brings down the probabilities of higher ranked tokens while lifting the probabilities of lower ranked tokens. This will lead to more creative outputs, allowing the model to explore diverse paths to complete the task. But very large temperature values might lead to incoherent responses and hallucinations.

Temperature is often used along with filtering techniques such as Top-p & Top-k.

Top-k

Recent open source models in 2026 have around 100k-250k tokens in their vocabulary. Thus at all times, a large number of tokens will have tiny ~0.00001 probabilities but it is still possible that such tokens get chosen. We want to avoid that.

Top-k is a filtering technique to only sample from the top k highest probability tokens.

Top-p

Top-p is a filtering technique to only sample from the smallest set of top tokens whose cumulative probability equals or exceeds $p$. In other words, we start from the token with the highest probability and keep including the next highest probability tokens until the total probability equals or exceeds $p$. We will only sample from this set of tokens.

For both top-k and top-p, the probabilities of selected tokens are re-normalized such that the sum of their probabilities totals to 1.

Order of operations

In practice, logits —> divide by temperature —> apply top-k filter —> get probabilities with softmax —> apply top-p filter —> re-normalize probabilities —> sample

In inference engine implementations, top-k is applied directly to logits because it results in the exact same token set as when applied to probabilities while also avoiding running softmax for the whole vocabulary.