The takeaway: When a language model writes, it doesn't "choose" its next word so much as roll a loaded die. At each step it produces a probability for every possible next token, and a sampling step picks one. The settings you may have seen — temperature, top-p, top-k — are the dials that control how that die is loaded: how much the model plays it safe versus takes a risk. Turn them down and you get careful, repeatable, sometimes dull output; turn them up and you get variety, at the risk of nonsense.

What the model actually outputs

A model doesn't emit words directly. For each next-token slot it produces a logit — a raw score — for every token in its vocabulary (tens or hundreds of thousands of them). A function called softmax turns those scores into a probability distribution: maybe "the" gets 8%, "a" 5%, "quantum" 0.001%, and so on. Generating text means repeatedly sampling one token from that distribution, appending it, and doing it again.

The simplest strategy is greedy decoding: always take the single highest-probability token. It's fully deterministic, but it tends to be repetitive and flat — real language isn't always the most probable next word, and greedy models are prone to loops ("the the the"). Every dial below is a way of introducing controlled randomness to fix that.

Temperature — how sharp or flat the odds are

Temperature rescales the logits before the softmax. Divide the scores by a number T:

• T below 1 sharpens the distribution — the likely tokens get even likelier, so the model becomes more focused and deterministic.
• T above 1 flattens it — unlikely tokens get more of a look-in, so output gets more varied and "creative."
• T = 0 is effectively greedy decoding: always the top token.

The popular shorthand "temperature = creativity" is a useful metaphor but a loose one. What temperature really controls is randomness, and past a point that randomness stops reading as creativity and starts reading as incoherence.

Top-k and top-p — trimming the candidate pool

Temperature reshapes the whole distribution; the other common dials instead cut it off so the model never picks from the long tail of absurd options.

• Top-k keeps only the k most-likely tokens and samples among those. Simple, but a fixed k is clumsy: sometimes there are only two sensible next words, sometimes fifty.
• Top-p (nucleus sampling) is smarter. It keeps the smallest set of tokens whose probabilities add up to at least p (say 0.9), then samples from that set. When the model is confident, the pool is tiny; when it's genuinely unsure, the pool widens. That adaptiveness is why top-p is the more common default.

A newer variant, min-p, keeps tokens whose probability is at least some fraction of the top token's — another way to prune the tail that behaves well at high temperatures. Whatever the cut, the surviving probabilities are re-normalised — rescaled to add back up to one — before a token is sampled. In practice these are combined: a top-p cut and a temperature, plus sometimes a repetition penalty that nudges the model away from words it just used.

Which settings, when

There's no universally "correct" setting — it depends on the job:

• Low temperature, tight top-p for tasks that need to be right and consistent: code, factual extraction, structured output, maths. You want the model's best guess, not its imagination.
• Higher temperature, looser top-p for brainstorming, fiction, marketing copy, or anywhere you want range and are willing to curate.

It's worth knowing that these are inference-time dials — they change how a finished model's probabilities are turned into text, not what the model knows. They can't add facts the model lacks, and cranking temperature won't make a small model smart; it will just make it wilder. But understanding them is the difference between an AI that reliably does what you asked and one that either bores you or goes off the rails — and it's usually one of the few knobs you actually get to turn.

Sources: The Curious Case of Neural Text Degeneration (Holtzman et al., 2019 — nucleus/top-p sampling), Hugging Face — Generation strategies, Hugging Face — Text generation configuration.