Temperature, Top-P, and Top-K Explained: Controlling LLM Randomness

June 23, 2026 · updated June 25, 2026 · How Claude Actually Works (part 12)

▶ Watch on YouTube & subscribe to The Stack Underflow

Same prompt. Two different answers. Most developers call that “the model being flaky.” It isn’t flaky — it’s sampling. Every time an LLM generates a token it produces a full probability distribution over every word in its vocabulary, then draws from that distribution. The three parameters this tutorial covers — temperature, top-k, and top-p — are the knobs that reshape that distribution before the draw happens. If you don’t understand what they do, you can’t reason about why output varies, why tightening them alone won’t fix hallucinations, or why the newest frontier models have removed them from the API entirely.

This is a conceptual mechanism that governs every transformer-based LLM in production, not just Claude. Understanding it makes you a better prompter, a more effective debugger, and — when you get to structured output and tool-calling — someone who knows exactly which layer to pull.

The one-sentence version: Temperature flattens or sharpens the token probability curve, top-k discards every token outside the top K ranks, and top-p discards every token below a cumulative probability threshold — all three are applied in that order before the final token is drawn.

The probability distribution under every token

Start with a concrete example. Prompt: “The cat sat on the ___”

Before any sampling parameter touches anything, the model runs its forward pass and produces a logit for every token in its vocabulary — a raw score encoding how strongly the model predicts that token comes next. Those logits are converted to probabilities via the softmax function, which exponentiates and normalizes them so that every probability sits between 0 and 1 and the entire vocabulary sums to exactly 1.

The result looks something like this:

This distribution is the model’s belief about what comes next — a bar chart where each bar’s width is proportional to how likely that word is. The final token is chosen via a weighted random draw: imagine throwing a dart at that bar chart. Most of the time you hit “mat.” Sometimes you hit “sofa.” Occasionally, against the odds, “roof.” That draw is why the same prompt produces different outputs on different runs. It is not randomness added for fun; it is the fundamental output mechanism of autoregressive language generation.

Temperature: reshaping the entire curve

Temperature is a scalar divisor applied to every logit before the softmax converts them to probabilities:

adjusted_logit[i] = raw_logit[i] / temperature

That one operation shifts the entire shape of the distribution:

High temperature (e.g. 1.5)
  Logits squeezed together → softmax sees smaller differences
  → Probabilities flatten out → long tail rises
  → Unlikely tokens get a real shot
  Effect: more varied, more creative, more chaotic

Temperature = 1.0
  Logits unchanged → raw model distribution
  Effect: no reshaping; model default behavior

Low temperature (e.g. 0.2)
  Logits spread apart → softmax amplifies differences
  → Distribution sharpens into a spike on the top token
  → At temp → 0: always the single most likely word
  Effect: deterministic-ish, repetitive, predictable

The critical insight is that temperature does not add knowledge. It only changes how much of the probability mass you allow to flow to lower-ranked tokens. You cannot surface a token the model never assigned probability to. High temperature trades coherence for variety; low temperature trades variety for consistency.

Top-K: a hard cutoff by rank

After temperature reshapes the curve, top-k truncation applies a blunt filter. With k = 3:

Ranked probabilities (after temperature):
  mat    → 0.45   ← keep
  sofa   → 0.20   ← keep
  floor  → 0.12   ← keep
  roof   → 0.07   ✗ discard
  rug    → 0.06   ✗ discard
  ...             ✗ discard all

Renormalize the 3 survivors to sum to 1.
Draw from the 3.

Top-k is fast and simple, but it has a meaningful flaw: it always keeps exactly K tokens regardless of how the probabilities are distributed. If the top 3 tokens account for 99% of the mass, keeping 3 is reasonable. If they account for 3% (the model is genuinely uncertain), keeping only 3 throws away most of the legitimate probability space.

This is why top-p was developed as an alternative.

Top-P: a smarter cutoff by cumulative mass

Top-p (also called nucleus sampling) adapts to the shape of the distribution rather than fixing a head count. With p = 0.90:

Sort tokens highest probability first.
Walk down the sorted list, adding probabilities:
  mat    0.45 → running sum = 0.45
  sofa   0.20 → running sum = 0.65
  floor  0.12 → running sum = 0.77
  roof   0.07 → running sum = 0.84
  rug    0.06 → running sum = 0.90  ← stop here
  (everything below discarded)

Renormalize the nucleus {mat, sofa, floor, roof, rug} to sum to 1.
Draw from it.

The nucleus — the smallest set of tokens whose cumulative probability reaches p — is not a fixed size. It shrinks when the model is confident (hitting 0.90 in 2 tokens) and expands when the model is uncertain (needing 40 tokens to reach 0.90). That adaptability is why top-p is the preferred default for general-purpose inference.

Peaked distribution (model confident):
  Nucleus hits p=0.90 after 2-3 tokens.
  Output stays tight.

Flat distribution (model uncertain):
  Nucleus takes 30-40 tokens to reach p=0.90.
  Output stays appropriately open.

Same knob, different nuclei — it self-adjusts.

How the three parameters stack together

The three parameters do not operate independently — they form a sequential pipeline, and the order matters:

Raw logits (one per vocabulary token)
    │
    ▼  Step 1 — Temperature scaling
       adjusted_logit[i] = raw_logit[i] / temperature
       → softmax converts to probabilities
    │
    ▼  Step 2 — Truncation (top-k OR top-p, or both)
       Top-k: keep the K highest, discard rest
       Top-p: keep the smallest set summing to P, discard rest
       (If both are set, the stricter cutoff dominates)
    │
    ▼  Step 3 — Renormalize survivors to sum to 1
    │
    ▼  Step 4 — Weighted random draw → one token

Temperature changes the shape that top-p will subsequently measure. If you raise temperature to 1.5 (flattening the distribution) and then set a tight top-p of 0.70, you first spread probability mass across the tail and then cut off at 0.70 — meaning the nucleus includes more unusual tokens than a top-p of 0.70 would at temperature 1.0. You cannot reason about top-p in isolation from temperature.

Most APIs let you set temperature alongside either top-k or top-p. Setting both top-k and top-p at once is allowed in most engines; the stricter truncation wins, and the interaction is hard to reason about. Pick one.

2026 update: frontier Claude models no longer expose these knobs

Here is the most important practical fact for Claude users in 2026: on Claude Opus 4.7, Claude Opus 4.8, and later models in the Opus 4.x family, the temperature, top_p, and top_k request fields are not exposed at the API. Sending a non-default value returns a 400 error (confirmed in Anthropic’s migration guide, May 2026).

Instead, Claude Opus 4.8 exposes an effort parameter that governs how much adaptive thinking — internally calibrated token budgets, not user-tunable temperature — the model applies to a request:

On Claude Sonnet 4.6 and earlier models, temperature/top-p/top-k remain available and behave exactly as described in this tutorial. The conceptual mechanism — temperature → truncation → draw — applies universally to every transformer-based model; what changes is whether the API surface hands you those knobs directly or manages them internally.

When to tune sampling parameters (on models that expose them)

One rule of thumb worth internalising: if you need a guaranteed output schema, do not try to enforce it with low temperature. Temperature controls variety; it does not enforce JSON validity, field presence, or type correctness. Use tool-calling or structured output mode for schema conformance — that path bypasses sampling for constrained tokens entirely. See how Claude uses tools for the mechanics.

A second rule: if your model keeps producing vague or unhelpful output, the culprit is almost always missing acceptance criteria in the prompt, not a temperature problem. Tightening temperature on a poorly specified prompt gives you a consistently bad answer faster. That problem is addressed directly in the next tutorial in this series.

Common misconceptions

• “Temperature 0 means the model is fully deterministic.” Mostly true in practice, but floating-point arithmetic, batching order, and GPU non-determinism can introduce tiny differences even at temperature 0. If you need bit-for-bit reproducibility, set a fixed seed in addition to temperature 0 — and verify that the API you’re using actually supports seeds.

• “High temperature makes the model smarter or more creative.” It makes the model less constrained, not more capable. The knowledge encoded in the weights is fixed; temperature only determines how far down the probability tail you’re willing to draw. You can easily get more-varied wrong answers at high temperature — the range of outputs expands, but so does the error rate.

• “Top-p and top-k do the same thing.” Both truncate the distribution, but by different criteria. Top-k is rank-based: always exactly K survivors regardless of probability spread. Top-p is mass-based: the nucleus size adapts to the distribution’s shape. Top-p handles both peaked and flat distributions gracefully; top-k requires manual tuning for each task context.

• “Lowering temperature reduces hallucinations.” Only sometimes, and only a little. If the model’s highest-probability token is an incorrect fact, temperature 0 will deliver that wrong fact confidently every single time. Hallucination is primarily a training and prompting problem, not a sampling one. The right tools are better context, clearer constraints, and retrieval — not tighter sampling.

Frequently asked questions

What temperature should I use with Claude by default? On older Claude models (Sonnet 4.6 and earlier), the API default is 1.0. For most production tasks — structured data, factual Q&A, code — start at 0.3–0.6 and lower from there if you need tighter consistency. Only go above 1.0 if you specifically want high-variance output and are prepared to filter results. On Claude Opus 4.7+ these parameters are not accepted; use effort to calibrate reasoning depth instead.

Can I set both top-k and top-p at the same time? Yes, most inference engines support it. The two truncations run in sequence; whichever cuts more tokens wins. In practice, pick one — mixing them makes the effective nucleus harder to reason about and adds no consistent benefit.

Why does the same temperature feel “wilder” for some prompts than others? Because the base distribution varies by prompt. A question the model has high confidence about (e.g., “what is 2+2?”) produces a sharply peaked distribution before temperature even touches it. Temperature then amplifies an already-peaked curve only slightly. A genuinely ambiguous or creative prompt produces a flat distribution to begin with; the same temperature amplifies that flatness substantially. Same knob, different starting curves.

If I need deterministic output, is temperature 0 enough? Not quite. Set temperature to 0, use a fixed seed if the API supports it, and for format conformance use structured output or tool-calling mode. Temperature controls sampling randomness; schema enforcement requires a separate mechanism that constrains the token choices available at each position.

Does nucleus sampling have any downsides? Yes. When the model is uncertain, top-p keeps a wide nucleus — which can mean including genuinely bad tokens that happen to have non-trivial probability mass. This is why very flat distributions (high model uncertainty) still produce incoherent output even with well-tuned top-p. The parameter cannot compensate for a model that simply does not know the answer. Some newer inference engines combine top-p with min-p (a floor on per-token probability relative to the top token) to reduce this effect.

Why did Anthropic remove temperature from the Opus 4.7+ API? Anthropic has not published a complete design rationale, but the direction is consistent with frontier model development more broadly: sufficiently capable models with internal adaptive reasoning produce better results by calibrating their own inference depth than by having humans tune a temperature scalar. The effort parameter gives practitioners a meaningful control surface — less granular than temperature, but more predictable in its effect on quality and cost.

Where this fits in the series

Understanding sampling mechanics is the foundation for several related concepts in this series. Tokenization explains what the vocabulary is that sampling draws from. The context window explains what the model is reading when it computes those logits. Few-shot prompting makes the argument directly: a well-chosen example set reshapes the effective distribution far more reliably than temperature tuning does — and the companion tutorial on pinning behavior with few-shot examples shows you how to apply that in practice. The next tutorial in sequence covers acceptance criteria — the prompt-side tool that prevents vague output without touching sampling at all. Browse all tutorials to follow the full series.