Sampling Parameters and Determinism

At each step of generating a response, a model produces a probability distribution over every possible next token, then samples one. Two parameters shape that sampling from different angles. Temperature reshapes the whole distribution: a low value sharpens it toward the single most likely token (near 0, generation is close to always picking the top choice), a high value flattens it so less-likely tokens get a real chance. top_p, or nucleus sampling, works differently — it truncates the distribution to the smallest set of tokens whose cumulative probability reaches p, then samples only from that reduced set, leaving the shape of what remains untouched.

Because the two act on the same step in different ways, the common guidance from providers is to adjust one or the other, not both at once, in a single request.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[{"role": "user", "content": "Suggest five taglines for a coffee shop."}],
    temperature=1.1,
    top_p=1.0,
)
print(response.choices[0].message.content)

A third knob, seed, targets reproducibility rather than variety: pass the same integer seed alongside otherwise identical parameters, and you get closer to the same output across repeated calls. It is not a hard guarantee — OpenAI's own guidance describes seeded requests as producing "mostly" consistent output, and ties this to a system_fingerprint value that changes if the backend configuration changes underneath you, which can itself shift results even with a fixed seed. max_tokens (or max_completion_tokens) rounds out the core set: it bounds length, not randomness, but a response cut short by this limit can look like a sampling problem if you're not watching the stop reason.

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    temperature=0.0,
    messages=[{"role": "user", "content": "Extract the total from this invoice: ..."}],
)
print(message.content)

The practical implication cuts across the whole course: Anthropic's current guidance for its newest models is to steer tone and creativity through prompting instead of sampling parameters, since temperature and top_p simply aren't honored there — while OpenAI, at the same point in time, keeps temperature and top_p as live, documented parameters across its current model families. This is exactly the kind of surface where a technique this course teaches (a knob exists, and does something specific) coexists with a detail that keeps shifting (whether a specific model still honors that knob). Always check a target model's current parameter support before you ship around it.