Prompting, Roles, and Structured Output

Every request to a large language model API is a list of messages, and each message carries two things: who is speaking and what they said. The model's whole job is to predict what the next message in that list should be. This sounds trivial, but it is the single idea everything else in this course builds on: prompting is not a monologue you write once, it is a conversation you construct, message by message, and the model only ever sees the list you hand it.

Each message has a role — a short tag telling the model (and anything wrapped around it) what kind of speaker produced that content. The most universal roles are user, for what the person asked, and assistant, for what the model previously said. Everything else — the channel for your own instructions — differs by provider, and getting it right is the first skill in this course.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[
        {"role": "developer", "content": "You are a concise support assistant. Never invent order numbers."},
        {"role": "user", "content": "Where is my order #48213?"},
    ],
)
print(response.choices[0].message.content)

OpenAI's Chat Completions and Responses APIs recognize five message roles: system, developer, user, assistant, and tool. The developer role gives an application's own instructions a clear, higher-priority place in what OpenAI calls its instruction hierarchy — its API reference describes it as carrying instructions the model should follow ahead of anything a user says. Many teams still reach for system out of habit, and the two behave close enough alike for most applications, but developer is the one designed to win when a user's message tries to argue with it.

Anthropic takes a different shape entirely. In the Messages API there is no system-role message you add to the messages array for your opening instructions — instead there's a separate top-level system parameter that sits outside the conversation entirely. A specific set of newer Claude models also accepts a system-role message inside messages, but only after the first user turn, as a way to inject fresh instructions partway through a long conversation without invalidating anything cached before it.

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    system="You are a concise support assistant. Never invent order numbers.",
    messages=[
        {"role": "user", "content": "Where is my order #48213?"}
    ],
)
print(message.content)

None of this is just API trivia. Separating instructions from user content is a real, if partial, defense: a model told "the text below is untrusted input" behaves differently than one that received the same text with no framing at all. It is not a security boundary — a determined attacker can still write user text that reads like an instruction — but it measurably reduces how often a model follows an instruction smuggled inside data it was only supposed to read.

The practical rule of thumb is narrow scope for the instruction channel: role, tone, and non-negotiable constraints belong in system or developer; the specific task, question, and any data to work on belong in user. When that data is long — a document, a codebase, a transcript — put it near the top of the user turn, above your actual question. Providers report this ordering consistently improves how well the model attends to the query that follows it.

Ask a model to "respond only in JSON" and most of the time it will — until it adds a stray sentence before the braces, forgets a required field, or invents a value for an enum you never listed. Early attempts to fix this relied entirely on the prompt: stronger wording, more examples, occasional retries when parsing failed. That approach caps out, because nothing is actually constraining what tokens the model can produce next.

Structured outputs replace prompting with a guarantee. Instead of hoping the model's free-text response happens to parse, you hand the API a JSON Schema, and the API constrains token generation so only schema-valid continuations are possible — a technique usually called constrained or grammar-based decoding. The result isn't "probably valid JSON"; it's JSON that is syntactically guaranteed to match the shape you specified, every time the request completes normally.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[
        {"role": "user", "content": "Extract the contact from: John Smith, john@example.com, Enterprise plan."}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "contact",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "email": {"type": "string"},
                    "plan": {"type": "string"},
                },
                "required": ["name", "email", "plan"],
                "additionalProperties": False,
            },
        },
    },
)
print(response.choices[0].message.content)

OpenAI's newer Responses API defines this under text.format, with type: "json_schema" and your schema attached; the older Chat Completions endpoint uses response_format with the same shape, and OpenAI's own migration guidance says plainly that response_format moved to text.format in the newer API rather than being replaced by something different. Both are distinct from the older, looser json_object mode: json_object only guarantees the output parses as JSON, with no check that it has the fields, types, or enum values your schema demands. Structured outputs, with strict: true on the schema, guarantee both.

The Python and JavaScript SDKs let you define the schema from a Pydantic model or Zod schema instead of hand-writing raw JSON Schema, and a .parse() helper returns an already-validated object. Under the hood it's the same constrained decoding either way — the SDK is just saving you from writing the schema by hand.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Extract the contact from: John Smith, john@example.com, Enterprise plan."}
    ],
    output_config={
        "format": {
            "type": "json_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "email": {"type": "string"},
                    "plan": {"type": "string"},
                },
                "required": ["name", "email", "plan"],
                "additionalProperties": False,
            },
        }
    },
)
print(next(b.text for b in response.content if b.type == "text"))

Anthropic's equivalent lives under output_config.format, again type: "json_schema" plus your schema, using the same constrained-sampling idea: your schema is compiled into a grammar the token sampler can't leave. The first request against a new schema pays a compilation cost; Anthropic's documentation states compiled grammars are then cached for 24 hours from last use, so repeated calls with the same schema shape are fast.

Two quirks are worth knowing before you rely on this in production. First, property ordering: Anthropic's structured outputs always emit required properties before optional ones, in schema order within each group — not necessarily the order you wrote them if you mixed required and optional fields, so don't assume your schema's raw property order survives into the response. Second, enum casing isn't fully guaranteed: the documentation notes a returned enum value can differ from your schema only in capitalization, so compare enum values case-insensitively rather than with strict string equality.

Structured outputs and tool or function calling solve related but different problems, and it's worth being precise about which one you reach for. Use schema-enforced JSON output (text.format / output_config.format) when the model is producing its final answer to your application in a shape you'll parse — a classification label, an extracted record, a UI-ready object. Use tool calling, covered in the next module, when the model needs to decide whether to take an action and what arguments to pass — searching a database, calling a function in your codebase. The two combine: a model can call a tool with a schema-validated argument set and still return a schema-validated final summary to the user in the same turn.

At each step of generating a response, a model produces a probability distribution over every possible next token, then samples one. Two parameters shape that sampling from different angles. Temperature reshapes the whole distribution: a low value sharpens it toward the single most likely token (near 0, generation is close to always picking the top choice), a high value flattens it so less-likely tokens get a real chance. top_p, or nucleus sampling, works differently — it truncates the distribution to the smallest set of tokens whose cumulative probability reaches p, then samples only from that reduced set, leaving the shape of what remains untouched.

Because the two act on the same step in different ways, the common guidance from providers is to adjust one or the other, not both at once, in a single request.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[{"role": "user", "content": "Suggest five taglines for a coffee shop."}],
    temperature=1.1,
    top_p=1.0,
)
print(response.choices[0].message.content)

A third knob, seed, targets reproducibility rather than variety: pass the same integer seed alongside otherwise identical parameters, and you get closer to the same output across repeated calls. It is not a hard guarantee — OpenAI's own guidance describes seeded requests as producing "mostly" consistent output, and ties this to a system_fingerprint value that changes if the backend configuration changes underneath you, which can itself shift results even with a fixed seed. max_tokens (or max_completion_tokens) rounds out the core set: it bounds length, not randomness, but a response cut short by this limit can look like a sampling problem if you're not watching the stop reason.

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    temperature=0.0,
    messages=[{"role": "user", "content": "Extract the total from this invoice: ..."}],
)
print(message.content)

The practical implication cuts across the whole course: Anthropic's current guidance for its newest models is to steer tone and creativity through prompting instead of sampling parameters, since temperature and top_p simply aren't honored there — while OpenAI, at the same point in time, keeps temperature and top_p as live, documented parameters across its current model families. This is exactly the kind of surface where a technique this course teaches (a knob exists, and does something specific) coexists with a detail that keeps shifting (whether a specific model still honors that knob). Always check a target model's current parameter support before you ship around it.