Evaluation, Cost, Latency, and Safety in Production

Reading through ten sample outputs and deciding "looks good" is not evaluation — it's a vibe check, and it stops scaling the moment you change a prompt, swap a model, or ship a fix and need to know whether anything got worse. An evaluation set replaces that with a fixed, representative collection of inputs — ideally with a reference answer or some other ground truth attached — that gets run and scored the same way every time, so a change to your prompt or model produces a number you can compare against the last number, not just a feeling.

The value isn't in running it once. It's in running the same eval set again every time something changes — a new model version, a reworded system prompt, a different retrieval pipeline — so a regression shows up as a dropped score before a person notices it in production.

OpenAI's evaluation tooling documents five grader types, and picking the right one for what you're actually checking matters more than picking the most sophisticated one. A string check does exact-match comparison against a reference — right for a fixed, correct answer with no acceptable variation. Text similarity uses embeddings to measure how semantically close an output is to a reference, useful when the wording can vary but the meaning shouldn't. A score model grader uses an LLM to assign a numeric score against criteria you describe — friendliness, helpfulness, adherence to a style guide — properties too subjective for exact matching. A label model grader uses an LLM to pick a category from a fixed list — concise versus verbose, on-topic versus off-topic. And Python code execution runs your own logic against the output — checking a word count, a required substring, whether it parses as valid JSON.

Reaching for a model-graded check when a code check would do wastes a model call on something deterministic; reaching for exact-match on something inherently variable in wording produces false failures on a genuinely correct answer.

def grade_response(question, response, rubric):
    judge_prompt = f"""Rate the response from 1-5 against this rubric: {rubric}

Question: {question}
Response: {response}

Respond with only the number."""
    result = call_model([{"role": "user", "content": judge_prompt}])
    return int(result.strip())

Automated graders are fast enough to run on every change, but they're not the only source of ground truth. Human annotation — ideally from someone with actual subject-matter expertise in what's being evaluated — catches what automated graders miss: subtle tone problems, infrequent edge cases, whether a response actually satisfies what a real user needed rather than just matching a pattern. A good annotation does double duty: it's a judgment call in its own right, and it's also the material you use to write or refine the automated graders that will run on the next thousand examples.

The practical workflow treats the eval set as something that grows: every time a real failure surfaces in production or testing, it becomes a new case in the set, so the same class of mistake gets caught automatically the next time, rather than being rediscovered by a person.

Two requests that produce the same quality answer can cost wildly different amounts and take wildly different times, and neither difference has anything to do with accuracy. Cost and latency are their own axes to optimize, separately from correctness, and the biggest lever for both — on both major providers — is reusing computation you've already paid for: prompt caching.

The core idea is simple. If a later request starts with the exact same tokens as an earlier one — the same system prompt, the same tool definitions, the same long reference document — the model doesn't need to reprocess that shared prefix from scratch. It reuses the saved intermediate state and picks up from where the new content begins. The prefix has to match exactly, token for token, up to whatever point you're relying on being cached; a single changed word early in a long system prompt invalidates everything after it.

OpenAI's caching is automatic and enabled by default for supported models — you don't add any parameter to benefit, though there's a minimum prefix length (documented around 1,024 tokens for current models) below which a request isn't eligible at all, and cached-portion tokens are billed at a steep discount. Anthropic's works the other way: you mark a cache_control: {"type": "ephemeral"} breakpoint explicitly on the content block you want cached, with up to four breakpoints per request. Anthropic's cache writes actually cost more than an ordinary request the first time, but cache reads afterward cost a small fraction of the normal rate — a net win only once that prefix gets reused at least once.

Both providers reward the same discipline regardless of the mechanism: put the stable part of your prompt — system instructions, tool definitions, reference documents — first, and put whatever changes on every single call — today's date, this specific user's question — last. Put the volatile piece first and it breaks the cacheable prefix for every request that follows it.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": long_reference_document,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "Summarize section 3."}],
)

When a task doesn't need an answer right now, both providers offer an asynchronous batch endpoint at roughly half the standard price, in exchange for a turnaround window — documented as up to 24 hours, often faster in practice — instead of an immediate reply. This is a strong fit for anything with no live user waiting: bulk classification, running an evaluation set, offline report generation. It's a poor fit for anything interactive, where a user is watching a screen for a reply.

The other lever is model choice itself: routing easy or common requests to a smaller, cheaper, faster model and reserving a larger model for the genuinely hard cases — the same routing pattern from the agents module, applied here to cost and latency rather than task specialization. A well-tuned router can cut average cost substantially without touching the accuracy of the hard cases that actually need the bigger model.

An earlier module covered defending against prompt injection — an attacker steering a model through instructions hidden in a tool result or a document. Production safety is broader than that one failure mode. Independent of any tool use at all, you generally need to screen what users type in and what the model writes back for content that violates your application's policy — hate speech, harassment, self-harm content, sexual content, violent content — regardless of whether an attacker was involved or the request was entirely sincere. That's what a content moderation classifier is for: a dedicated, purpose-built check on content itself, separate from whether the content came from a trusted or untrusted source.

from openai import OpenAI

client = OpenAI()

result = client.moderations.create(
    model="omni-moderation-latest",
    input="User-submitted text to check before it reaches the main model.",
)
flagged = result.results[0].flagged
categories = result.results[0].categories

OpenAI's moderation endpoint, currently built around its omni-moderation-latest model, classifies both text and image inputs (not audio), is free to use, and returns a flagged boolean alongside per-category categories and category_scores fields. You can also request moderation scores inline alongside a normal Responses API call, scoring both the input you sent and the output the model produced, without a separate request.

The documented nuance worth internalizing: OpenAI's own guidance is to treat these scores as signals for your application's policy, not as an automatic blocking decision. A model's own safety-aware refusal can discuss the harmful topic it's declining to help with, and that discussion can itself trip a category flag — so a blanket "flagged means block" rule can end up suppressing the exact safe refusal you wanted to reach the user.

Content moderation, prompt-injection defenses, and abuse-pattern monitoring cover three different failure modes, and none substitutes for the others. Moderation classifies a single piece of content against policy categories. The prompt-injection defenses from earlier in this course — labeling untrusted content, stating an untrusted-data policy in the system prompt, screening tool outputs — address an attacker trying to steer the model through content it reads, which a policy-category classifier isn't designed to catch at all. And rate limiting or abuse-pattern monitoring — capping requests per account, flagging accounts that repeatedly trigger the same refusal — addresses volume and persistence: one message might look entirely innocuous on its own while a pattern of hundreds of near-identical attempts from one account is the actual signal something is wrong.

A production system needs some version of all three, aimed at the failure mode each one actually catches.