AI Engineering: Building Production LLM Applications · Evaluation, Cost, Latency, and Safety in Production

Safety and Abuse Prevention in Production

~12 min

By the end of this lesson

Apply content moderation to screen user input and model output, treating a moderation flag as a signal rather than an automatic decision.
Recognize the layered nature of production safety: moderation, prompt-injection defenses, and abuse monitoring each address a different failure mode.

An earlier module covered defending against prompt injection — an attacker steering a model through instructions hidden in a tool result or a document. Production safety is broader than that one failure mode. Independent of any tool use at all, you generally need to screen what users type in and what the model writes back for content that violates your application's policy — hate speech, harassment, self-harm content, sexual content, violent content — regardless of whether an attacker was involved or the request was entirely sincere. That's what a content moderation classifier is for: a dedicated, purpose-built check on content itself, separate from whether the content came from a trusted or untrusted source.

python

from openai import OpenAI

client = OpenAI()

result = client.moderations.create(
    model="omni-moderation-latest",
    input="User-submitted text to check before it reaches the main model.",
)
flagged = result.results[0].flagged
categories = result.results[0].categories

OpenAI's standalone moderation endpoint, checked before content reaches the main model.

OpenAI's moderation endpoint, currently built around its omni-moderation-latest model, classifies both text and image inputs (not audio), is free to use, and returns a flagged boolean alongside per-category categories and category_scores fields. You can also request moderation scores inline alongside a normal Responses API call, scoring both the input you sent and the output the model produced, without a separate request.

The documented nuance worth internalizing: OpenAI's own guidance is to treat these scores as signals for your application's policy, not as an automatic blocking decision. A model's own safety-aware refusal can discuss the harmful topic it's declining to help with, and that discussion can itself trip a category flag — so a blanket "flagged means block" rule can end up suppressing the exact safe refusal you wanted to reach the user.

Content moderation, prompt-injection defenses, and abuse-pattern monitoring cover three different failure modes, and none substitutes for the others. Moderation classifies a single piece of content against policy categories. The prompt-injection defenses from earlier in this course — labeling untrusted content, stating an untrusted-data policy in the system prompt, screening tool outputs — address an attacker trying to steer the model through content it reads, which a policy-category classifier isn't designed to catch at all. And rate limiting or abuse-pattern monitoring — capping requests per account, flagging accounts that repeatedly trigger the same refusal — addresses volume and persistence: one message might look entirely innocuous on its own while a pattern of hundreds of near-identical attempts from one account is the actual signal something is wrong.

A production system needs some version of all three, aimed at the failure mode each one actually catches.

A single user message passes content moderation cleanly — no category is flagged. Does that mean the message poses no risk to your application?Answer it yourself first, then open this.

Not necessarily. Moderation checks the content of one message against policy categories; it doesn't catch a prompt-injection attempt hidden in a document the model later reads, and it doesn't catch a pattern of many similar low-risk messages from one account that add up to abuse. Those need their own defenses.