Evaluating LLM Applications

Reading through ten sample outputs and deciding "looks good" is not evaluation — it's a vibe check, and it stops scaling the moment you change a prompt, swap a model, or ship a fix and need to know whether anything got worse. An evaluation set replaces that with a fixed, representative collection of inputs — ideally with a reference answer or some other ground truth attached — that gets run and scored the same way every time, so a change to your prompt or model produces a number you can compare against the last number, not just a feeling.

The value isn't in running it once. It's in running the same eval set again every time something changes — a new model version, a reworded system prompt, a different retrieval pipeline — so a regression shows up as a dropped score before a person notices it in production.

OpenAI's evaluation tooling documents five grader types, and picking the right one for what you're actually checking matters more than picking the most sophisticated one. A string check does exact-match comparison against a reference — right for a fixed, correct answer with no acceptable variation. Text similarity uses embeddings to measure how semantically close an output is to a reference, useful when the wording can vary but the meaning shouldn't. A score model grader uses an LLM to assign a numeric score against criteria you describe — friendliness, helpfulness, adherence to a style guide — properties too subjective for exact matching. A label model grader uses an LLM to pick a category from a fixed list — concise versus verbose, on-topic versus off-topic. And Python code execution runs your own logic against the output — checking a word count, a required substring, whether it parses as valid JSON.

Reaching for a model-graded check when a code check would do wastes a model call on something deterministic; reaching for exact-match on something inherently variable in wording produces false failures on a genuinely correct answer.

def grade_response(question, response, rubric):
    judge_prompt = f"""Rate the response from 1-5 against this rubric: {rubric}

Question: {question}
Response: {response}

Respond with only the number."""
    result = call_model([{"role": "user", "content": judge_prompt}])
    return int(result.strip())

Automated graders are fast enough to run on every change, but they're not the only source of ground truth. Human annotation — ideally from someone with actual subject-matter expertise in what's being evaluated — catches what automated graders miss: subtle tone problems, infrequent edge cases, whether a response actually satisfies what a real user needed rather than just matching a pattern. A good annotation does double duty: it's a judgment call in its own right, and it's also the material you use to write or refine the automated graders that will run on the next thousand examples.

The practical workflow treats the eval set as something that grows: every time a real failure surfaces in production or testing, it becomes a new case in the set, so the same class of mistake gets caught automatically the next time, rather than being rediscovered by a person.