Cost and Latency Optimization

Two requests that produce the same quality answer can cost wildly different amounts and take wildly different times, and neither difference has anything to do with accuracy. Cost and latency are their own axes to optimize, separately from correctness, and the biggest lever for both — on both major providers — is reusing computation you've already paid for: prompt caching.

The core idea is simple. If a later request starts with the exact same tokens as an earlier one — the same system prompt, the same tool definitions, the same long reference document — the model doesn't need to reprocess that shared prefix from scratch. It reuses the saved intermediate state and picks up from where the new content begins. The prefix has to match exactly, token for token, up to whatever point you're relying on being cached; a single changed word early in a long system prompt invalidates everything after it.

OpenAI's caching is automatic and enabled by default for supported models — you don't add any parameter to benefit, though there's a minimum prefix length (documented around 1,024 tokens for current models) below which a request isn't eligible at all, and cached-portion tokens are billed at a steep discount. Anthropic's works the other way: you mark a cache_control: {"type": "ephemeral"} breakpoint explicitly on the content block you want cached, with up to four breakpoints per request. Anthropic's cache writes actually cost more than an ordinary request the first time, but cache reads afterward cost a small fraction of the normal rate — a net win only once that prefix gets reused at least once.

Both providers reward the same discipline regardless of the mechanism: put the stable part of your prompt — system instructions, tool definitions, reference documents — first, and put whatever changes on every single call — today's date, this specific user's question — last. Put the volatile piece first and it breaks the cacheable prefix for every request that follows it.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": long_reference_document,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "Summarize section 3."}],
)

When a task doesn't need an answer right now, both providers offer an asynchronous batch endpoint at roughly half the standard price, in exchange for a turnaround window — documented as up to 24 hours, often faster in practice — instead of an immediate reply. This is a strong fit for anything with no live user waiting: bulk classification, running an evaluation set, offline report generation. It's a poor fit for anything interactive, where a user is watching a screen for a reply.

The other lever is model choice itself: routing easy or common requests to a smaller, cheaper, faster model and reserving a larger model for the genuinely hard cases — the same routing pattern from the agents module, applied here to cost and latency rather than task specialization. A well-tuned router can cut average cost substantially without touching the accuracy of the hard cases that actually need the bigger model.