How to Evaluate an LLM Feature Before It Touches a Customer

Here is the release process for most LLM features I get called in to review. Someone writes a prompt. They try eight or ten examples by hand. The outputs look good. It ships. Two weeks later a customer screenshots something embarrassing, someone tweaks the prompt to fix that specific case, and nobody checks whether the tweak broke the other 200 cases nobody was tracking.

That is not a testing gap, it is the absence of a test suite. You would not ship a payments module on the strength of "I tried a few transactions and they looked right". Evaluation is the unit test layer for probabilistic software, and it is the difference between an LLM feature you can maintain and one you can only apologise for.

Step one: build the eval set from reality, not imagination

The most common mistake is inventing test cases. Invented cases cluster around the happy path because that is what your brain generates. Real failures live in the messy tail: the customer who writes in Sheng mixed with English, the PDF that is a photograph of a printed table, the question that assumes a feature you do not have.

Sources that work, in order of value:

• Existing support tickets and chat logs. These are your distribution.
• Production traces from a pilot, sampled across the whole day rather than office hours.
• Cases your domain expert flags as "this is where people get it wrong".
• Adversarial cases you write on purpose: prompt injection payloads, out-of-scope questions, requests for things the system must refuse.

Aim for at least 50 cases to start and 150 to 300 once the feature is live. Below about 50 you cannot distinguish a real regression from noise; a single failure moves the number by two percentage points. Stratify by category and store the category on each case, because "we went from 88 to 84 percent" is useless while "refund questions dropped from 94 to 61 percent" is actionable.

Keep it in the repo, in version control, as JSONL. It is code.

{"id": "ret-014", "category": "refund_policy", "difficulty": "hard",
 "input": "I paid via paybill on Friday but the parcel never came, can I get my money back",
 "must_contain": ["7 days"],
 "must_not_contain": ["I am not able to help"],
 "reference": "Refunds are available within 7 days of delivery failure. The customer must supply the M-Pesa reference.",
 "expects_tool": "lookup_order"}
{"id": "oos-003", "category": "out_of_scope", "difficulty": "easy",
 "input": "What is the exchange rate today?",
 "must_not_contain": ["KES", "rate is"],
 "expects_refusal": true}

Step two: separate the checks that are cheap and certain from the ones that are not

Run three tiers, and be deliberate about which questions belong in which.

Tier 1, deterministic assertions. No model involved. Did the output validate against the schema? Did it call the expected tool and not call a forbidden one? Does it contain the required citation? Is latency under budget? Is cost under budget? These run on 100 percent of cases, cost nothing, and catch a surprising share of real bugs.

Tier 2, LLM-as-judge. For the genuinely subjective properties: is the answer faithful to the retrieved context, does it actually address the question, is the tone right. Judges are reported to agree with human raters roughly 85 percent of the time on well-specified tasks, which is often higher than two humans agree with each other - but that number is a property of a good rubric, not of the technique.

Tier 3, human review. A sample, weekly, by someone who knows the domain. This exists to keep the judge honest.

Writing a judge that is worth trusting

Three rules, learned the hard way.

Binary or short ordinal criteria, never a 1 to 10 score. Ask a model to rate quality out of ten and you get 7 and 8 forever, with no discriminating power. Ask "is every factual claim in the answer supported by the provided context: yes or no" and you get a signal you can threshold.

Give the judge the reference and the context. A judge scoring faithfulness without the source document is guessing. A judge with the source is doing an actual comparison.

Force it to cite before it scores. Put the evidence field first in the schema so the model commits to a quotation before committing to a verdict.

import anthropic, json

client = anthropic.Anthropic()

JUDGE_SCHEMA = {
    "type": "object",
    "properties": {
        "unsupported_claims": {
            "type": "array",
            "items": {"type": "string"},
            "description": "Verbatim claims from the answer with no support in the context.",
        },
        "grounded":      {"type": "boolean"},
        "answers_question": {"type": "boolean"},
        "verdict": {"type": "string", "enum": ["pass", "fail", "borderline"]},
    },
    "required": ["unsupported_claims", "grounded", "answers_question", "verdict"],
    "additionalProperties": False,
}

JUDGE_SYSTEM = (
    "You grade a support answer. First list every factual claim in the answer "
    "that is NOT supported by the context, quoting it verbatim. Then decide. "
    "'grounded' is true only if that list is empty. Numbers, dates, policy "
    "durations and prices must match the context exactly. Style is irrelevant."
)

def judge(question: str, context: str, answer: str) -> dict:
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1500,
        system=JUDGE_SYSTEM,
        output_config={"format": {"type": "json_schema", "schema": JUDGE_SCHEMA}},
        messages=[{"role": "user", "content":
            f"<question>{question}</question>\n"
            f"<context>{context}</context>\n"
            f"<answer>{answer}</answer>"}],
    )
    return json.loads(resp.content[0].text)

Calibrate the judge before you trust it

This is the step almost everyone skips, and it is the one that makes the whole exercise honest. Label 50 outputs by hand. Run the judge on the same 50. Compare.

def agreement(human_labels, judge_labels):
    n = len(human_labels)
    agree = sum(h == j for h, j in zip(human_labels, judge_labels))
    po = agree / n
    # chance agreement, for Cohen's kappa
    p_h = sum(human_labels) / n
    p_j = sum(judge_labels) / n
    pe = p_h * p_j + (1 - p_h) * (1 - p_j)
    kappa = (po - pe) / (1 - pe) if pe < 1 else 1.0
    fp = sum(1 for h, j in zip(human_labels, judge_labels) if j and not h)
    fn = sum(1 for h, j in zip(human_labels, judge_labels) if h and not j)
    return {"raw_agreement": round(po, 3), "kappa": round(kappa, 3),
            "judge_too_lenient": fp, "judge_too_harsh": fn}

Raw agreement above 0.9 with a kappa above 0.6 means the judge is usable. Below that, the problem is your rubric, not the model: it is ambiguous, or it is asking one question that is really three. Split it and re-measure. Also look at the direction of the errors - a judge that is systematically lenient will happily wave through the regression you built it to catch.

Wire it into CI

Once you have a frozen dataset and a calibrated judge, the gate is ordinary software engineering. Per-category thresholds, not a single global number:

import json, pytest, statistics

CASES = [json.loads(l) for l in open("evals/support_v3.jsonl")]

THRESHOLDS = {          # minimum pass rate per category
    "refund_policy":  0.90,
    "order_status":   0.95,
    "out_of_scope":   1.00,   # must never answer these
    "prompt_injection": 1.00, # must never call a forbidden tool
}

@pytest.fixture(scope="session")
def results():
    out = []
    for case in CASES:
        run = run_feature(case["input"])
        checks = {
            "schema_ok":  run.schema_valid,
            "no_forbidden_tool": not (set(run.tools_called) & FORBIDDEN),
            "contains":   all(s in run.text for s in case.get("must_contain", [])),
            "excludes":   all(s not in run.text for s in case.get("must_not_contain", [])),
            "latency_ok": run.latency_ms < 6000,
        }
        if case["category"] not in ("out_of_scope", "prompt_injection"):
            checks["grounded"] = judge(case["input"], run.context, run.text)["grounded"]
        out.append({"case": case, "run": run, "passed": all(checks.values()),
                    "checks": checks})
    return out

@pytest.mark.parametrize("category,floor", THRESHOLDS.items())
def test_category_pass_rate(results, category, floor):
    subset = [r for r in results if r["case"]["category"] == category]
    rate = sum(r["passed"] for r in subset) / len(subset)
    failing = [r["case"]["id"] for r in subset if not r["passed"]]
    assert rate >= floor, f"{category}: {rate:.0%} < {floor:.0%}, failing {failing}"

def test_cost_budget(results):
    p95 = statistics.quantiles([r["run"].cost_usd for r in results], n=20)[18]
    assert p95 < 0.02, f"p95 cost per request {p95:.4f} exceeds budget"

Run this on every pull request that touches a prompt, a schema, a retrieval parameter or a model string. Put the pass rates in the PR description. The rule that makes it stick: a prompt change with no eval run does not merge. Prompts are code with a much higher blast radius and no compiler.

Evaluate the parts, not just the whole

For a RAG feature specifically, a single end-to-end score tells you something is wrong but not what. Measure the two halves separately:

• Retrieval: recall at k (did the passage containing the answer appear in the top k at all) and mean reciprocal rank. If recall at 10 is 0.6, no amount of prompt engineering will save you - the generator never sees the answer.
• Generation: groundedness given the retrieved context, plus answer relevance.

To measure retrieval you need a small labelled set mapping questions to the document IDs that contain the answer. It is tedious to build once and it pays for itself the first time you change chunk size and want to know whether you helped.

After launch: keep measuring

Pre-production evals become production guardrails. Run the cheap deterministic checks on 100 percent of live traffic - schema validity, forbidden tool calls, latency, refusal rate, cost per request. Sample 5 to 10 percent for judge scoring, because judging every request doubles your bill for no extra information.

Then watch for drift on three axes: input drift (people start asking about something new), model drift (a provider updates a model, or you upgrade), and prompt drift (twelve small edits nobody evaluated individually). Log the prompt version, model ID and retrieval parameters on every trace, so that when the number moves you can bisect instead of speculate.

What good looks like

You should be able to answer these five questions in under a minute:

• What is the current pass rate, by category?
• What changed since the last release, and which cases flipped?
• Which failures are retrieval and which are generation?
• What does this feature cost per thousand requests?
• When did a human last check the judge?

If you cannot, you are not shipping an LLM feature. You are running an unmonitored experiment on your customers, and eventually one of them will publish the results.