How to Write LLM Evals: Testing AI Apps with Real Data

June 23, 2026 · updated June 25, 2026 · How Claude Actually Works (part 23)

▶ Watch on YouTube & subscribe to The Stack Underflow

Running a few prompts and deciding “yeah, looks good” is not a test strategy — it is a vibe check. And vibes regress silently. You edit one line in your system prompt, fifty cases that were passing now have one quietly flipping to red, and nobody notices until a user files a bug report. This article is about converting “good” from a feeling into a number that a CI pipeline can block on.

The shape of a real eval is simple: a dataset, one or more graders, a score, and a gate. Four pieces. That is the whole stack.

The one-sentence version: Replace manual spot-checking with a dataset-plus-grader pipeline that outputs a pass-rate your CI system can block a merge on — because if you can’t score it, you can’t ship it.

Why vibe checks fail at scale

When you have three prompts and a pair of eyes, a qualitative review works fine. The moment you have a non-trivial prompt, a model that gets updated, and a team making edits, that approach collapses under two distinct failure modes.

Silent regression is the dangerous one. A change that fixes case 12 can silently break case 47. With a vibe check, you looked at cases 1, 5, and 12; case 47 never got a second glance. The regression ships and a user finds it.

No baseline is the second failure mode. Without a recorded number, you cannot tell whether this week’s build is better or worse than last week’s. “It felt about the same” is not a release criterion for anything else you build — it should not be one for LLM features either.

Vibes do not version-control. Numbers do.

Before eval harness                After eval harness
──────────────────────             ──────────────────────────────
Dev runs 3 prompts                 Harness runs all 50 cases
Dev nods, ships                    Score: 92% (46/50 passed)
Case #47 breaks silently           Score drops to 82% → gate fires
User files bug 3 days later        Merge is blocked immediately

Step 1 — Build a dataset of real, hard cases

Every eval starts with a dataset: a structured collection of inputs paired with what a good output looks like. Think of it as a spreadsheet with two essential columns.

The input column holds real prompts — pulled from production logs, edge cases users actually hit, or inputs you know have tripped up the model before. The expected column holds what “good” looks like: an exact answer, an acceptable shape, a rule, or a rubric.

Start with 20 real, hard cases. Twenty genuine cases with known tricky behavior beat a thousand synthetic happy-path examples every time. As your system ships, every production failure becomes a new row. The eval gets sharper the more it is confronted with reality.

The key discipline: curate from production traffic and failure reports, not from your imagination of what might go wrong. Your imagination is optimistic. Production is not.

Step 2 — Choose a grader for each criterion

Not every output can be judged the same way. The right strategy is to match the grader — the function that decides pass or fail — to the criterion being tested.

Grader types by output kind
────────────────────────────────────────────────────────────
Grader type      Best for
────────────────────────────────────────────────────────────
Exact match      Factual recall, deterministic single answers
Schema valid     JSON/XML structure, type checking, required fields
Rule-based       Regex patterns, numeric ranges, word-count limits
LLM-as-judge     Fluency, tone, helpfulness, open-ended quality
────────────────────────────────────────────────────────────

The principle is: use the cheapest, most reproducible grader that covers the criterion. Exact match is free and deterministic. Schema validation is nearly free. Rule checks require a bit of regex craft. LLM-as-judge — using a model (often a capable, cheaper model like claude-sonnet-4-6) to grade another model’s output — is the right tool for fuzzy criteria that no rule can fully capture, but it is the most expensive and least stable option. Reserve it for the cases where the other grader types genuinely cannot reach.

A single output can have multiple criteria and therefore multiple graders. A customer-service response might need to pass a schema check (structured fields present), a rule check (response under 200 words), and an LLM-judge check (tone is empathetic). Each criterion gets its own grader. Do not try to combine them into one omnibus prompt — mixing them makes failures harder to diagnose.

Step 3 — Run and score

Execute your application against every row in the dataset. Each row resolves to a pass or a fail. Tally them up and express the result as a pass-rate.

Dataset: 50 cases
Results: 46 passed, 4 failed
Score:   92%

Now “good” is a number. You can store it in a time-series, diff it between commits, attach it to a pull request, and set a threshold. The score is only as meaningful as the dataset it came from — but that is a dataset quality problem, not a scoring problem. Keep the two concerns separate.

One practical note: if your application is non-deterministic (as most LLM features are), a single score over a fixed dataset will show run-to-run variance. This is expected. Keep sample sizes large enough that a single case flipping does not swing the score materially, and document whether you are running evals at temperature: 0 (more reproducible, less representative of production) or at production temperature (more representative, higher variance). Both choices are valid; what matters is being consistent and noting it in the eval metadata.

Step 4 — Wire the score into CI as a regression gate

A harness you run manually once in a while is better than nothing. A regression gate wired into your CI pipeline is the version that actually catches regressions before they ship.

CI pipeline — eval gate
────────────────────────────────────────────
Trigger: pull request opened or commit pushed
                    │
                    ▼
         Run eval harness on new build
                    │
                    ▼
           Compute pass-rate score
                    │
              Score >= 90%?
             /             \
           YES               NO
            │                 │
         PASS              FAIL
      merge allowed      merge blocked
                      (score dropped to 88%)

Set the threshold as a product decision, not a technical one. A customer-facing summarization feature where errors erode trust might need a 95% gate. An internal debug tool might be fine at 80%. What matters is that the threshold is explicit, versioned alongside the code, and raised deliberately as quality improves — not left at an arbitrary number forever.

The other benefit of CI integration is trend visibility. Track the score over time. A score that was 94% six months ago and is now 89% — even though it still clears the 85% gate — is telling you something important about drift. You want to see that before it becomes a user complaint.

The LLM-as-judge caveat

An LLM judge lets you scale grading for fuzzy, open-ended outputs that rule-based checks cannot handle — tone, coherence, factual nuance, style adherence. For many real-world LLM features, an LLM judge is the only practical path to automated grading at scale.

But the judge is itself a model, which introduces a set of well-documented failure modes.

Anthropic’s own published eval guidance (docs.anthropic.com, 2025) recommends calibrating LLM-as-judge graders closely against human experts before trusting them in CI. The calibration process is simple: take 20–30 eval cases, have a team member grade each one independently, then compare the judge’s verdicts. If they disagree on more than 15–20% of cases, you are measuring with a miscalibrated instrument. Fix the judge prompt, tighten the rubric, or escalate to a stronger model before treating the judge’s output as ground truth.

The honesty principle here: an LLM judge that your team has not calibrated is not a grader — it is a second vibe check, now automated and invisible.

How to apply this in practice

Here is a concrete starting recipe for a new LLM feature:

1. Before launch: Pull 20 real cases from beta testing or manual QA. For each, record the input and what a good output looks like. Assign grader types per criterion.

2. At launch: Run the eval harness against the production build. Record the baseline score. Set the CI gate threshold 3–5 percentage points below the baseline (to catch real regressions without being noise-sensitive to minor variation).

3. After every production incident: Add the failing case to the dataset. Every bug is free data. The dataset grows sharper the more production traffic it sees.

4. On model or prompt changes: Run the full eval harness before merging. Treat a score drop of more than 3–4 percentage points as a blocking issue, even if it still clears the gate.

5. Quarterly: Audit the dataset. Are the cases still representative of current traffic? Have new product capabilities made old cases irrelevant? Prune stale rows, add fresh ones.

The tools are secondary. You can run this pattern with a simple Python script, a dedicated framework like DeepEval or Promptfoo, or Anthropic’s own evaluation guidance in the cookbook. The pattern — dataset, graders, score, gate — is what matters. The tooling is an implementation detail.

Common misconceptions

“I need hundreds of examples before evals are useful.” You do not. Twenty well-chosen, representative, hard cases catch more real regressions than five hundred synthetic happy-path examples. The forcing function is representativeness and difficulty, not volume. Start small and grow from production failures.

“LLM-as-judge is cheating.” It is not cheating — it is a pragmatic tool for criteria that are genuinely hard to express as rules. Tone, coherence, and factual nuance cannot be reliably detected with regex. The risk is miscalibration, not the approach itself. Calibrate the judge, version the judge prompt, and it becomes a legitimate grader.

“A high eval score means the model is good.” A high score means the model performs well on your dataset. If your dataset does not represent production traffic — if it is all easy cases or doesn’t cover the distributions users actually hit — the score describes the dataset, not the product. The eval is only as good as the cases in it.

“Evals are a one-time setup.” They are a living artifact. Every production incident that reveals a gap in coverage is a bug in your eval dataset as much as it is a bug in the model. Teams that treat eval datasets as static quickly find themselves with a harness that measures a past version of the problem.

Frequently asked questions

How do I decide what threshold to set for the CI gate? Run your eval on a known-good baseline — ideally the version currently in production. Record that score. Set your gate 3–5 percentage points below it. This catches real regressions without blocking on minor run-to-run variance. Raise the threshold deliberately as your dataset matures and your quality improves — never lower it without a documented reason.

What if my LLM output is non-deterministic? Can I still eval it? Yes. Non-determinism is why you need a large enough dataset that the score is statistically stable. A single case flipping does not move a 50-case score by much. If you need tight reproducibility, fix temperature to 0 for eval runs and note it in the eval metadata. The score will be slightly optimistic relative to production, which is worth knowing.

What counts as a “production failure” worth adding to the dataset? Any case where a user complained, where you found the output surprising in retrospect, or where a manual review flagged an issue. The signal is “this case exposed a gap between what the model does and what production demands” — not just “the output was wrong once.” Edge cases and adversarial inputs belong here too.

Do I need a separate dataset for every model I test? Not necessarily. You need the same dataset run against different model versions or prompt variants. The dataset is the constant; the model ID or prompt is the variable. Comparing claude-opus-4-8 vs. claude-sonnet-4-6 on the same 50-case dataset is how you make an informed cost-versus-quality trade-off rather than a vibes-based one.

Should I use a strong or weak model as the LLM judge? Match the judge’s capability to the complexity of the criterion. For straightforward rubrics (is this JSON valid? does this mention X?), almost any model works. For nuanced criteria (is this factually accurate? is this tone appropriate for medical context?), use a more capable model — something in the Sonnet 4 or Opus 4 tier. A weak judge on a hard criterion is worse than no judge, because it looks like confidence while introducing noise.

How do evals relate to the acceptance criteria I write before building a feature? Directly. The acceptance criteria tutorial covers how to write precise, testable success criteria for LLM outputs. Those criteria map one-to-one to eval dataset rows: each criterion becomes an input-expected pair, and each condition in the criterion becomes a grader. Writing acceptance criteria and building a dataset are the same intellectual move — one is pre-build, the other is post-build.

Where this fits in the series

This tutorial is part of How Claude Actually Works, a course that builds a mechanistic understanding of how Claude processes context, integrates with tools, and operates in production. Evals sit at the intersection of engineering rigor and LLM uncertainty — they are how you bring normal software-quality discipline to a system that does not give you deterministic outputs.

If you are earlier in the series and want the foundation: the Claude Stack mental model explains the layered architecture this reliability work sits inside, and how the context window works explains why context engineering decisions affect what you need to test.

If you came here from the prompt-design track: few-shot prompting and pinning model behavior with examples are the upstream techniques — evals are how you verify those techniques are actually working in production.

The next tutorial in the series covers prompt injection — what happens when user-supplied or fetched content tries to hijack your model’s behavior, and the layered defenses every agent builder needs. See Prompt Injection Attacks Explained.

Browse all tutorials to follow the full course sequence.