Your Few-Shot Examples All Look the Same: Diversity in Few-Shot Prompting

July 24, 2026 · Claude Certified Architect (CCA) Prep (part 37)

▶ Watch on YouTube & subscribe to The Stack Underflow

You give Claude two examples of the extraction task you want. Both examples happen to be short, single-line invoices with a clean total at the bottom. On your test set, it works beautifully — until a real invoice shows up with a multi-line total, a discount line, or a foreign currency symbol in an unexpected place, and the model produces a confidently wrong answer shaped exactly like your two examples. Nothing is “broken.” The model did exactly what you showed it. The problem is what you showed it.

The one-sentence version: Few-shot examples don’t just demonstrate a format — they define the output distribution the model samples from, so two examples that all look the same teach the model to expect only that shape, and it collapses on anything else.

The question

Your 2 few-shot examples are all the same format, and the model overfits to it on edge cases. What’s the fix?

• A) Remove the examples
• B) Add diverse examples covering the range and edge cases
• C) Repeat the same example
• D) Increase temperature

Pause here and pick an answer before you scroll — this is Domain 4 (Prompt Engineering & Structured Output) of the Claude Certified Architect practice set, question 4 of 10.

The answer

B — add diverse examples covering the range and edge cases.

The mechanism: few-shot examples are not decoration on top of an instruction. They are the strongest signal in the prompt for what shape the output should take. Claude generalizes from the pattern across your examples, and if every example shares the same structure, the “pattern” the model extracts is narrower than the task actually is. It has effectively memorized a template, not learned a task.

TWO EXAMPLES, ONE SHAPE                    DIVERSE EXAMPLES, FULL RANGE

  ex.1: single-line invoice                  ex.1: single-line invoice
  ex.2: single-line invoice                  ex.2: multi-line w/ discount
                                              ex.3: foreign currency symbol
        ●●  ← tight cluster                  ex.4: missing/ambiguous total
        (model infers: "this IS the task")
                                                    ●   ●
  new input: multi-line invoice                   ●  ●    ●
       ↓                                       ●    ●   ●
  forced onto the learned template            ← spread covers the real
  → wrong field mapping, silent error            distribution of inputs

                                              new input: multi-line invoice
                                                   ↓
                                              falls inside the range the
                                              examples already covered

Each example is a sample the model uses to estimate the boundaries of the task. Two examples of the same shape place all of your samples in one tight cluster, so the model’s implicit boundary around “valid output” is drawn tightly around that cluster — anything outside it gets forced back in, badly. Examples that deliberately span the range (different line counts, a discount, a currency variant, an edge case with a missing field) push that boundary out to where the real inputs actually live. This is the same intuition as a training set for a classifier: a dataset with no diversity produces a model that only works on inputs resembling the training data. Few-shot prompting is in-context learning on a tiny, ad-hoc dataset — the same rule applies, just at prompt-time instead of training-time.

Why the other options fail

• A) Remove the examples. This treats the symptom as “examples are the problem” rather than “these specific examples are too narrow.” Removing them throws away the formatting and structural signal few-shot examples exist to provide — you’re back to zero-shot, where the model has to guess the exact output shape from instructions alone, which is usually less reliable on structured-extraction tasks, not more.
• C) Repeat the same example. This makes the overfitting strictly worse. Duplicating a homogeneous example doesn’t add information; it just increases the model’s confidence that the one shape it has seen is the only shape, sharpening the same narrow boundary rather than widening it.
• D) Increase temperature. Temperature changes how the model samples across its own probability distribution at each token — it adds randomness, not correctness. The distribution the model is sampling from was already narrowed by the homogeneous examples; turning up randomness on a narrow, wrong distribution just produces more varied wrong answers. It doesn’t touch the actual cause.

The concept behind it

Few-shot prompting works because a language model treats the examples in context as evidence about the task, not just as formatting hints. This has a direct, practical consequence for anyone designing prompts for production extraction, classification, or generation tasks: example selection is a data-curation problem, not an afterthought.

A few concrete rules follow from the mechanism:

This generalizes beyond few-shot examples specifically. It’s the same underlying idea as why a JSON schema needs to explicitly allow null for fields that might legitimately be absent (see preventing hallucinated fields), and why forcing structured output with tool_choice beats hoping a prompt produces valid JSON (see guaranteeing valid JSON output) — in every case, you’re constraining or shaping the model’s output distribution deliberately, rather than leaving it to infer the boundaries from whatever happens to be in front of it. Few-shot examples are one more lever for shaping that distribution, and like any lever, an unrepresentative sample pulls it the wrong way.

FAQ

How many few-shot examples do I need? There’s no fixed number — it depends on how varied the task’s real inputs are. What matters more than count is coverage: two examples that span the meaningful variation in your inputs (format, edge cases, ambiguity) will outperform five examples that are all minor rewordings of the same case. Start small, diversify first, then add volume only if specific failure modes persist.

Should few-shot examples include tricky edge cases, or just typical inputs? Include at least one deliberately tricky or boundary case, not only the typical case. If your production inputs will sometimes have a missing field, an unusual format, or an ambiguous value, show the model at least one example of handling that — otherwise the first time it meets that input in production is also the first time it’s ever “seen” it.

Does adding more few-shot examples always improve results? No — past a point, more examples of the same shape add nothing (and can even increase latency and cost for no accuracy gain), while a small number of well-chosen diverse examples can meaningfully improve edge-case handling. Diversity of coverage matters more than raw count.

When should I use few-shot prompting instead of relying on instructions alone? Reach for few-shot examples whenever the output has a precise structural or formatting requirement that’s easier to show than to describe — exact field names, a specific tone, a particular way of handling ambiguity. For tasks where the instruction alone is unambiguous and the format is simple, zero-shot with a clear instruction (and a forced schema, if the output needs to be machine-parseable) is often sufficient and cheaper.

Not affiliated with, authorized, or endorsed by Anthropic. “Claude” and “Claude Certified Architect” are trademarks of Anthropic. These are original practice questions for study, not real exam content — verify current exam details on Anthropic’s official pages.

Where this fits in the CCA series

This is question 4 of 10 in the Domain 4 (Prompt Engineering & Structured Output, 20% of the exam) practice set. If you haven’t seen the full five-domain breakdown yet, start with the Claude Certified Architect (CCA) Exam Guide. For more on shaping model output deliberately rather than hoping for the best, see How Do You Guarantee Valid JSON? Forcing Structured Output, A Number Comes Back as a String: Validating Structured Output Types, and It Ignores the Rule Buried in the Prompt: Where to Put Instructions — all Domain 4 questions about turning “the model usually does this” into “the model reliably does this.” Browse all tutorials for the rest of the series.