10,000 Docs Overnight, Cost Matters: Batch Processing LLM Jobs

July 24, 2026 · Claude Certified Architect (CCA) Prep (part 40)

▶ Watch on YouTube & subscribe to The Stack Underflow

Someone on your team just asked for structured fields — vendor, amount, date, category — pulled out of 10,000 scanned invoices, and they need it “by tomorrow morning.” Nobody said “as fast as possible.” Nobody said “streamed to a dashboard.” They said overnight, and they said keep the bill down. That single sentence changes which API call is correct, and it’s exactly the kind of scenario the CCA exam likes to test, because the wrong answer isn’t broken — it’s just needlessly expensive.

The one-sentence version: When you have high volume and no latency deadline, the Message Batches API is the answer — same model, same schema, roughly half the cost, because you’re trading real-time delivery for a processing window the provider can schedule efficiently.

The question

You must extract fields from 10,000 documents overnight — cost matters, latency doesn’t. Best API choice?

• A) One synchronous call per doc
• B) The Message Batches API (async, ~50% cheaper)
• C) A bigger model
• D) Streaming

Pause here and pick an answer before you scroll — the mechanism behind it matters more than the letter.

The answer

B — the Message Batches API. It’s async: you submit the whole set of 10,000 extraction requests as one batch job, Anthropic processes them within a defined window (typically same-day, well inside “overnight”), and you retrieve the results once the job completes. In exchange for giving up per-request, real-time latency, you get roughly a 50% discount off the standard per-token price for that model — same weights, same prompt, same JSON schema forcing the shape, just delivered on a schedule instead of the instant you ask.

The mechanism is a straight trade: latency for cost, and only when latency was never a requirement in the first place.

SYNCHRONOUS PATH (option A)                 BATCH PATH (option B)
────────────────────────────                ─────────────────────
10,000 live API calls                        1 batch submission
  │                                            │  containing 10,000
  ├─ call 1  ──▶ response (now)                │  individual requests
  ├─ call 2  ──▶ response (now)                ▼
  ├─ call 3  ──▶ response (now)          ┌───────────────────┐
  │   ...                                │  QUEUED / PROCESSING│
  ├─ call N  ──▶ response (now)          │  (async, off the    │
  │                                       │   live request path)│
  rate limits bite at scale              └─────────┬──────────┘
  full per-token price × 10,000                    │ within the
  each call blocks on network + queue              │ processing window
                                                     ▼
                                            ┌───────────────────┐
                                            │  RESULTS FILE      │
                                            │  10,000 responses, │
                                            │  ~50% cheaper      │
                                            │  per token         │
                                            └───────────────────┘

Rule: high volume + not latency-bound → the Batches API.

The reason the discount exists at all: a synchronous call has to be served the moment it lands, competing for capacity with every other live request on the system. A batch job has no such constraint — the provider can pack it into spare capacity and schedule it, which is precisely the kind of workload a cloud provider prices cheaper. You’re not getting a worse model or a degraded schema. You’re getting the same extraction, later, for less.

Why the other options fail

• A) One synchronous call per doc. This works — the extraction will succeed 10,000 times over — but it’s the expensive path for a job that explicitly has no latency requirement. Ten thousand live calls also means you’re now managing concurrency and rate limits for no reason: you’d be paying full price and building queueing logic to solve a problem the batch endpoint already solves for you, for free, at a discount.
• C) A bigger model. Model size is an accuracy lever, not a cost or throughput lever. The question never says the smaller model is getting fields wrong — it says cost matters. Reaching for a bigger model when the actual constraint is “10,000 calls is expensive” doesn’t touch the constraint; it makes the per-call cost worse while doing nothing about the volume problem.
• D) Streaming. Streaming exists to reduce perceived latency for a human watching tokens arrive — think a chat UI where users don’t want to stare at a blank screen. Nothing in this scenario has a human watching a screen in real time. Streaming doesn’t reduce cost, doesn’t reduce total tokens processed, and solves a UX problem this job doesn’t have.

The concept behind it

This question is really about a single design habit: before you pick an API shape, name the actual constraint. Every LLM job sits somewhere on two independent axes — how much volume, and how much latency tolerance — and the right integration choice falls out of where you land.

The trap in the exam question — and in real system design — is treating “cost matters” as an accuracy problem (reach for a bigger or smaller model) or a speed problem (reach for streaming), when it’s neither. Cost, in a high-volume job with no latency deadline, is a scheduling problem, and the Batches API is the scheduling-aware endpoint. The tell is almost always in the phrasing: “overnight,” “by end of day,” “process the backlog,” “no rush” — any language that says a human isn’t waiting synchronously on the result is a green light to batch.

This generalizes past document extraction. Nightly ETL jobs pulling structured data out of support tickets, bulk classification of a backlog of images or transcripts, re-running an evaluation suite across thousands of test cases, backfilling embeddings metadata — all of it is the same shape: volume is high, a person isn’t refreshing a page waiting on any single item, and the win is entirely in unit economics. The pattern to internalize is: synchronous is the default only until volume and latency tolerance say otherwise — then batch is strictly better, not a compromise.

One caveat worth carrying into production: batch jobs still need the same schema-forcing and validation discipline as any other structured-output call (see How Do You Guarantee Valid JSON?) — a malformed field in request 4,812 out of 10,000 doesn’t get a human’s attention in real time, so your validation and retry logic has to run after the batch completes, not during.

FAQ

Does the Message Batches API return results in real time as each item finishes? No — that’s the entire trade you’re making. You submit the full set of requests as one job and poll for completion (or check back after the processing window), then retrieve a results file covering everything at once. If you need per-item results the moment they’re ready, you’re describing a different constraint than “cost matters, latency doesn’t,” and synchronous calls (or a queue you build yourself) are the right tool.

Is the ~50% batch discount specific to one model, or does it apply broadly? Treat it as a general pricing pattern across the model lineup rather than a one-off promotion tied to a single model — but exact discount percentages and any changes to them are the kind of detail that shifts over time. Verify the current rate on Anthropic’s official pricing page before you build a cost projection around it.

What happens if one of the 10,000 requests in a batch fails or times out? A well-designed batch workflow treats the results file as partial by default: check for per-item errors (a doc that failed to parse, a schema mismatch) and re-submit just the failures — as a small follow-up batch or as individual synchronous calls if the failure count is small. This is the same bounded-retry discipline covered in Your Retry Loop Never Stops, just applied at the batch-item level instead of the single-call level.

Can I combine batching with prompt caching to cut cost even further? Yes, when the requests in the batch share a large common prefix — the same long extraction instructions or the same schema definition repeated across all 10,000 documents — caching that shared prefix stacks with the batch discount, since they address different parts of the cost (repeated input tokens vs. real-time-serving overhead). They’re independent levers, not alternatives.

Where this fits in the CCA series

This is Domain 4 — Prompt Engineering & Structured Output (20% of the exam) — where the skill isn’t clever wording, it’s building contracts: forced schemas, validated output, and now, the right delivery mechanism for the volume you’re actually processing. It sits alongside How Do You Guarantee Valid JSON? (forcing the shape), A Number Comes Back as a String (validating the shape), and Your Retry Loop Never Stops (recovering when the shape is wrong) — batch processing is the fourth leg: getting that same disciplined extraction to run economically at scale.

Not affiliated with, authorized, or endorsed by Anthropic. “Claude” and “Claude Certified Architect” are trademarks of Anthropic. These are original practice questions for study, not real exam content — verify current exam details on Anthropic’s official pages.

Start with the Claude Certified Architect exam guide for all five domains on one diagram, or browse all tutorials to keep going through the series.