LLM Batch Processing: Building Reliable Production Workflows

Batch APIs sell a 50% discount, but the discount is the smallest part of the design. The real work is what happens when a job finishes with some records failed, some missing and some duplicated. This post covers a small controller that reconciles every record, plus three points that change the economics: cache hits inside a batch are best-effort, the result retention clock starts at creation, and a high failure rate can make batching more expensive than not batching.

Check the arithmetic before you batch

Anthropic's Message Batches documentation states the 50% saving and a 24-hour expiry, with most batches finishing in under an hour. That makes the break-even easy to compute. If records that fail or expire get re-run synchronously at full price, a batch costs 0.5 + f of the synchronous price, where f is the fraction re-run. At f = 50% you have saved nothing, and you have added a controller, a poll loop and a day of latency risk. Measure your failure and expiry rate on a canary of a few hundred records before moving a workload.

The discount also stacks with prompt caching, but not reliably. The same page says that because batch requests run asynchronously and concurrently, cache hits are best-effort, with typical hit rates between 30% and 98% depending on traffic. Two consequences: budget at the low end until you have measured, and use the 1-hour cache duration with identical cache_control blocks on every request, since a batch can run longer than the default 5-minute cache lifetime.

Retention counts from creation

Batch results are kept for 29 days after the batch is created, not after it ends. A controller that waits for a human to review before downloading, or that is paused during an incident, can lose the output of a job that finished correctly. Download and store results as soon as the job ends, and treat the 29-day figure as a deadline for your own reconciliation too. The same page notes that request and response data is stored for up to 29 days and that you can delete a batch after processing. If you promise customers shorter retention, delete as part of the controller.

Admission rules

A workload belongs in a batch when nobody waits on the answer, each record can finish alone, and applying a result twice does no harm. Document classification, nightly extraction, embeddings and evaluation runs qualify. A step that must react to live state, such as a tool call inside an approval flow, does not. The OpenAI Batch API uses the same 24-hour window, and the Gemini Batch API positions batch jobs for large, nonurgent work.

The controller

Separate the provider's job from your business batch. The provider owns inference. You own what was requested, which outputs were accepted, and whether every record reached a final state. Use explicit states and keep the provider's raw status in a separate field:

prepared -> submitted -> running -> results_available -> reconciling -> completed
running -> expired | cancelled | provider_failed
reconciling -> retry_required | completed_with_errors

Freeze an input manifest before submitting. Give each record a stable ID derived from the business object and operation, not a line number, and store a digest of the whole manifest next to the provider job ID. Join outputs to inputs by that ID only. OpenAI requires a unique custom_id per request, and Amazon states that Bedrock batch output order may differ from input order.

{
  "record_id": "invoice:7842:extract:v3",
  "schema_version": 3,
  "request_digest": "stored-digest",
  "payload": { "model": "configured-model", "input": "reference-to-stored-input" }
}

Fail unsupported jobs at submission

A feature that works synchronously may not exist in batch mode. Bedrock's batch inference does not support tool calling for Converse-format jobs (a toolConfig fails with a 400), and you must use the model's native request format instead. Keep a small capability matrix per provider, model and endpoint, with the date it was last tested, and reject a job that needs something the batch path lacks. Do not strip a required schema to get a submission through, because the output then passes the API and fails your contract. Run a canary batch whenever the provider, model, SDK or request schema changes.

Reconcile, then retry records

A job ending as completed says nothing about each record. Compare manifest IDs with result IDs from both the output and error files, reject unknown IDs, reject duplicates unless byte-identical, and classify each record as succeeded, retryable, permanent failure, cancelled or expired. A missing ID is never a success. The invariant to check is that the five sets are disjoint and their union equals the manifest. If not, the batch stays in reconciling.

Build the retry batch from retryable records only, with a new attempt identity and the original ID kept as lineage. Retry transport errors and expiries. Do not retry invalid JSON, unsupported features or policy refusals without a change to the request. Anthropic notes that a cancelled batch can still contain results produced before cancellation, so reconcile those first.

Stream large result files to storage with a digest and mark them usable only after validation, since downloads can fail late: one OpenAI Python issue reports a large result download failing after substantial transfer. Apply results through an idempotent consumer keyed by operation, record ID, request digest and result version, writing the ledger row and the business change in one transaction. A failed local write is retried from the stored result and never costs another model call.

Tests that matter

• Outputs in a different order, one duplicate, one missing and one unknown ID.
• A job that completes with individual failures, and one that expires with partial output.
• A download that stops midway, and a reconciler that crashes after persisting a result.
• A business write that succeeds while its acknowledgement is lost, then replayed.

The acceptance check is stored evidence that every manifest ID has one terminal state, every accepted output passes your schema, and replaying reconciliation and delivery produces no further writes.

References

• Anthropic Message Batches
• OpenAI Batch API
• Gemini Batch API
• Amazon Bedrock batch inference and input and output
• OpenAI Python issue 2959

---