Handling LLM API Rate Limits in AI Workflows

LLM API rate limits become a workflow problem when several jobs share one provider quota. Each job looks harmless alone, their combined requests and tokens trigger 429s, and if every worker retries on its own, the retries add pressure. A higher retry count does not fix this. Model calls need one admission layer that accounts for requests, tokens and concurrency before dispatch, with retries reserved for recovery.

Why per-request retries fail

Providers enforce more than a request counter. OpenAI's rate limits guide covers request and token dimensions with remaining-limit and reset headers, and Anthropic's rate limits page covers requests, input tokens and output tokens per minute. A burst of long documents drains input tokens while the request count looks modest, and many short tasks do the reverse. The OpenAI Cookbook example warns that unsuccessful requests still count toward the limits and recommends exponential backoff with random jitter. Backoff is necessary and is not capacity control: fifty workers that back off together come back together.

Three decisions should stay separate. Should this job enter the model-call queue now? When is there enough request, token and concurrency capacity to dispatch it? If the call fails, should it retry, wait for a reset or stop?

Mirror the provider's accounting exactly

A local budget is only useful if it counts what the provider counts. Anthropic's page has details that a generic scheduler gets wrong:

• Limits can bite at shorter intervals than the stated one. The page says 60 requests per minute might be enforced as 1 request per second, so short bursts can trigger errors. Smooth dispatch instead of releasing a whole minute's budget at the top of the minute.
• The API uses a token bucket, so capacity replenishes continuously rather than resetting at fixed intervals. A timer that wakes up at the next minute boundary leaves capacity unused.
• For most Claude models only uncached input tokens count toward input tokens per minute. Cache reads do not, and the page's own example is a 2,000,000 ITPM limit with an 80% cache hit rate handling about 10,000,000 total input tokens per minute. Counting full prompt length in your local budget throttles you well below the real ceiling.
• Output limits count tokens as they are produced, and max_tokens does not factor in. Reserving max_tokens per job wastes output capacity on this provider. Check how your own provider estimates before copying either rule.

Keep one capacity record per provider, model and credential scope, since two keys do not necessarily have independent quotas. Treat response headers as observations: they arrive after the dispatch decision and cannot stop the first burst. Run a conservative local budget and reconcile it with the reported remaining capacity and reset times.

Add a separate in-flight limit. Sockets, memory, downstream tools and worker slots can run out before a token bucket empties, and Temporal's worker performance guidance exposes explicit poller and concurrent-activity controls for the same reason.

Build a shared admission path

Route every model call through one logical service: a library over shared storage, a gateway or a queue consumer. Workers that share a quota must read and update the same counters. Each pending call carries what scheduling and recovery need:

ModelJob {
  job_id
  workflow_id
  provider
  model
  credential_scope
  priority_class
  enqueued_at
  input_tokens_estimated
  output_tokens_reserved
  attempt
  deadline_at
  idempotency_key
}

The scope names the quota boundary, the class separates interactive, scheduled and bulk work, and the deadline stops a stale request being dispatched after its result stopped mattering. Reject impossible jobs early. A document larger than the model's limit will not fit by waiting, so route it to chunking or review. If the deadline is shorter than the current queue delay, fail with a specific capacity error.

Schedule for tokens, requests and fairness

function dispatch_next(now):
  reconcile_provider_resets(now)
  job = choose_eligible_job(now)
  if job is none:
    return sleep_until_next_event()
  needed = {
    requests: 1,
    input_tokens: job.input_tokens_estimated,
    output_tokens: job.output_tokens_reserved,
    concurrency: 1
  }
  if not budgets.can_reserve(job.quota_key, needed):
    return sleep_until_budget_refill(job.quota_key, needed)
  budgets.reserve_atomically(job.quota_key, needed)
  mark_dispatched(job)
  call_provider(job)

Reserve every budget atomically, so a request slot is never held without its tokens. Plain FIFO puts a customer-facing request behind a large import, and strict priority starves batch work, so use weighted fair scheduling or reserve a share per class. Static per-worker fractions of the quota waste capacity when a worker idles and overload the provider when autoscaling adds workers.

Send bulk work somewhere else

The cheapest way to protect interactive traffic is to take bulk work off its quota. Anthropic's Message Batches API has its own rate limits, shared across models and separate from the Messages API limits. They cover requests per minute and the number of batch requests in the processing queue (200,000 on the Start tier). A nightly classification job that does not need an answer in seconds belongs there, which also removes it from your scheduler's problem set.

A 429 is not always a rate limit

Classify before retrying. Anthropic documents that reaching an organization's monthly spend cap returns HTTP 429 with error_code: enforced_spend_limit_reached and no retry-after header, and that retrying, including automatic SDK retries, fails until access resumes at 00:00 UTC on the first of the next month. A scheduler that treats every 429 as transient will spin on that error all month. Branch on the error code, page a human for a spend cap, and mark the quota scope closed until the stated time. When a retry-after header is present, the docs say earlier retries will fail, so wait at least that long.

Feed every 429 back into the shared budget, not only the worker that saw it: record provider, model, scope, estimated tokens and attempt, update the shared next-eligible time, and avoid logging prompts. Bound retries by count and by deadline, never retry a malformed request, bad credential or oversized context, and let exactly one layer own 429 retries. If the HTTP client, the model adapter and the workflow engine each make three attempts, one logical call becomes up to twenty-seven requests.

Client libraries already carry retry policy, and it is worth reading before you add yours. In the current OpenAI Python SDK source, a call is retried twice by default, backoff is capped at 8 seconds, and a Retry-After value is honored only up to two minutes. Beyond that the SDK stops retrying and raises, so a long server-directed wait reaches your code as an error rather than a sleep. The SDK also obeys a nonstandard x-should-retry response header in both directions, which means a gateway you control can force or forbid client retries for a given response. Check what your SDK version does before you decide which layer owns retries, and set max_retries to 0 on the client when your own scheduler is the owner.

Observe the scheduler, not only provider errors

Fewer 429s prove nothing if the scheduler achieves that by holding work forever. Track admitted, queued, dispatched, rejected, expired and cancelled jobs, queue age by class, estimated against actual tokens, in-flight calls, 429s by attempt and scope, retry delay, deadline exhaustion and starvation per class. Alert on queue age and expired deadlines before 429 totals. Job IDs, estimates and status codes are enough to debug most capacity incidents, so leave prompts and credentials out of telemetry.

Test under controlled overload

Build a fake provider with separate request and token budgets, reset headers, injected 429s and variable latency. Cover a request burst with spare tokens, a few huge prompts with spare requests, interactive jobs arriving during a long bulk queue, the provider shrinking its advertised budget mid-test, several workers starting at once, a 429 with no retry time, a spend-cap 429, a queued job hitting its deadline, and a worker crashing after reserving capacity. Assert that reservations never exceed any budget, interactive queue age stays within target, bulk work still progresses, expired jobs are not dispatched, and retries stay in one layer. Wrap reservations in a lease so a dead worker returns its capacity. If it dies after sending, the outcome is unknown, so record that and let the workflow's recovery policy decide.

Start by listing every service that shares the quota. Put one model and credential scope behind the shared admission path, add request and token reservations that match the provider's own accounting, and run the cases above before extending to the next model.

References

• OpenAI rate limits guide for request and token limit dimensions and response headers.
• Anthropic rate limits for token-bucket replenishment, short-interval enforcement, cache-aware ITPM, OTPM and max_tokens, the Batches API limits, and the spend-cap 429.
• OpenAI Cookbook rate-limit handling for exponential backoff with jitter and the warning about unsuccessful requests.
• Temporal worker performance for poller and concurrent-activity controls.
• openai-python _base_client.py for the default retry count, the two-minute Retry-After ceiling and the x-should-retry header.

---