Reliable LLM API calls require more than a successful request path. This checklist shows how to design retries, rate-limit handling, timeouts, idempotency, circuit breakers, and fallbacks so production AI workflows remain predictable when models, networks, or providers fail.
Overview
LLM applications operate across several failure boundaries: your application, the network, the provider gateway, the model service, and sometimes a separate tool or retrieval system. A request can fail because it was rejected immediately, exceeded a quota, took too long, returned an incomplete stream, or succeeded at the provider while the client lost the response. Each case needs a different response.
The goal is not to retry every error. Uncontrolled retries can increase load, duplicate side effects, exhaust budgets, and make an outage worse. A reliable design classifies failures, applies bounded recovery, records enough context to diagnose the incident, and gives users a clear outcome when recovery is not possible.
Before implementing resilience, define the request contract:
- Work type: Is the call a synchronous user interaction, a background job, a batch task, or a tool-using workflow?
- Latency budget: How long can the complete operation take, including retries and fallback attempts?
- Failure tolerance: Is an approximate answer acceptable, or must the application fail closed?
- Side effects: Does the model call trigger an email, database write, purchase, or other non-repeatable action?
- Output contract: What makes a response usable: valid JSON, required fields, citations, a complete stream, or a safety decision?
Keep this article alongside your broader production LLM API integration checklist. Reliability settings should support, rather than replace, authentication controls, observability, prompt testing, and cost governance.
Checklist by scenario
1. Transient network errors and service interruptions
Use retries only when the failure is plausibly temporary and repeating the operation is safe. Start with a small maximum attempt count and exponential backoff. Add jitter so many workers do not retry at the same instant.
- Classify connection resets, DNS failures, gateway errors, and temporary unavailability separately from validation or authentication failures.
- Use an overall deadline for the operation, not just a timeout on each attempt.
- Calculate backoff within the remaining deadline.
- Log the attempt number, request correlation ID, provider response class, and final disposition.
- Return a controlled error or queue the work when the deadline is exhausted.
A useful policy is: retry a small number of times for transient failures, stop immediately for permanent failures, and send exhausted work to a queue or fallback path when the workflow permits it.
2. Rate limits and concurrency spikes
LLM rate limits may apply to requests, tokens, concurrency, or a combination of dimensions. Treat a rate-limit response as a scheduling signal, not as a generic server error.
- Read provider response headers or structured error details when available, but do not assume every provider exposes the same fields.
- Honor an explicit retry delay when it is supplied, subject to your maximum deadline.
- Apply exponential backoff with jitter when no usable delay is available.
- Limit concurrency before requests reach the provider using a queue, semaphore, or token-bucket-style controller.
- Separate interactive traffic from batch traffic so background work cannot consume the entire budget.
- Track both request volume and token volume; a request-count limit does not explain every rejection.
Do not solve rate limiting by multiplying retries. If demand is persistently higher than capacity, reduce concurrency, shorten inputs, prioritize work, cache eligible results, or route selected tasks to an approved alternative.
3. Slow responses and timeouts
An AI API timeout should be configured as a deliberate part of the user experience. Set separate limits for connection establishment, time to first byte, inactivity during streaming, and total request duration where your client supports them.
- Choose a total deadline based on the workflow, not an arbitrary library default.
- Ensure the application, proxy, load balancer, worker, and provider client do not have contradictory timeout values.
- For streaming, distinguish a slow first token from a stalled stream.
- Cancel abandoned requests when the caller disconnects, if cancellation is supported.
- Decide whether a partial response is useful or must be discarded.
- Measure time to first token, total duration, output length, and timeout stage.
Retries after a timeout require special care: the provider may have completed the request even though your client did not receive the result. For read-only generation this may mainly increase cost and latency; for tool-enabled workflows it can repeat an external action.
4. Provider or model fallback
A fallback is a defined alternative, not an automatic second attempt against any available model. Document which failures activate it and what quality trade-off is acceptable.
| Failure condition | Preferred response | Fallback consideration |
|---|---|---|
| Invalid request or malformed schema | Fix the request and stop | Do not hide application bugs with fallback |
| Authentication or permission failure | Alert and repair configuration | Use another provider only if credentials and policy are already approved |
| Temporary provider outage | Bounded retry, then fail over if appropriate | Verify prompt, model capability, privacy, and output compatibility |
| Rate limit or capacity rejection | Throttle and reschedule | Route only when the alternative has available capacity and acceptable cost |
| Quality or schema validation failure | Repair, re-prompt, or reject | Use a model with a compatible capability, not merely a different endpoint |
Test fallbacks with representative prompts, long inputs, structured outputs, tool calls, and safety constraints. A provider migration checklist can help when the alternative changes tokenization, message formats, model behavior, or streaming semantics: LLM API Migration Guide.
5. Side effects and idempotency
Make the model-generation step and the side-effect step separate whenever possible. Have the model propose an action, validate it, and execute it through a controlled application path. Use an idempotency key or workflow execution ID for operations that must not run twice.
- Generate one stable operation ID before the first attempt.
- Pass that ID through logs, queues, provider metadata, and downstream services where supported.
- Store completed operations so a retry can return the original result instead of repeating the action.
- Require explicit confirmation or a policy check for high-impact actions.
- Assume a timeout creates uncertainty about completion until the operation is reconciled.
What to double-check
Provider behavior differs, and SDK defaults can change. Before releasing a reliability policy, verify the following in the provider documentation and your installed client version:
- Which status codes and structured errors represent rate limits, overload, invalid requests, and authentication failures?
- Does the SDK retry automatically? If so, which errors, how many times, and with what backoff?
- Are streaming responses retried, resumed, or restarted?
- How are cancellations handled, and can an abandoned request continue consuming capacity?
- Does the provider support idempotency keys for the operation you are using?
- What happens when a response is truncated, contains invalid structured output, or ends without a completion signal?
- Are input limits, output limits, concurrency limits, and token quotas measured independently?
- Can your logs capture request metadata without storing sensitive prompts or responses?
Build a small failure matrix and test it deliberately. Include connection failure, timeout before response, timeout during stream, rate-limit response, provider error, malformed output, duplicate delivery, and fallback exhaustion. Combine these tests with automated prompt evaluation so resilience does not mask a degraded answer: automated prompt testing for LLM apps.
Common mistakes
- Retrying every error: Permanent failures become slower and more expensive. Classify before retrying.
- Using only per-request timeouts: Several attempts can exceed the user’s actual deadline. Enforce one end-to-end budget.
- Ignoring jitter: Synchronized workers can create a second traffic spike after an outage.
- Assuming timeout means failure: The provider may have completed the request. Reconcile uncertain operations before repeating them.
- Failing over without capability checks: A different model may not support the same context size, tools, output format, or quality requirement.
- Letting SDK defaults define policy: Defaults are implementation details. Set and document application-level behavior explicitly.
- Returning raw provider errors: Expose a useful user message while keeping diagnostic details in protected logs.
- Measuring only success rate: Track latency, retries, rate-limit events, fallback rate, incomplete outputs, cost, and queue age.
When to revisit
Review this checklist before seasonal planning cycles, concurrency increases, load tests, provider changes, model migrations, or major prompt and workflow changes. Revisit it immediately after an incident involving timeouts, duplicate actions, unexplained rate limits, incomplete streams, or unexpected fallback usage.
Keep a versioned resilience policy with these fields:
- Request type and business owner
- Maximum end-to-end deadline and attempt count
- Retryable and non-retryable failure classes
- Backoff, jitter, and concurrency settings
- Fallback models or workflows and their known limitations
- Idempotency and reconciliation procedure
- Metrics, alerts, and escalation thresholds
- Last test date and next review trigger
To put the checklist into practice, start with one production workflow. Map its failure modes, set a single deadline, disable ambiguous automatic retries, add structured attempt logging, and test each decision in the failure matrix. Then introduce bounded retries, throttling, and a tested fallback one at a time. This incremental approach makes reliable LLM API calls easier to observe, compare, and improve as your AI development workflow changes.