When an external service slows down, retrying feels helpful. If every caller retries immediately, the same policy can add traffic to an already struggling dependency. Resilience needs a limit on how much additional work a failed request is allowed to create.
Budget the whole request
Set connection and request timeouts according to the operation’s needs, and keep all attempts within an overall deadline. A customer-facing request cannot spend its entire time budget waiting for one optional dependency. Decide whether a partial result, queued job, or clear failure is appropriate.
AWS’s Builders’ Library explains how retries, backoff, and jitter interact. Backoff increases the wait between attempts; jitter varies that wait so callers do not all return together. Cap attempts and delays, and check whether your client library already retries before adding another layer.
Distinguish an unknown result from a failed action
A timeout after submitting an order does not prove that the order was never created. Use a stable operation identifier and a server-side idempotency contract where repeated requests can cause side effects. If the destination cannot support that contract, plan a status check or reconciliation step.
Do not retry invalid input or an authorization failure as though it were a transient connection problem. Classify errors using the provider’s documented behavior, including any retry guidance returned with rate limits.
Test recovery under pressure
Simulate a slow dependency, a brief interruption, and a sustained outage. Observe attempt counts, queue growth, connection usage, and the time taken to recover after the dependency returns. For a checkout scenario, confirm that users receive a useful outcome and that duplicate orders cannot be created. Keep the chosen limits visible in configuration and revisit them when traffic or dependency behavior changes.
