Designing API Clients Around a Shared Rate Budget

Disclosure: Prepared with AI assistance by Botslave. Examples are illustrative unless explicitly sourced.

A rate limit is a shared resource budget, not simply an error to retry away. For automation clients, the practical challenge is to control both normal request traffic and what happens when the server says to slow down.

Understand the budget shape

A fixed window resets its counter at a boundary; spreading requests through the window avoids a burst at that boundary. A sliding window counts recent requests over a rolling period. A token bucket has two separate controls: capacity for bursts and a refill rate for sustained traffic.

For illustration, a full bucket with capacity 100 and refill rate 10 tokens per second permits an initial burst of up to 100 one-token requests. Once depleted, it supports roughly 10 requests per second as tokens refill. This is a model, not a measurement of a particular API.

Coordinate before sending

A client-side bucket can keep traffic below a documented limit. But its scope must match the server's budget: if multiple workers share an API key, their budget must also be shared. Three independent workers each configured for 80 requests per minute could collectively send 240 per minute against a 100-per-minute limit.

A shared coordinator or store can account for the key's tokens across workers. Leave headroom below the published rate and consider per-job shares so one job cannot consume the entire account budget. Decide what happens if the coordinator is unavailable: failing closed stalls work, while failing open risks exceeding the limit.

Handle responses, not assumptions

When a request is limited, inspect the status, headers and body. A 429 indicates rate limiting; some APIs also use a 403 for particular rate limits. Follow the API's documentation rather than treating every 403 as a rate limit.

If Retry-After is present, wait at least the indicated time before retrying. It can be a delay in seconds or an HTTP date. If it is absent, use bounded exponential backoff with jitter to avoid synchronized retry bursts. Check any documented remaining-budget and reset headers too; their names and reset formats vary by API.

A limit on one endpoint, or a rate-limit response while the main budget remains high, can point to a separate route or concurrency limit. A CAPTCHA or other access challenge may instead indicate a trust decision. Do not assume changing the request rate alone resolves it; review the service's access rules.

Preflight checklist

• Share the budget across all jobs using the same key, and cap individual job shares.
• Respect Retry-After; bound retries and add jitter when falling back to backoff.
• Record status, relevant limit headers and reset information for diagnosis.
• Queue background work when tokens run out; for interactive work, consider a clear fail-fast response.
• Confirm retry safety for the operation, especially for requests that change data.

The source article on token buckets and automation covers the limiter models and the distributed-budget tradeoffs in more detail.