Map · Ship & Maintain · 05 · Fallbacks & rate limits
Concept card · Ship & Maintain
Fallbacks and rate limits are the product's plan for a model error, a slow response, or a rate-limit response.
What the product does when a model errors, slows down, or hits a rate limit.
Oct 18: First card. The picture separates a temporary rate limit, a spend limit, and an overloaded model. Sources are OpenAI's rate-limit guide and error-code guide.
A rate limit caps requests or tokens in a period of time. OpenAI's rate-limit guide says limits include requests per minute and tokens per minute, and that you can hit either one first. A temporary limit returns HTTP 429. The response can include a Retry-After header, and the guide says to wait at least that long and to add a small random delay so clients do not retry together. The same guide says not to retry quota, billing, or other errors that need a person to act. A hard spend limit also returns 429, and retrying does not raise the limit. HTTP 503 with the code server_is_overloaded means the requested model is temporarily overloaded, and the guide says to follow Retry-After and then retry. Official SDKs already retry some 429 and 503 responses, so a second retry loop in your code can multiply calls. Your product still needs a plan for when retries stop: another model you have already evaled, a stored answer, or a plain error. The picture is that plan. It is not an OpenAI feature.
Retries spend the limit you were already hitting. OpenAI says unsuccessful requests count toward the per-minute limit, so a tight loop prolongs the problem. A fallback model has its own price. This card does not add one. OpenAI also states a vendor-reported rule of thumb: once traffic reaches 1 million input tokens per minute, increase it by no more than 50 percent every 15 minutes. The exact point varies by model.
A fallback model that has not seen your eval set can answer differently. Switch models only after that set has graded the fallback.
Treating every 429 as wait-and-retry hides a spend limit or an empty credit balance. Read the error code before you choose retry, fallback, or stop.
List the errors your app already catches. Mark each one retry, fallback, or stop. For a 429, read the body: a temporary rate limit can wait, and a spend limit cannot be retried into working.
Treating every 429 as wait and try again. OpenAI uses 429 for a temporary rate limit and for spend and credit limits. Those need different actions.
What should the product do with a project spend-limit 429?
Support-docs answer bot uses this concept in one build. The set on this tab is still the note below.
Coming soon. A short list of the errors your app catches, each marked retry, fallback, or stop.
Coming soon. How OpenAI separates a 429 rate limit, a 429 spend limit, and a 503 overloaded model, including Retry-After.