Cost
A smaller or faster model can change both the wait and the bill. OpenAI's latency guide says smaller models usually run faster and cheaper, and that you still check quality on your task.
Home · Shipping · 04 · Latency
Concept card · Shipping
Latency
What this gets you. See which step the user waits on.
Next on this goal: Fallbacks & rate limits.
Latency is how long a user waits for a reply, counting retrieval (the search for source text), tool calls, and the time the model spends generating tokens (pieces of text).
What changed
Oct 18: First card. The picture adds retrieval and a tool in front of token generation. It has no clock. Sources are OpenAI's latency and streaming guides, and the OpenTelemetry GenAI span conventions.
1 What it is
OpenAI's latency guide says generating tokens is almost always the highest latency step in a model call. It also states a heuristic that cutting half the output tokens may cut about half the latency. That half is vendor-reported, not a measurement from this card. OpenAI's streaming guide says that by default the API generates the entire output before sending it, and that streaming lets you process the beginning while the model continues generating the rest. Retrieval and tool calls sit outside that model call. OpenTelemetry's span conventions time retrieval and tool execution as their own operations, from start until the operation finishes or errors. The wait a person feels is those steps plus the model. The picture has no milliseconds.
2 Why it matters
Quality
Cutting the reply short to save time can drop the part the user needed. Shorten the reply only after the eval set (saved examples you already know how to grade) still passes. Streaming shows the start while the rest is still generating, which helps when the user can read the beginning.
Risk
OpenAI's streaming guide says streaming makes moderation harder, because a partial reply is harder to evaluate, and that moderation scores requested with the generation arrive after the full output. A fast wrong reply is still wrong.
3 How to apply · try today
On one request, write down three waits you can see: time until retrieval returns, time until the tool returns, and time until the first token and the last token. You do not need a benchmark suite to see which step dominates.
Common mistake
Timing only the model call and calling that the product's latency. The user is also waiting on retrieval and tools.
Self-check
What does OpenAI's streaming guide say streaming lets you do?
See it working in the support assistant example.
Coming soon. One request with three written waits: retrieval, tool, and first and last token.
Questions
- What is latency?
- Latency is how long a user waits for a reply, counting retrieval, tool calls, and the time the model spends generating tokens.
- Does the picture show a measured time?
- No. The picture lists steps. It does not show milliseconds. OpenAI's half-the-output heuristic is vendor-reported on the latency guide.
- Does streaming finish the reply sooner?
- OpenAI's streaming guide says you can process the beginning while the model continues generating the rest. The full reply is still generated.