# Latency

> Latency is how long a user waits for a reply, counting retrieval, tool calls, and the time the model spends generating tokens.

How long a request takes, including retrieval, tools, and the model.

Track: Ship & Maintain
Access: free
Card: https://buildsparkai.com/latency
Plain text: https://buildsparkai.com/latency.md
Last checked: 2026-10-18

## What it is

OpenAI's latency guide says generating tokens is almost always the highest latency step in a model call. It also states a heuristic that cutting half the output tokens may cut about half the latency. That half is vendor-reported, not a measurement from this card. OpenAI's streaming guide says that by default the API generates the entire output before sending it, and that streaming lets you process the beginning while the model continues generating the rest. Retrieval and tool calls sit outside that model call. OpenTelemetry's span conventions time retrieval and tool execution as their own operations, from start until the operation finishes or errors. The wait a person feels is those steps plus the model. The picture has no milliseconds.

## Why it matters

**Cost.** A smaller or faster model can change both the wait and the bill. OpenAI's latency guide says smaller models usually run faster and cheaper, and that you still check quality on your task. This card does not add a price.

**Quality.** Cutting the reply short to save time can drop the part the user needed. Shorten the reply only after the eval set still passes. Streaming shows the start while the rest is still generating, which helps when the user can read the beginning.

**Risk.** OpenAI's streaming guide says streaming makes moderation harder, because a partial reply is harder to evaluate, and that moderation scores requested with the generation arrive after the full output. A fast wrong reply is still wrong.

## Try today

On one request, write down three waits you can see: time until retrieval returns, time until the tool returns, and time until the first token and the last token. You do not need a benchmark suite to see which step dominates.

## Common mistake

Timing only the model call and calling that the product's latency. The user is also waiting on retrieval and tools.

## Self-check

What does OpenAI's streaming guide say streaming lets you do?

Process the beginning of the reply while the model is still generating the rest

## Practice and deep

Coming soon. One request with three written waits: retrieval, tool, and first and last token.

Coming soon. How OpenAI separates token generation from streaming, and how OpenTelemetry times a retrieval span and a tool span on their own.

## Sources

- [OpenAI, Latency optimization](https://developers.openai.com/api/docs/guides/latency-optimization)
- [OpenAI, Streaming API responses](https://developers.openai.com/api/docs/guides/streaming-responses)
- [OpenTelemetry, Semantic conventions for generative AI spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md)

Opened 2026-10-18. Vendor, framework, or standards docs. This card does not add a rate, a usage count, or a score of its own. Numbers quoted from a source are labelled vendor-reported or as that document's own words. The half-the-tokens heuristic is OpenAI's wording on the latency guide, not a measurement made for this card.