What is Latency in AI? — Plain-Language Definition

What is Latency?

Latency is the delay between a request and the system's response. In AI products, latency usually means how long a user waits after sending a prompt, uploading a file, or triggering an automated task.

Why It Matters

Latency has a direct impact on user experience.

• low latency feels fast and responsive
• high latency feels frustrating and unreliable

The acceptable latency depends on the product. A customer support assistant may need near-real-time replies. A background report generator can tolerate much more delay.

What Affects Latency

Several parts of the system contribute to latency:

• model size
• prompt length
• retrieval steps
• reranking steps
• tool calls
• network overhead
• queueing and infrastructure limits

This is why AI latency is often a full-system problem, not just a model problem.

Key Takeaway

Latency is the speed the user feels. Reducing it often requires improving the whole pipeline, not just swapping models.