Latency

Latency in AI systems measures the time from when a request is sent to when a response is fully received. For user-facing applications, latency directly impacts user experience and satisfaction.

Latency components:

• Network latency: Time to reach the API
• Queue time: Waiting for processing capacity
• Processing time: Model inference duration
• Token generation: Time to produce output
• Response transmission: Sending results back

Factors affecting latency:

• Model size (larger = slower)
• Input/output length
• Server load and capacity
• Geographic distance to API
• Batch size and queuing

Latency benchmarks:

• Excellent: <500ms total
• Good: 500ms-2s
• Acceptable: 2-5s
• Poor: >5s

Optimisation strategies:

• Use streaming for perceived speed
• Choose appropriate model size
• Cache common responses
• Optimise prompts (fewer tokens)
• Use edge deployments