Why latency decides whether people will talk to your voice agent
A voice agent that answers correctly but a second and a half too late still feels broken. Here is the budget that has to be met, and where the milliseconds go.
In natural conversation, the gap between one person finishing and the next starting averages around 200 milliseconds. Push past roughly 500ms and the other party starts to wonder whether they were heard. Past a second, they repeat themselves or talk over you. This is not a preference, it is a well-documented property of how people take turns in speech, and it does not relax because the other party is a machine.
That gives a hard budget. Everything between the customer finishing their sentence and the first syllable of the reply has to fit inside it.
Where the milliseconds actually go
A spoken reply passes through four stages, and they are additive:
| Stage | What happens | Typical cost |
|---|---|---|
| Endpointing | Deciding the customer has actually stopped, not just paused | 50-200ms |
| Transcription | Speech to text on the captured audio | 100-300ms |
| Reasoning | The model deciding what to say and looking anything up | 200-800ms |
| Speech synthesis | Text to speech, to first audible byte | 100-300ms |
Run those sequentially at their worst and you are at two seconds before the customer hears anything. The work is in overlapping them rather than shortening any one in isolation.
Streaming, not batching
The single biggest gain is refusing to wait for each stage to complete before starting the next. Transcription streams partial results while the customer is still speaking. The model begins reasoning on those partials. Synthesis begins on the first clause of the reply rather than the finished sentence, so audio starts playing while the rest is still being generated.
Partial transcripts feed the model before the customer has finished the sentence The reply is synthesised clause by clause, so time-to-first-audio is decoupled from reply length Retrieval runs in parallel with generation rather than blocking it A long answer costs no more before the first syllable than a short one
Endpointing is the underrated half
Deciding when someone has finished speaking is harder than it sounds. Wait too long and you add dead air to every turn. Cut too early and you interrupt someone who was thinking mid-sentence, which is far more damaging than being slightly slow. Aggressive silence thresholds are the most common cause of a voice agent that feels rude.
Handling interruption matters as much as speed. If a customer starts talking over the agent, the agent has to stop immediately, discard what it was going to say and listen. An agent that talks through an interruption reads as broken regardless of how fast it responded.
Latency and audio quality are the same problem
Jitter buffers trade latency for smoothness. Retransmission trades latency for completeness. Every mechanism that protects audio quality on a poor network does it by adding delay, which is why transport choice and the latency budget cannot be designed separately. That is covered in the piece on WebRTC and audio quality.
Frequently asked questions
What is a good response time for an AI voice agent?
Under a second from the customer finishing to the first audible syllable. Between one and two seconds people notice the delay; beyond two they typically repeat themselves or talk over the agent.
Does a longer answer take longer to start?
It should not. If synthesis is streamed clause by clause, the time to first audio is roughly constant regardless of how long the full reply turns out to be.
Why does my voice bot interrupt people?
Almost always endpointing. The silence threshold for deciding a turn has ended is set too short, so a mid-sentence pause is read as the customer finishing.
Does adding a knowledge lookup slow the reply down?
Only if retrieval blocks generation. Run in parallel, the lookup usually completes inside the reasoning window and costs nothing observable.
See it on your own calls
Tell us how your contact centre runs today and we will map it to your setup.
Related
WebRTC and why call audio quality decides accuracy
Packet loss, codecs and what they do to transcription.
Choosing STT and TTS engines
Deepgram, ElevenLabs, Cartesia, Sarvam and Azure.
The handoff is where most AI support fails
Carrying context from AI to human without a rebuild.
One platform for AI agents and human agents
Why integrating two vendors costs more than it saves.