A caller asks the AI voice agent: "How long does the consultation take?"
Agent A responds in 350ms. Agent B responds in 1,200ms. The same words come out of both. The same answer is given. Agent A keeps the caller engaged for 4 minutes through the rest of the qualification. Agent B is hung up on at second 47 of the call.
The difference between 350ms and 1,200ms is the difference between sounding like a person and sounding like an automated system five years ago. It is also the difference between a converted lead and a lost call. This post is the practical breakdown of latency budgets for AI voice agents: why sub-second is the working threshold, the five layers that consume the budget, where the engineering wins are, and what to ask vendors.
Why latency matters at the second-by-second level
Human conversational turn-taking has a measurable rhythm. Studies on conversational latency (Stivers et al. 2009; Levinson 2016) put the modal cross-cultural turn gap at roughly 200ms, with most conversations falling in a 0 to 700ms range between speaker turns. Beyond about 800ms, listeners start to perceive the gap as awkward or as a sign of disengagement, processing difficulty, or absence.
For AI voice agents, the perceived-latency threshold sits around 800 to 1,000ms. Below that, the agent feels conversational. Above that, the agent feels mechanical. By 2 seconds, callers begin to repeat themselves, ask "are you still there?" or hang up.
Three behavioral signals from production deployments:
- Caller hang-ups concentrate in calls where p95 turn latency is over 1.5 seconds.
- Caller-side speech overlap (the caller starts talking before the agent finishes) rises sharply when latency drifts over 1 second, because the caller assumes the agent has stopped.
- Calls consistently under 800ms feel "unsurprisingly responsive" and rarely surface as a complaint.
The category target for AI voice latency in 2026 is sub-1-second consistently, sub-500ms for the best implementations.
The five-layer latency budget
A voice-agent turn passes through five layers, each adding latency:
- Voice activity detection (VAD). The system has to decide the caller has stopped speaking. Conservative VAD waits 300 to 500ms after silence to avoid false-positive turn endings. Aggressive VAD cuts to 100 to 200ms but increases the chance of cutting off the caller mid-sentence.
- Speech-to-text (STT). The caller's audio gets transcribed. Streaming STT (transcription arrives as the caller speaks) finishes 100 to 300ms after the VAD-detected end of speech. Batch STT runs 500ms to 2 seconds slower.
- LLM inference. The transcribed input plus conversation history runs through the language model. First-token latency varies by model: frontier-class fast models (Flash-tier reasoning) sit around 200 to 400ms, GPT-4o around 300 to 500ms, smaller models (3B to 7B) under 100ms. Total response generation can run 400 to 1,500ms depending on output length and model.
- Text-to-speech (TTS). The model's text output gets synthesized to audio. Streaming TTS (audio playback begins before the full text is synthesized) starts in 100 to 300ms. Batch TTS waits for the full text and runs 400 to 800ms.
- Telephony round-trip. The audio has to travel: caller's phone to carrier to vendor infrastructure to caller's phone. Twilio Media Streams adds roughly 150 to 300ms one-way, so 300 to 600ms round-trip.
Stack the floors of each layer (300 + 150 + 250 + 150 + 300 = 1,150ms) and you are already over budget. The best implementations hit sub-1-second by overlapping layers (streaming STT during VAD, streaming TTS during inference) rather than running them strictly sequentially.
What each layer actually costs in milliseconds
A representative breakdown for a well-tuned 2026 implementation:
| Layer | Best case | Typical | Worst case |
|---|---|---|---|
| VAD | 100ms | 300ms | 500ms |
| STT (streaming) | 80ms | 200ms | 400ms |
| LLM (first token) | 200ms | 350ms | 700ms |
| TTS (streaming start) | 80ms | 200ms | 350ms |
| Telephony round-trip | 200ms | 300ms | 500ms |
| Total (sequential) | 660ms | 1,350ms | 2,450ms |
The total row is deceiving in one respect: the layers can run partially in parallel. Streaming STT starts producing tokens before the caller finishes speaking, which means the LLM can start processing during the tail of caller speech. Streaming TTS starts playing the agent's response before the full response is generated. Sophisticated implementations achieve 400 to 700ms perceived latency by aggressively overlapping these phases.
Where the engineering wins are
The biggest wins by layer:
1. VAD tuning. Aggressive VAD (200 to 300ms silence threshold) cuts perceived latency by 200 to 300ms versus default settings. The cost is occasional caller cutoff, which feels worse than a slow agent. The right answer is adaptive VAD that learns from the caller's speech rhythm: long-pauser callers get longer thresholds, short-pauser callers get aggressive thresholds.
2. Streaming everything. STT and TTS both have batch and streaming modes. Streaming is universally better for voice. Vendors still using batch are 300 to 800ms slower than they need to be.
3. Smaller LLMs on shorter turns. Most turns of an intake conversation are short responses ("yes," "no," "the address is"). These do not require a frontier-tier model. Routing short-response turns to a 3B to 7B model and the substantive turns to a frontier model cuts average inference latency by 200 to 400ms.
4. Co-locating infrastructure with the carrier. Twilio Media Streams round-trip varies by 100 to 200ms based on geographic infrastructure. Deploying voice infrastructure in the same region as the carrier endpoints cuts measurable latency.
5. Prefetching predicted next-turn TTS. If the agent is likely to respond with one of a small set of answers, the TTS for those answers can be prefetched while the LLM is still generating. This cuts perceived latency on common-response turns by 200 to 400ms.
The point: latency is not a single number. It is a budget allocated across five layers, with overlap opportunities. Vendors who think of it as "the LLM is slow" are missing 40 to 60% of the available optimization.
The latency floor in 2026 and where it is heading
The current floor for production AI voice in 2026 is roughly 400ms perceived end-to-end latency, achievable by best implementations on optimized infrastructure with streaming everything and aggressive VAD. This is below the human conversational threshold and is genuinely indistinguishable from a fast human responder in most contexts.
Three things still pushing the floor lower:
- Smaller specialized voice models (sub-1B parameter models tuned specifically for voice agents) are trending toward 100ms first-token latency.
- End-to-end voice models (GPT-4o Realtime and equivalent native-audio frontier models) collapse the STT-LLM-TTS pipeline into a single model, cutting layer-handoff overhead by 100 to 200ms.
- Edge deployment (model inference closer to the caller geographically) is starting to cut telephony round-trip on premium vendors.
The likely floor by the end of 2026 is 250 to 350ms perceived latency for top-tier implementations. The category will largely have converged on "indistinguishable from human" within 18 months.
Five vendor-evaluation questions
The diligence pass for latency takes 15 minutes. Five questions:
- What is your p50 and p95 turn latency? Specific numbers, measured end-to-end. Vendors who answer "we are fast" without numbers do not measure.
- What is your latency at turn 15 of a long conversation versus turn 1? Latency drift on long calls is real (covered in detail here). The answer should be that turn 15 is not materially slower than turn 1.
- Which models do you use, and do you use streaming STT and TTS? "Streaming" is the right answer. Anything else is unoptimized.
- What is your VAD threshold, and is it adaptive? The right answer is adaptive VAD around 200 to 400ms baseline.
- Where is your voice infrastructure located, and how does that interact with my caller geography? Ideally co-located with the carrier endpoints serving the caller's region.
If a vendor cannot give specific numbers for p50 and p95, the latency story is marketing, not engineering.
For the production-failure context where latency surfaces, see what AI voice agents fail at in the first 30 days. For the integration architecture, see CRM integration models. To see latency-tuned voice in a working demo, book a call.