The first time someone experiences a great AI voice agent, they are often surprised by how natural it feels. There is no awkward multi-second pause after you speak. The agent picks up immediately, responds quickly, and sounds present in the conversation. This does not happen by accident — it is the result of a carefully engineered streaming audio pipeline where every component has been designed and tuned to minimize the time between the caller finishing a sentence and the AI beginning its response.
The naive approach to Voice AI treats the system as a simple loop: record the user's utterance completely, transcribe it, run it through an LLM, synthesize the response, play it back. This works but creates terrible latency — a 5-second utterance plus 2 seconds of transcription plus 1 second of LLM response plus 2 seconds of TTS generation means the caller waits 10 seconds before hearing anything. Real production systems replace every batch operation with a streaming equivalent, achieving total response latencies under 600 ms.
The Pipeline: A Race Against Time
A Voice AI pipeline for phone conversations involves six major stages that run in sequence for each conversational turn. Each stage contributes latency that stacks on top of the PSTN and network delays we discussed in our telephony post. Understanding the pipeline is the first step to optimizing it — you cannot improve what you cannot measure and model.
- 1. Audio Capture — RTP audio received from the PSTN, chunked into 20ms frames
- 2. VAD — Voice Activity Detection identifies speech vs. silence in real-time
- 3. ASR — Automatic Speech Recognition produces a rolling transcript as audio arrives
- 4. LLM — Large Language Model generates response text given the full conversation
- 5. TTS — Text-to-Speech synthesizes audio from the response text
- 6. Playback — Synthesized audio sent back via RTP to the caller
Streaming ASR: Never Wait for Silence
Traditional batch ASR systems require the complete utterance before transcription begins. For a 10-second utterance, you wait 10 seconds before you even have the transcript to send to the LLM. Streaming ASR changes this fundamentally by producing rolling partial transcripts as audio arrives, frame by frame. The transcript is marked as interim (unstable) until end-of-turn is detected, at which point a final stable transcript is produced.
Leading streaming ASR systems like Deepgram Nova-2, AssemblyAI Streaming, and Whisper Live produce partial transcripts every 200 to 500 ms. This enables a powerful technique called speculative transcript feeding: rather than waiting for end-of-turn, a well-designed Voice AI pipeline begins processing the transcript as soon as a complete meaningful phrase appears. If the caller says "I want to book an appointment for tomorrow..." the LLM can begin reasoning about appointment booking before the caller finishes saying "...around 3pm." This pipeline overlap is one of the biggest sources of latency reduction in modern Voice AI architectures.
Deepgram Nova-2 on clean 16 kHz audio: approximately 200 ms to first transcript word. On 8 kHz narrowband telephony audio typical of Indian PSTN calls: 300 to 450 ms. This single stage is often the largest contributor to perceived response latency.
Streaming LLM Inference
Modern LLMs support token streaming: rather than generating the complete response internally and then returning it all at once, they emit tokens one by one as they generate them. The time between sending the prompt and receiving the first token — called Time To First Token or TTFT — is the critical latency metric for Voice AI. For small fast models like Gemini Flash or GPT-4o-mini, TTFT can be as low as 50 to 150 ms. For large frontier models under load, it can stretch to 500 to 800 ms.
The key insight for Voice AI is that you do not need the complete LLM response before starting TTS synthesis. You need only a natural speech boundary — a period, a clause break, or a comma — before you can synthesize and start playing the first chunk of audio. This means the AI can begin speaking its first sentence while the LLM is still generating sentences three and four. Implemented well, the caller cannot tell the difference between the AI knowing the full answer immediately versus assembling it on the fly.
Streaming TTS: The Last Latency Frontier
Traditional TTS systems require the complete text before synthesis begins — generating audio for a 30-word sentence might take 600 to 900 ms. Streaming TTS systems like ElevenLabs Streaming, Cartesia, and Deepgram Aura accept a token stream and begin synthesis immediately as words arrive. The first audio chunk from a streaming TTS system can appear in as little as 80 to 200 ms after receiving the first few words of text.
This means the AI agent can literally begin playing the first word of its response while still receiving text from the LLM for the remainder of the sentence. The overlap between LLM generation and TTS synthesis is where the most dramatic latency wins live. In a well-optimized pipeline with a fast LLM and a low-latency streaming TTS, the total TTFT for audio playback can reach 300 to 400 ms even accounting for PSTN network delay — well within the threshold where conversations feel natural.
Interruption Handling: The Hardest Part
Streaming the AI's speech introduces a problem that batch systems do not face: what happens when the caller interrupts mid-sentence? A production-grade system must handle this gracefully and instantly. When VAD detects new speech from the caller while the AI is speaking, the system must simultaneously silence TTS playback, flush any queued audio chunks, cancel any in-flight TTS generation requests, cancel any in-progress LLM streaming if possible, and restart the ASR pipeline for the new utterance. This cascade of cancellation operations must happen in under 50 ms to feel seamless.
The engineering challenge is that all these components operate asynchronously. The TTS audio buffer might have 800 ms of audio queued up. The LLM might be mid-sentence. Coordinating clean cancellation across all these asynchronous streams without leaking state, without orphaned goroutines, and without the next turn starting with stale context is genuinely difficult. It is one of the areas where production Voice AI systems diverge most sharply from proof-of-concept demos.
The Total Latency Budget for Cirio
Adding everything up for a well-optimized pipeline on Indian telephony: PSTN and network delay (30 to 50 ms), RTP jitter buffering (20 ms), VAD detection (10 ms), streaming ASR to first meaningful phrase (300 ms), LLM time to first token with speculative feeding (100 to 250 ms), streaming TTS first chunk (100 ms), and network back to caller (30 ms). Total: approximately 590 to 760 ms. Cirio targets under 650 ms on clean calls and accepts up to 900 ms on degraded 3G connections. Below 400 ms feels nearly instant. Above 1,200 ms the conversation starts to feel broken.
