Natural conversation lives and dies on latency. When a voice agent takes longer than a human would to respond, callers hesitate, interrupt, or hang up. Hitting sub-second round-trip latency across speech-to-text, reasoning, and text-to-speech is the single hardest engineering constraint in production voice AI.
The Latency Budget
Every voice turn passes through the same pipeline: capture audio, detect end-of-speech, transcribe, generate a response, synthesize speech, and stream it back. To stay under 600ms perceived latency, each stage gets a strict budget — and the biggest wins come from overlapping stages rather than optimizing them in isolation.
Voice Activity Detection (VAD)
A responsive agent must know the instant a caller stops speaking. We run a lightweight streaming VAD on the raw audio so end-of-turn is detected in tens of milliseconds, not by waiting for a fixed silence timeout. This alone removes hundreds of milliseconds of dead air.
Streaming Everything
We never wait for a full transcript before reasoning, and we never wait for a full response before speaking. Partial transcripts feed the model as they arrive, and generated tokens stream into the text-to-speech engine sentence-by-sentence so the caller hears the first words while the rest is still being produced.
- Streaming STT: Interim results start the reasoning step early.
- Token streaming: The first synthesizable chunk is sent to TTS immediately.
- Chunked TTS: Audio plays back as it is generated, not after.
Edge Deployment & WebRTC
We route audio over low-latency WebRTC sockets and place inference nodes close to the caller. Removing legacy telephony hops and keeping the audio path on modern transport shaves the network overhead that traditional SIP stacks quietly add.
Handling Interruptions
Real conversations are full of interruptions. Active double-talk detection lets the agent stop mid-sentence the moment the caller speaks, discard the interrupted response, and re-plan — the behavior that makes an agent feel human rather than robotic.
Conclusion
Sub-second voice is an architecture problem, not a model problem. Overlap the pipeline, stream aggressively, detect turns fast, and deploy close to the user — and callers stop noticing they are talking to an AI at all.