How do you implement voice-enabled AI agents for phone and IVR systems?

Voice AI agents combine speech-to-text, NLU, dialogue management, and text-to-speech in a real-time pipeline. Architecture: telephony interface (Twilio, Vonage, or SIP integration), ASR (speech-to-text using Whisper, Deepgram, or AssemblyAI), NLU/dialogue engine (LLM-based with conversation state management), TTS (text-to-speech using ElevenLabs, Azure TTS, or OpenAI TTS), and action execution (tool calls during conversation). Latency optimization is critical for natural conversation: target sub-300ms for first response byte, use streaming ASR and streaming TTS, implement turn-taking detection (end-of-utterance detection vs manual barge-in), and pre-generate common responses. Implementation: use WebSockets for real-time audio streaming, implement VAD (Voice Activity Detection) for efficient processing, handle interruptions gracefully (stop TTS playback when user starts speaking), and maintain conversation context across turns. Telephony specifics: handle DTMF inputs (keypad presses), implement call transfer to human agents, record calls for quality assurance (with consent), and support multiple languages with automatic language detection. Quality metrics: task completion rate, average handle time, customer satisfaction score, and containment rate (percentage resolved without human transfer). Test extensively with diverse accents, background noise conditions, and edge cases like silence, cross-talk, and ambiguous requests.