The gap between a voice demo and a voice product is measured in milliseconds and interruptions. This course closes it.
Everyone has talked to a voice assistant that talks over them, pauses two seconds before every reply, or keeps monologuing after they've said 'stop'. That gap between a demo and something people will actually use on a phone call is what this course is about, and it comes down to milliseconds and manners. You'll build the loop end to end: turning speech into text as the person is still speaking, deciding when they've actually finished, generating a reply, and speaking it back before the silence gets awkward — all inside a budget of roughly half a second. Then the parts that make or break it: interruption handling that feels natural, audio transport that survives a weak connection, and real phone calls over the telephone network, with keypad tones, voicemail and calling rules handled properly. You'll compare the assembled pipeline against newer models that take audio straight in and out, profile where your milliseconds are going, let the agent look things up mid-conversation without an awkward pause, and finish with the measurements and monitoring that justify a launch. Python throughout, with browser and telephony code where it belongs.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Waiting for someone to finish talking already costs you a second. Learn to transcribe as they speak, and to know when they're done.
Control pace, emphasis and pauses, start playing before the sentence is finished, and stop instantly the moment someone cuts in.
Real networks drop packets and jitter. Learn what carries live audio well, and how to keep a call intelligible when the connection dips.
Nothing feels more robotic than an agent that talks over you or won't stop. Getting interruption right is most of the perceived quality.
Some models take audio in and give audio back, no transcription in between. Faster and more natural — but you give up control. Learn when to.
Your agent feels sluggish and you don't know where the time went. Profile every hop, close the gaps, and hold the budget under real load.
The phone network is older and stranger than the web — narrow audio, keypad tones, voicemail, and rules about who you may legally call.
Look something up or book something without an awkward silence — plus confirming risky actions out loud and handing off to a human cleanly.
'Sounds good to me' is not a launch criterion. Measure accuracy, speed, task success and cost per minute — then go live with a rollback plan.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Build and ship a complete realtime voice agent — reachable over the web (WebRTC/LiveKit) or over a real phone number (Twilio or equivalent) — that holds a natural conversation, correctly handles barge-in and turn-taking, calls at least two tools mid-conversation (one read-only, one gated behind an explicit confirmation), and meets a latency budget you state and measure (e.g. p95 time-to-first-audio under 700ms). Submit the working agent, a recording or transcript of at least 3 real test calls exercising tool use and at least one genuine interruption, and a written latency report showing your measured p95 against your stated budget.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking a "Voice & Realtime AI Agents" course. It covers, in order: the realtime voice stack (ASR-LLM-TTS loop, the sub-500ms latency budget, audio fundamentals); streaming speech-to-text (VAD, endpointing, confidence, noise, WER); streaming text-to-speech (SSML/prosody, voice cloning, first-byte latency, chunking, interruption); realtime transport (WebRTC vs WebSockets, LiveKit/Pipecat, jitter buffers, loss handling, SFU scaling); turn-taking and interruption (barge-in policy, VAD tuning, overlap, backchannels, context-aware endpointing); speech-to-speech models (OpenAI Realtime API, Gemini Live, vs the assembled pipeline); latency engineering (profiling under load, caching, region selection, percentiles, SLOs); telephony and channels (SIP, Twilio Media Streams, DTMF, call control, mulaw, compliance); function-calling voice agents (tools mid-call, memory, confirmation gates, handoff, error recovery); and evaluating and shipping voice (voice-specific evals, monitoring, cost, A/B testing, load testing, incident response). My context: 1. The voice agent I want to build is: [describe — web widget, phone line, both, and what it needs to do] 2. My current voice-AI experience: [none / I've called a TTS or ASR API once / I've built an assembled pipeline / I've used a Realtime API] 3. My biggest concern right now is: [latency / turn-taking feeling robotic / telephony integration / cost / I don't know where to start] 4. My stack: [language, whether I'm using LiveKit/Pipecat/a Realtime API, telephony provider if any] Given that, answer: - Which module should I start with, and why? - Name the first three things I should measure about my own pipeline before optimizing anything. - Given my biggest concern, name the single most relevant task in this course and what I should build from it first. - If I only had one weekend, what's the smallest version of my voice agent idea that would actually prove the concept? - Name one thing this course will NOT solve for me (e.g. a specific compliance question, or model quality itself), so I have the right expectations.
Take a voice pipeline (yours from this course, or another open one) and produce a full latency optimization pack: stage-by-stage profiling under at least 3 concurrency levels, identification of the highest-variance bottleneck with a concrete fix applied and measured, a TTS/response cache with a reported hit rate, and a written latency SLO with a genuinely measured baseline. Submit the before/after profiling data, the specific fix, and the SLO document.
Build a phone-reachable voice agent that replaces a traditional press-1-for-sales IVR: it must handle at least 3 intents via speech, accept DTMF as an equally valid fallback for the same intents with DTMF taking priority on conflict, detect voicemail versus human answer on at least one outbound test call, gate any outbound calling through a documented compliance check (consent, AI disclosure, calling window), and gracefully hand off to a human with session context attached when needed. Submit a real call log/recording, the compliance gate's test results, and the full call-flow design.
The session model, function calling, and interruption handling behind Module 6.