Real phone calls, answered and placed by a conversational AI.
Project
AI engineering
Year
2025
Our focus
Voice AITwilioWebSockets
How do you connect a real phone call to a conversational AI in real time?
Bridge the two sockets and stay out of the way. Twilio opens a Media Streams connection carrying the caller's audio, the bridge authenticates a conversational AI session with a signed URL, and relays audio frames in both directions. Prompt, opening line and language are set per call.
System architecture · Voice AI Agent
01 /What does the bridge actually do?
Very little, deliberately. It accepts Twilio's media-stream socket, opens an authenticated session with the conversational AI, and passes audio frames between them in both directions.
Keeping it thin is the design. Every millisecond of processing inserted into a live call is a millisecond of awkward silence, and a phone conversation has a much lower tolerance for latency than a chat interface. The bridge relays; it does not think.
02 /Why authenticate with signed URLs?
Because a socket endpoint that anyone can connect to is a bill waiting to happen, and a voice agent session costs real money per minute.
A signed URL scopes authorisation to one session, so a connection has to have been issued rather than merely discovered.
03 /How is interruption handled?
The caller talks over the agent, and the agent stops. That behaviour is what separates a conversation from a voicemail system, and people do it constantly without noticing.
Handling barge-in properly means tracking call state through its start, media and stop events so the bridge knows what is currently playing and can cut it.
04 /What can be configured per call?
Prompt, first message and language, plus dynamic variables passed in at session open. The same deployment can answer as a clinic in one call and a dealership in the next.
That is what makes the bridge reusable rather than a single-purpose bot. The behaviour lives in the session configuration, not in the code.
05 /How do outbound campaigns use the same path?
A campaign runner places the call through Twilio's REST API, and once it connects it becomes the same media stream the inbound path already handles.
Seven reference variants shipped with it, covering inbound answering, outbound campaigns and automation flows, so a new use case starts from a working example rather than from scratch.
06 /What we delivered
Bidirectional Twilio-to-AI audio bridge
Signed-URL session authentication
Per-call prompt and language control
Inbound and outbound campaign flows
07 /The outcome
A real-time telephony bridge shipped with seven reference variants covering inbound, outbound and automation flows.
08 /Build at a glance
Telephony
Twilio voice webhooks and REST
Transport
Twilio Media Streams, bidirectional
AI session
ElevenLabs conversational AI over WebSocket
Authentication
Signed URL per session
Bridge role
Frame relay only, kept deliberately thin
Interruption
Barge-in handled via call state tracking
Per-call config
Prompt, first message, language, dynamic variables
Directions
Inbound answering and outbound campaigns
Shipped with
Seven reference flow variants
COMMON QUESTIONS
Questions people actually ask
How do you connect Twilio to a conversational AI?
Have TwiML open a Media Streams socket on the call, then run a bridge that authenticates a session with the AI provider and relays audio frames in both directions. The bridge should stay thin, because any processing added in the middle becomes audible latency in a live conversation.
What is barge-in handling in a voice AI agent?
Letting the caller interrupt and having the agent stop talking immediately. People do this constantly in real conversation without thinking about it, and an agent that talks over an interruption stops feeling like a conversation. It requires tracking call state so the bridge knows what is currently playing.
Can one AI phone agent handle different businesses?
Yes, if the behaviour lives in the session rather than the code. Prompt, opening message, language and dynamic variables are all set per call here, so the same deployment can answer as one business on one call and another on the next.
Why do voice AI sessions need signed URLs?
Because an open socket endpoint is a cost exposure. Voice agent sessions bill per minute, so an endpoint anyone can connect to is a bill anyone can run up. A signed URL scopes authorisation to a single session that had to be issued rather than found.