All worknestonexStart a project
SELECTED WORK/11/AI ENGINEERING

Voice AI Agent

Real phone calls, answered and placed by a conversational AI.

Project
AI engineering
Year
2025
Our focus
Voice AITwilioWebSockets

How do you connect a real phone call to a conversational AI in real time?

Bridge the two sockets and stay out of the way. Twilio opens a Media Streams connection carrying the caller's audio, the bridge authenticates a conversational AI session with a signed URL, and relays audio frames in both directions. Prompt, opening line and language are set per call.

Voice AI Agent architecture: Twilio media streams bridged frame by frame in both directions to a conversational AI session, with prompt and language set per call
System architecture · Voice AI Agent

01 /What does the bridge actually do?

Very little, deliberately. It accepts Twilio's media-stream socket, opens an authenticated session with the conversational AI, and passes audio frames between them in both directions.

Keeping it thin is the design. Every millisecond of processing inserted into a live call is a millisecond of awkward silence, and a phone conversation has a much lower tolerance for latency than a chat interface. The bridge relays; it does not think.

02 /Why authenticate with signed URLs?

Because a socket endpoint that anyone can connect to is a bill waiting to happen, and a voice agent session costs real money per minute.

A signed URL scopes authorisation to one session, so a connection has to have been issued rather than merely discovered.

03 /How is interruption handled?

The caller talks over the agent, and the agent stops. That behaviour is what separates a conversation from a voicemail system, and people do it constantly without noticing.

Handling barge-in properly means tracking call state through its start, media and stop events so the bridge knows what is currently playing and can cut it.

04 /What can be configured per call?

Prompt, first message and language, plus dynamic variables passed in at session open. The same deployment can answer as a clinic in one call and a dealership in the next.

That is what makes the bridge reusable rather than a single-purpose bot. The behaviour lives in the session configuration, not in the code.

05 /How do outbound campaigns use the same path?

A campaign runner places the call through Twilio's REST API, and once it connects it becomes the same media stream the inbound path already handles.

Seven reference variants shipped with it, covering inbound answering, outbound campaigns and automation flows, so a new use case starts from a working example rather than from scratch.

06 /What we delivered

  • Bidirectional Twilio-to-AI audio bridge
  • Signed-URL session authentication
  • Per-call prompt and language control
  • Inbound and outbound campaign flows

07 /The outcome

A real-time telephony bridge shipped with seven reference variants covering inbound, outbound and automation flows.

08 /Build at a glance

TelephonyTwilio voice webhooks and REST
TransportTwilio Media Streams, bidirectional
AI sessionElevenLabs conversational AI over WebSocket
AuthenticationSigned URL per session
Bridge roleFrame relay only, kept deliberately thin
InterruptionBarge-in handled via call state tracking
Per-call configPrompt, first message, language, dynamic variables
DirectionsInbound answering and outbound campaigns
Shipped withSeven reference flow variants
COMMON QUESTIONS

Questions people actually ask

How do you connect Twilio to a conversational AI?

Have TwiML open a Media Streams socket on the call, then run a bridge that authenticates a session with the AI provider and relays audio frames in both directions. The bridge should stay thin, because any processing added in the middle becomes audible latency in a live conversation.

What is barge-in handling in a voice AI agent?

Letting the caller interrupt and having the agent stop talking immediately. People do this constantly in real conversation without thinking about it, and an agent that talks over an interruption stops feeling like a conversation. It requires tracking call state so the bridge knows what is currently playing.

Can one AI phone agent handle different businesses?

Yes, if the behaviour lives in the session rather than the code. Prompt, opening message, language and dynamic variables are all set per call here, so the same deployment can answer as one business on one call and another on the next.

Why do voice AI sessions need signed URLs?

Because an open socket endpoint is a cost exposure. Voice agent sessions bill per minute, so an endpoint anyone can connect to is a bill anyone can run up. A signed URL scopes authorisation to a single session that had to be issued rather than found.

THE WORK BEHIND IT

Services this project drew on

ANOTHER PERSPECTIVE / NEXT PROJECTRAG from first principles

Retrieval-augmented generation, built from the ground up.