nestonexStart a project How do you let a caller interrupt an AI phone agent mid-sentence?
Track call state and stop sending audio the moment inbound speech is detected. A bidirectional media stream carries the caller and the agent simultaneously, so the bridge knows what is currently playing and can cut it. Without that, the agent talks over the caller and the call stops feeling like a conversation.
People interrupt constantly and without noticing. They start answering before a question finishes, they cut in to correct a misheard detail, they say yes halfway through the options.
An agent that keeps talking through that is instantly recognisable as a machine, and callers respond by talking louder or hanging up. Barge-in is not a refinement on top of a working voice agent, it is part of what makes one work at all.
Which state the call is in, and what is currently playing. A media stream sends discrete events for the call starting, for each audio frame, and for the call ending, and the bridge tracks that lifecycle rather than treating audio as an undifferentiated flow.
When inbound speech is detected during agent playback, the bridge stops relaying the agent's audio immediately and signals the AI session that it was interrupted, so the agent's own state reflects that it did not finish.
Because every millisecond of processing inserted into the path becomes audible. A phone call has far less tolerance for delay than a chat interface, where a second of thinking reads as considered rather than broken.
This is the argument for keeping the bridge deliberately thin. It authenticates the session, relays frames in both directions and tracks state. It does not transcribe, analyse or enrich, because anything it does is time the caller spends in silence.
Echo, most often. If the agent's own output is picked up on the inbound leg it can trigger a false interruption, which sounds like the agent cutting itself off.
Background noise does the same on a call from a car or a street. The detection threshold becomes a tuning problem with no universally correct setting, trading false interruptions against sluggish ones.
Our voice agent bridges Twilio Media Streams to an ElevenLabs conversational AI session, authenticated per call with a signed URL, relaying audio frames in both directions with prompt, opening message and language set per call.
It shipped with seven reference variants covering inbound answering, outbound campaigns and automation flows, so a new use case starts from a working example rather than from an empty file.
The ability for a caller to interrupt the agent mid-sentence and have it stop immediately. It requires tracking call state so the system knows what is currently playing, and cutting that playback as soon as inbound speech is detected.
Usually because processing was added into the audio path. Every step between the caller speaking and the agent responding is audible silence, which is why a bridge should relay and authenticate rather than transcribe or enrich in the middle of a live call.
TwiML opens a WebSocket carrying the call's audio as base64-encoded frames, with discrete events for the stream starting, each media frame, and the stream stopping. It is bidirectional, so the same socket carries the agent's audio back to the caller.
Partly, and it is a tuning problem rather than a solved one. Noise from a car or a street can trigger a false interruption, so the detection threshold trades cutting off wrongly against responding sluggishly. There is no setting correct for every caller.