How AI Voice Agents Handle Interruptions and Turn-Taking: VAD, End-of-Turn Detection and Barge-In

Speech activity is not the same as turn completion

One of the easiest mistakes to make in voice AI is to treat “the microphone became quiet” as equivalent to “the caller is finished.” Those are not the same event.

A caller can pause because they are thinking, searching for a date, breathing, correcting themselves, listening for confirmation, or simply speaking slowly. In a phone conversation, even a short hesitation can carry meaning. If the system answers too early, it cuts the caller off. If it waits too long, the interaction feels sluggish.

That is why production turn-taking is better understood as a sequence of decisions rather than a single detector. OpenAI’s current Realtime documentation, for example, distinguishes a silence-based server_vad mode from semantic_vad, which considers whether the user’s utterance appears complete based on the words they have spoken.¹ This is a vendor-specific implementation, but the distinction is broadly useful: detecting the end of acoustic activity and deciding that the conversational floor has been yielded are separate problems.

What is VAD in voice AI?

Voice Activity Detection (VAD) is a mechanism for detecting whether human speech is present in an audio stream and, depending on the implementation, when that speech starts or stops. In realtime systems, VAD events can be used to segment audio, trigger transcription, mark turns, start responses or interrupt an agent that is already speaking.

OpenAI’s Realtime API exposes events such as input_audio_buffer.speech_started and input_audio_buffer.speech_stopped. Its silence-based mode also exposes settings such as an activation threshold, prefix padding and silence_duration_ms; shorter silence durations detect turns faster.¹

Google’s Live API exposes a similar class of timing controls under its own terminology. Its automatic activity detection includes start-of-speech sensitivity, end-of-speech sensitivity and a silence duration setting. Google explicitly documents the trade-off: a longer silence duration allows longer gaps in the user’s speech, but increases latency.³

The important point is not that every provider implements VAD in the same way—they do not. The important point is that silence thresholds are a control surface, not a universal definition of conversational completion.

VAD vs. endpointing

The words VAD, endpointing and end-of-turn detection are sometimes used loosely, but they are useful to separate conceptually.

These layers can be combined in one product or distributed across several components. The names are not a formal industry standard, but the separation helps explain why a system can detect silence correctly and still interrupt the caller at the wrong moment.

What is semantic VAD?

Semantic VAD adds linguistic completion to the timing decision. OpenAI’s current implementation uses a semantic classifier that scores how likely the user is to be done speaking. When that probability is low, the system can wait longer; when it is high, it can respond sooner. OpenAI gives the example of an utterance trailing off with “ummm…” receiving a longer timeout than a definitive statement.¹

This is not proof that one specific semantic-VAD design is universally superior. It is evidence that production turn-taking can use more than raw silence. Recent research is moving in the same direction: the 2026 JAL-Turn paper combines acoustic and linguistic representations to predict whether the conversational state should hold or shift.⁴ Its reported results belong to its datasets and experimental setup, so they should not be generalized into a universal benchmark.

Why pauses, filler words and self-corrections are hard

Human speech is not a clean sequence of complete sentences. People say “uh,” “one second,” “actually,” restart names, spell email addresses, pause before giving cardinals and dates, or change an appointment request halfway through the sentence.

Consider a caller saying:

A short fixed silence threshold can interpret the pause after “at” as the end of the turn and start answering too early. A very long threshold reduces that risk, but makes clear turns slower. The design problem is therefore not simply “make VAD faster.” It is a trade-off between turn-boundary confidence and response latency.

This is also why there is no defensible universal statement such as “500 ms is the correct silence threshold.” Appropriate values depend on language, audio conditions, caller population, product behavior, model/provider settings and the business task. The right value is something to evaluate, not something to inherit blindly from a demo.

What is barge-in?

Barge-in is the ability for a caller to begin speaking while the agent is talking and cause the system to change what it is doing. It is related to VAD, but it is not the same thing.

VAD may provide the signal that new user speech has started. Barge-in defines the interaction policy that follows: should the current model response be cancelled? Should local audio playback stop? Should buffered audio be discarded? Should the conversation history be truncated? Should the new user audio be accepted immediately?

Google’s Live API makes this distinction concrete. Its default activity handling can treat the start of user activity as an interruption, cutting off the current model response. Google also exposes an interrupted signal that a client can use to stop and empty its playback queue.³

Stopping audio is only half of interruption handling

The most important interruption problem is often hidden after the speaker stops hearing the audio.

A realtime system can contain several versions of an assistant response at once:

If the caller interrupts after hearing only the first half of a response, the conversation state should not behave as if the caller heard the entire response. OpenAI’s current Realtime Agents SDK explicitly addresses this. In WebSocket scenarios, the SDK can track playback and truncate assistant audio to what the user actually heard; its Python SDK documentation recommends a playback tracker for delayed or remote playback such as telephony.²

This leads to a useful production principle:

Without that reconciliation, the model can later refer back to instructions, prices, confirmation questions or disclosures that were generated but never delivered to the caller. In a business workflow, that is not merely awkward—it can change the correctness of the next turn.

Barge-in is not the same as full duplex

Another common terminology problem is to call any system with interruption support “full duplex.” That is too broad.

A turn-based system can support barge-in by listening for user speech while it is playing the agent’s response and then cancelling that response. A genuinely full-duplex dialogue architecture goes further: it models simultaneous or overlapping speech as a normal conversational state rather than as an exception that simply terminates the agent’s turn.

The Moshi research system is a useful reference point. Its architecture models the user and system speech streams in parallel and was designed to handle overlapping speech, interruptions and interjections without relying on explicit speaker-turn segmentation.⁵ That makes it a research example of full-duplex modeling, not a reason to relabel every barge-in implementation as full duplex.

Turn-taking is part of voice AI latency

Latency is not only model inference time. An agent can have fast speech recognition, fast reasoning and fast synthesis, yet still feel slow because the system waits too long before deciding the user has finished.

Conversely, reducing the end-of-turn wait aggressively can make a system appear faster while increasing the number of premature interruptions. That is why turn-decision latency should be evaluated separately from the broader end-to-end latency waterfall discussed in our guide to voice AI latency.

Google’s Live API documentation explicitly notes that increasing the required duration of non-speech allows longer gaps without ending the user’s activity, but also increases model latency.³ OpenAI’s server VAD likewise documents that shorter silence-duration values detect turns more quickly.¹ These are implementation-specific controls, but both illustrate the same engineering trade-off.

Common turn-taking failure patterns

How should turn-taking be tested?

Turn-taking should be tested with audio behaviors that occur in real calls, not only clean scripted sentences. A useful pre-production test matrix includes:

• short and long mid-sentence pauses;
• filler words such as “um,” “uh,” and “one second”;
• self-corrections and restarted sentences;
• names, dates and numbers spoken with hesitation;
• caller speech beginning while the agent is speaking;
• brief backchannels such as “yeah” or “right”;
• background speech or TV noise that may trigger false activity;
• telephone audio with delayed playback;
• multiple interruptions in the same agent response;
• verification that conversation history matches the portion of audio actually played.

The goal is not to prove that interruptions can never fail. It is to determine how the chosen turn policy behaves across the caller population and business workflows that matter. This topic becomes part of a broader regression strategy in our forthcoming guide to testing AI voice agents before production.

Why phone calls make turn-taking harder than a clean microphone demo

Turn detection is often evaluated first in a quiet browser demo, but telephone audio introduces additional variables. The audio may be narrowband or compressed, callers may use speakerphone, echo cancellation may behave differently across devices, and network buffering can shift the apparent timing between speech and playback. In a bridged telephony architecture, the component deciding to interrupt may also be physically separated from the component that is actually playing the audio.

That separation matters. The model may have already generated 1.5 seconds of speech while the telephony provider has played only 700 milliseconds. If the caller interrupts at that moment, “cancel the model response” and “stop what the caller hears” are two related but different operations. The application needs a consistent view of generated, buffered and played audio.

This is why playback tracking becomes especially valuable in remote or delayed playback. OpenAI’s Realtime Agents documentation specifically calls out telephony as a case where a playback tracker can help truncate interrupted responses at the actual playback position rather than assuming generated audio has already been heard.²

Echo and background speech can create false interruption signals

A barge-in system is useful only if it can distinguish the caller from other audio with reasonable reliability. Speakerphone echo, a television, another person in the room or call-center background speech can all increase the probability of false activity detection. Vendor controls such as start-of-speech sensitivity can help tune this behavior, but there is no single sensitivity value that is correct for every phone environment.

For business deployments, the test set should include the same channel the agent will use in production. A browser microphone test does not fully represent PSTN, SIP or mobile-call behavior.

An interruption is a state transition, not just an audio event

It is useful to model barge-in as a small state machine. Before the caller interrupts, the agent may be generating text, synthesizing audio and sending audio to a playback buffer at the same time. The user-speech event forces the system to decide which of those processes should stop and which state should be preserved.

Different providers expose different parts of this lifecycle. Google surfaces interruption and activity events in its Live API; OpenAI’s SDK exposes interruption events and playback-aware truncation behavior. The exact event names are vendor-specific, but the underlying engineering need is the same: an interruption changes both media state and conversational state.

A five-step model for production turn-taking

For practical architecture reviews, B2B Voice uses the following conceptual model. It is an engineering framework, not an industry standard:

This framing matters because a polished voice can hide weak timing behavior. Natural conversation requires more than fast speech synthesis: the system has to manage the conversational floor correctly.

Frequently asked questions

The real turn-taking problem is state, not silence

The simplest voice demos can treat silence as a trigger: user stops, agent talks. Production phone systems need a more careful model. They must distinguish speech activity from turn completion, treat interruptions as a control problem, and keep generated state aligned with heard state.

That is also why turn-taking belongs next to—not inside—the broader architecture and latency discussions. If you want the full system view, start with how AI voice agents work. If you want to understand the speed trade-offs, see what causes voice AI latency.