How AI Voice Agents Work: STT, Realtime Speech, LLMs, TTS, Telephony and Tool Calling

What actually happens when you talk to an AI voice agent?

Imagine calling a dental office and saying: “Hi, I’d like to book a cleaning next Tuesday afternoon.”

From the caller’s perspective, this sounds like one simple sentence. Behind the scenes, a production system may need to coordinate several layers:

The two main AI voice-agent architectures

Cascaded architecture

Cascaded architecture should not be dismissed as “old voice AI.” Modern implementations can stream and pipeline stages instead of waiting for every stage to fully finish before the next begins.

Its biggest advantage is modularity and observability. Engineers can inspect what the speech recognizer produced, what text reached the model, which tool was called, what the tool returned and what response was passed into TTS.

That makes cascaded systems especially useful when predictable workflows, existing text-based agents, auditability or component-level optimization matter.

Realtime speech-to-speech / audio-native architecture

In this architecture, the main model can work directly with audio input and output. That means the system may preserve more acoustic information such as timing, stress, rhythm, intonation, hesitation and emphasis instead of flattening every turn into text first.

The production AI voice-agent stack

How does a phone call reach the AI?

For a phone-based agent, the process usually begins before any AI model receives the audio.

SIP is not the audio stream

SIP—Session Initiation Protocol—is primarily a signaling and session-control protocol. It is used to establish, modify and terminate communication sessions.

Realtime audio is typically transported separately through media protocols and media sessions. That distinction matters because telephony signaling and audio transport are different engineering layers.

WebRTC and WebSockets

Not every voice agent uses a phone number. Browser or mobile voice experiences may use WebRTC, while server-based applications may exchange realtime audio over bidirectional WebSocket connections.

Audio is not transmitted as words

Before an AI system understands a sentence, it receives audio frames. Those frames may need to be decoded, resampled, buffered or re-encoded depending on the telephony provider, codec, model interface and network path.

Different interfaces support different codecs and sample rates. Claims such as “all AI phone calls use one specific codec or sample rate” should therefore be treated as provider-specific, not universal.

Jitter and buffering

Realtime audio packets do not always arrive at perfectly regular intervals. Variation in arrival timing is called jitter.

Buffers can smooth playback, but they introduce a trade-off: more buffering may improve stability while also increasing delay.

How does the agent know when you have finished speaking?

This is one of the hardest problems in natural voice interaction.

Speech
Speech
Speech
Silence
Silence
Threshold reached
→ End of turn

The simplest systems rely heavily on silence and Voice Activity Detection. But silence does not always mean the speaker is finished.

Consider: “I need an appointment for... uh... probably Tuesday.”

A fixed silence threshold can answer too early and interrupt the caller. More advanced systems may combine acoustic signals with linguistic or semantic information to estimate whether the thought is actually complete.

How cascaded systems understand speech

Streaming speech recognition often produces intermediate hypotheses instead of waiting for a full sentence.

"I need..."
"I need an app..."
"I need an appointment..."
"I need an appointment Tuesday."

Earlier partial results may change as more audio arrives. Eventually the recognizer returns a stable or finalized transcript.

That distinction matters for names, addresses, phone numbers, dates, account numbers and appointment times. A production system should be cautious about irreversible actions based on unstable partial transcription.

The transcript paradox in audio-native systems

In a cascaded system, the transcript is normally the explicit text representation passed into the reasoning stage. In some audio-native systems, the architecture can be different:

That means transcript accuracy alone may not reveal whether the agent understood the correct entity, selected the correct tool or completed the right business outcome.

What does the reasoning layer actually do?

Depending on the architecture, the reasoning layer may determine:

• what the caller wants;
• what information is missing;
• whether to ask a clarification question;
• whether approved company knowledge is needed;
• whether a tool or external system should be called;
• which tool and arguments should be used;
• whether the requested action is permitted;
• whether the caller should be transferred;
• what the agent should say next.

A strong production design usually separates probabilistic model reasoning from deterministic business rules and authoritative external data.

Prompt context vs RAG vs live business data

Retrieval-Augmented Generation can improve grounding by supplying relevant information from a knowledge base, but it should not be described as a guarantee that hallucinations cannot happen.

How tool calling turns conversation into action

Tool calling is what allows a voice agent to move beyond conversation.

Caller:
"Can you book me Tuesday at 3?"

        ↓

Model requests:
get_availability(
  service="consultation",
  date="Tuesday"
)

        ↓

Application / Tool Layer:
queries calendar

        ↓

Tool result:
2:30 PM
3:00 PM
4:30 PM

        ↓

Agent:
"Yes, 3 PM is available.
Would you like me to book it?"

        ↓

Caller:
"Yes."

        ↓

Model requests:
create_booking(...)

        ↓

Application:
creates appointment

        ↓

Authoritative result:
booking_id = ABC123

        ↓

Agent:
"You're booked for Tuesday at 3 PM."

End-to-end example: booking an appointment

Read tools and write tools should not be treated equally

How does the AI speak back?

In a cascaded architecture, generated text is passed to a TTS system. Modern TTS can stream audio before the entire response has been synthesized.

But TTS quality is not only about whether the voice sounds realistic. Real production tests should include:

• names and surnames;
• phone numbers;
• addresses;
• currencies;
• dates and appointment times;
• acronyms;
• domain-specific terminology;
• language switching;
• speaking rate and pauses.

Where AI voice-agent latency actually comes from

There is no single component called “voice latency.” A spoken response may depend on several stages.

What happens when the caller interrupts the AI?

Barge-in is often described as “the AI stops talking when the user speaks.” That is only part of the problem.

The key hidden problem is playback-state reconciliation. The server may have generated more text than the caller actually heard before interrupting. If conversation history assumes the whole response was delivered, the system state can become inconsistent with reality.

What happens when something goes wrong?

Production systems should be designed around failure paths, not just successful demos.

The duplicate-write problem is especially important. If a booking API succeeds but the response is lost in a network timeout, blindly retrying the same action can create two appointments.

How does human handoff work?

Human handoff contains at least three separate design questions:

1. Trigger: Why should the call be transferred?
2. Telephony: Where should the live call be routed?
3. Context: What information should the human receive?

Common transfer triggers include explicit caller requests, repeated misunderstanding, sensitive operations, unsupported requests, tool failures or policy-defined escalation.

There is also a terminology difference between blind/cold transfer and attended/warm transfer. A system should not call a transfer “warm” unless the telephony behavior genuinely includes attended-transfer semantics.

What happens after an AI voice call?

{
  "intent": "appointment_booking",
  "appointment_booked": true,
  "appointment_time": "...",
  "human_handoff": false,
  "follow_up_required": true
}

How production voice agents are monitored

Reading transcripts is useful, but it is not enough. A production system may track:

• call/session ID;
• speech-start and speech-end events;
• end-of-turn decisions;
• STT interim and final events;
• model request timing;
• retrieval requests and results;
• tool names, arguments and outcomes;
• TTS start and first-audio timing;
• interruptions and cancelled playback;
• human-transfer attempts;
• errors and connection events;
• business outcomes.

Percentile analysis matters too. A low average latency can hide a small but important group of calls with very slow tail performance.

How should AI voice agents be tested?

Security: the caller is untrusted input

Voice does not remove AI-agent security risks. A caller can still attempt to override instructions, access unauthorized information or trigger unsafe actions.

Recording, transcription, PII and retention requirements depend on jurisdiction and business context, so those questions should be treated separately from the technical architecture.

Common misconceptions about AI voice agents

Does an AI voice agent use ChatGPT?

Not necessarily. “AI voice agent” describes a category of systems, not a single model or vendor.

A voice agent may use a text-based language model behind STT and TTS, a realtime multimodal model, a specialized speech model, deterministic workflow logic or multiple models across different stages.

How does an AI voice agent connect to a CRM?

Usually through a controlled integration layer using APIs, function calling, webhooks, workflow automation, database connectors or integration platforms.

The model should not be assumed to “know” whatever exists in the CRM. The application must retrieve the relevant data and enforce appropriate permissions.

Why do AI voice agents have latency?

Because a spoken answer may depend on network transport, turn detection, speech recognition, reasoning, retrieval, tools, TTS, buffering and playback.

Optimizing one benchmark—such as model TTFT—does not automatically make the complete phone conversation fast.

What happens if the agent does not know the answer?

There is no universal fallback. A production system should define one explicitly.

Depending on the workflow, the agent may:

• search an approved knowledge source;
• ask a clarification question;
• say that the information is unavailable;
• call an authoritative external system;
• take a message;
• transfer the conversation to a human.

Generating an unsupported answer should not be the fallback strategy.

Frequently asked questions