Audio Agent Overview
The Audio Agent is the Agent type in the GPTBots DevSpace built for voice interaction. Users speak instead of type, and the Agent replies with voice, delivering a real-time calling experience close to talking with a real person. It can be embedded in a web page as a voice conversation widget, or answer incoming calls directly through a phone system, making it well suited for voice customer service, phone navigation, voice assistants, spoken-language practice, and similar scenarios.
Compared with a text Agent, the Audio Agent differs in three ways:
- Input and output are voice: how voice is processed varies across the different modes.
- Real-time sessions: a single conversation is maintained over a persistent connection, users can speak up to interrupt at any time (barge-in), and reconnection after a dropped connection is supported.
- Billed by voice usage: credits are deducted based on speech recognition, speech synthesis, and call duration, rather than by the number of text messages.
Core Concepts
Understanding the following concepts will help you configure and debug an Audio Agent:
| Concept | Description |
|---|---|
| ASR (Speech Recognition) | Automatic Speech Recognition transcribes what the user says into text for the large model to understand. |
| LLM (Large Model Inference) | Generates a reply based on the recognized text and context. This layer is the same as a text Agent and can call knowledge bases, databases, memory, and other capabilities. |
| TTS (Speech Synthesis) | Text-To-Speech synthesizes the model-generated text reply into voice and reads it aloud to the user. |
| VAD (Voice Activity Detection) | Voice Activity Detection determines when the user starts and finishes speaking, and is the basis for natural sentence segmentation and voice interruption. |
| Real-time call | A single voice conversation runs over one persistent connection, and the Agent listens and responds as it goes. |
The Three Voice Engine Modes
The Audio Agent offers three engine modes, which determine who handles the "voice — text — voice" pipeline. After creation, you can switch between them at the top of the configuration page; different modes display different models and parameters.
| Mode | Pipeline | Characteristics | Suitable scenarios |
|---|---|---|---|
| Real-time model (Real-time model / REALTIME) |
Lowest latency; a single real-time voice large model handles audio input and output end to end | Lowest latency and natural tone, but relatively limited control over timbre and text | Real-time calls that demand the lowest latency and a conversational style |
| ASR-LLM-TTS | Speech recognition → text large model → speech synthesis, a three-stage pipeline | Highest latency, strongest text-model capabilities, and highest controllability; you can freely combine ASR, LLM, and TTS models | Voice customer service that needs complex business logic, tool calls, and strict script control |
| LLM+TTS | A large model that supports voice input understands the voice directly → speech synthesis | Medium latency; skips the standalone ASR stage, letting the large model "understand" the voice directly | Cases where the chosen large model natively supports voice input and you want to simplify the pipeline |
For selection guidance, see Best Practices. All three modes share capabilities such as knowledge bases, databases, memory, call control, and welcome guidance; only the way the voice pipeline is implemented differs.
Key Features
- Real-time voice conversation: the user speaks, the Agent responds in real time, forming a multi-turn call.
- Voice interruption (barge-in): users can speak up to interrupt while the Agent is talking; sensitivity is adjustable, matching real conversation habits.
- Welcome guidance: a welcome message is played at the start of a call; you can upload two looping videos of a virtual human avatar—"speaking" and "waiting"—to give the voice conversation a visual presence.
- Call control: supports automatic follow-up prompts when the conversation goes quiet, and automatic hang-up on timeout or prolonged silence, preventing idle calls or unlimited occupancy.
- Background sound: you can overlay preset ambient sounds such as office, café, or rain onto the call, or upload custom audio to set the mood (effective only in ASR-LLM-TTS and LLM+TTS modes).
- TTS text cleanup: removes or replaces symbols such as parentheses and Markdown markup in the reply text before reading it aloud, so symbols are not read out.
- Image recognition: the conversation supports recognizing 1 image, so voice scenarios can also process image information.
- Voice quality validation: the platform includes built-in validation that automatically filters out overly short audio and common recognition "hallucination" phrases (such as "Thanks for watching" or "Please subscribe"), reducing false triggers.
- Phone / SIP access: you can bind a Twilio number for inbound calls, or route inbound calls through a third-party SIP system, letting the Agent answer the phone directly. See the Phone System Integration Guide.
Quick Start
- Create: go to DevSpace → Create Agent and select the "Audio Agent" type.
- Choose the engine mode and model: in the Voice Engine panel at the top of the configuration page, choose Real-time model / ASR-LLM-TTS / LLM+TTS, and pick a model for each stage (Audio LLM / ASR / LLM / TTS).
- Configure voice and call parameters: set voice interruption, pause segmentation, welcome message, quiet-conversation follow-ups, automatic hang-up, and so on as needed—see the Configuration Guide.
- Debug and try it out: start a voice conversation in the debug window to verify whether recognition is accurate, interruption is responsive, and the timbre and speaking speed are appropriate.
- Integrate and go live: connect the Audio Agent to your business, with two supported methods:
- Website embedding: embed it as a voice conversation widget on your website.
- Phone system access: bind a number or route inbound calls via SIP transfer.
Billing and Concurrency
Voice calls deduct credits based on voice usage (speech recognition, speech synthesis, and call duration). When credits are insufficient or the concurrent-call limit is exceeded, new incoming calls are rejected. Before going live, make sure your account has sufficient credits and enough concurrency quota to meet expected call volume. For common questions about phone access, refer to the FAQ section of the Phone System Integration Guide.