Audio Agent Best Practices

This article gathers common practices for the Audio Agent across selection, tuning, and going live, helping you get to a voice Agent that "hears clearly, answers accurately, and speaks naturally" faster. For the meaning and value ranges of each parameter, see the Configuration Guide.

How to Choose the Engine Mode

First decide on the mode based on your needs, then pick the model:

Want the lowest latency and natural, conversational speech → Real-time model (REALTIME): lowest latency, well suited to scenarios sensitive to reaction speed with relatively open scripts, such as casual-chat practice and voice navigation.
Need strong text capabilities, tool calls, and strict scripts → ASR-LLM-TTS: highest latency, the text large model offers the highest controllability, and it can reliably integrate knowledge bases, databases, and tools—well suited to voice customer service, business processing, and similar scenarios; the timbre is determined by a standalone TTS, making it easy to unify the brand voice.
The model natively supports voice input and you want to simplify the pipeline → LLM+TTS: medium latency, skips a standalone ASR stage, and lets the large model "understand" the voice directly for a shorter pipeline.

VAD and Interruption Tuning

VAD determines "when the Agent considers you to have finished speaking," which directly affects the experience. Make trade-offs by scenario:

Voice customer service / hotline (needs to be fast and interruptible): raise the voice interruption sensitivity appropriately so users can cut in at any time; pause segmentation can be slightly smaller to shorten the response wait. The trade-off is being more easily interrupted by mistake in noisy environments.
In-depth Q&A / narration reading (needs to be stable, with fewer accidental interruptions): lower the voice interruption sensitivity and increase pause segmentation appropriately to avoid cutting in while the user is pausing to think.
Noisy environments: in the Real-time model's Preset mode, raise the volume activation threshold to reduce false triggers from background noise.
Preset mode vs. Semantic mode (Real-time model only): use Preset mode for stable, predictable sentence segmentation; if you want the Agent to judge whether the user has finished speaking based on semantics, use Semantic mode together with response speed to fine-tune the pace.

We recommend trying the defaults first (pause segmentation 500 ms, voice interruption 0.5), then fine-tuning step by step against actual recordings, changing only one parameter at a time.

How to Write the Identity Prompt

In voice scenarios the user "hears" rather than "reads," so prompts should be concise and conversational:

Clearly define the role and boundaries: who it is, what it can do, and what it will not discuss.
Require short-sentence answers, avoiding long paragraphs, bullet points, tables, and other structures unsuitable for reading aloud.
Establish conversational phrasing: read out numbers, amounts, and times in natural spoken form as much as possible, reducing stiff terminology.
Specify the language and tone: for example, "Answer in Simplified Chinese, politely and concisely."

TTS Text Cleanup

Text replies from large models often contain symbols, which get "read" out if synthesized directly. Use the Remove and Replace options to clean them up:

Remove: parentheses (), square brackets [], title marks 《》, asterisks *, hash signs #, and other Markdown and typographic symbols, to avoid reading out "asterisk" or "hash."
Replace with a space: for symbols that require a pause but should not be pronounced, improving the naturalness of sentence segmentation.
Enter paired brackets as a single unit (such as ()), and separate multiple symbols with commas.

Pairing this with an identity prompt that tells the model "do not output Markdown" works even better—two measures reinforcing each other.

Welcome Guidance and Virtual Human Avatars

The welcome message should be short and get straight to the point, stating identity and purpose in one sentence, for example, "Hello, this is XX customer service. How can I help you?"
The speaking / waiting avatar videos must have their first and last frames join together, otherwise there will be an obvious frame jump when looping. The speaking avatar corresponds to "speaking," and the waiting avatar corresponds to "listening."
For multilingual business, remember to configure the welcome message and avatar separately for each language.

Call Control Strategy

Set up a "safety net" for every conversation to avoid idling and resource occupancy:

Quiet-conversation control: set a reasonable silence timeout (default 10 seconds) with a natural quiet-conversation prompt, such as "Are you still there? Would you like me to continue?" to bring the user back.
Automatic hang-up: use maximum call duration (default 600 seconds) and maximum number of silences (default 5 times) as a fallback to prevent unlimited calls or lines held open with no one responding for a long time. Keep the hang-up prompt within 20 tokens—a brief goodbye is enough.

Notes on Using Background Sound

The background volume should not drown out the human voice (default 30, range 1~50); too high will lower recognition accuracy and call clarity.
Custom background sound must be mp3 and ≤ 1 MB.

Speech Recognition Quality

For specific domains, make good use of ASR transcription instructions to prompt proper nouns, brand names, and industry terminology, which significantly improves recognition accuracy.
When the user's language is fixed, specify the voice language rather than using "Auto-detect" for better stability.
The platform has built-in voice quality validation: it automatically discards audio that is too short (less than 1 second) and filters out common recognition "hallucination" phrases (such as meaningless results like "Thanks for watching" or "Please subscribe"), reducing false triggers. Such results being ignored is normal and requires no extra configuration.

Going Live and Integration

Website embedding: embed the Audio Agent as a voice conversation widget on your website; see iframe Integration.
Phone access: bind a Twilio number for inbound calls, or route inbound calls through a third-party SIP system; see the Phone System Integration Guide.

Billing and Concurrency

Voice calls deduct credits based on voice usage. When credits are used up, incoming calls are rejected, so keep a buffer.
Watch the concurrent-call limit: new incoming calls beyond the limit are rejected. Before going live, assess concurrency needs based on expected call volume.
Call records can be viewed in Logs / Conversation Records, including the conversation source, caller number, multi-turn Q&A, and runtime sequence, making it easy to review and optimize.