ElevenLabs 教程：如何搭建与品牌调性一致的 AI 语音助手

Imagine calling a retail store to request a refund, being greeted by a human-like AI, and receiving the money in your bank account within minutes. Every interaction is seamlessly occurring without the need for human support personnel. Once considered fictional, AI-powered voice assistants that can handle complex use cases are now a reality.

More enterprises are turning to AI voice assistants for customer-facing applications. Powered by state-of-the-art voice agents and generative AI, these assistants support intent matching, low-latency responses, and functional integrations across use cases.

With tools like ElevenAgents, it’s easier than ever to embed voice assistants into your customer support workflows. To build an AI voice assistant, define the job it should do, then connect speech to text, an LLM, and text to speech into one real-time pipeline. Add your knowledge base and business tools, write system instructions and fallback rules, test for latency and safety, and measure results after launch.

This guide shows how to build an AI voice assistant that aligns closely with your brand.

Summary

An AI voice assistant connects speech to text, an LLM, and text to speech to create conversational AI that can take action.
Keep round-trip response below 500 ms with voice activity detection (VAD) so the voice AI sounds natural and reduces call abandonment.
Integrate with the internal knowledge base and tools to enable context-aware, accurate, and helpful AI responses across different scenarios.
Apply safeguards that require explicit verbal confirmation and hand off to a live human agent when handling emotionally charged or complex queries.
Use ElevenAPI to access high-accuracy interactive voice and audio models designed for live interactions.

What is an AI voice assistant?

An AI voice assistant is software that understands spoken language and lets users interact with connected systems. It combines speech to text, AI models, and text to speech technologies to turn conversations into actions.

Beyond transcribing text, AI voice assistants can carry out tasks, such as launching an app, retrieving data, and controlling machines. Depending on the voice model you use, some AI assistants can sound virtually indistinguishable from human recordings.

AI voice assistants, however, are not a new technology. Many consumers use Siri, Alexa, and Google Assistant, commercial voice assistants that respond to simple commands. But as AI technologies evolve, more organizations are building AI voice assistants for more complex enterprise workflows.

For example, Admiral, an insurance provider, built an AI voice assistant to support millions of customers. The AI assistant answers queries, retrieves documents, and frees human agents from repetitive tasks.

If you’re planning to build an AI voice assistant for commercial or industrial workflows, read on. We’ll take you through steps to build one, using a Python AI voice assistant example to highlight key implementation details.

How to build an AI voice assistant: Define the job your AI voice agent should do

Voice AI commonly automates customer support across call centers. However, given an AI voice agent’s broad capabilities, you can build an AI voice assistant for almost any use case, including sales, product, and HR.

Regardless of your voice agent’s application, make sure it aligns with the business logic. To build a high-performing voice assistant, consult domain experts on common end-user challenges. Then, map the customer interaction journey and set measurable indicators.

For example, if you’re applying voice AI for a sales funnel, the agent must be able to ask pre-qualification questions.

Our AI sales development representative (SDR), Dom, did this, achieving 92% qualification accuracy.

Choose success metrics, channels, and escalation paths

When deployed in an enterprise environment, the agent must perform consistently against specific goals. The narrower the goal, the easier it is to assess and fine-tune the agent.

Therefore, ask questions like these early in development:

What is the tolerable response latency?
What is the expected containment rate?
Does the agent need to converse in multiple languages?
Can the AI assistant handle basic requests from end-to-end without human intervention?
In what circumstances should the agent escalate to human support?

Build from raw APIs or use ElevenAgents?

Before you write code, decide how much of the voice stack you want to own. You can wire speech to text, an LLM, and text to speech together yourself through ElevenAPI, or you can use ElevenAgents, a managed platform that bundles those pieces.

Building from raw APIs gives you full control over every layer. You choose the LLM, write the orchestration code, and own turn-taking, interruption handling, telephony, and monitoring. This path suits teams with specific architecture requirements, and it is the one the Python walkthrough in this guide follows.

ElevenAgents brings models, tools, a knowledge base, and workflows into one dashboard. It connects to telephony providers, such as Twilio and Amazon Connect, to business tools such as Salesforce, Zendesk, and HubSpot. It also includes simulations and automated tests before launch, plus analytics on success rate, CSAT, and transcripts afterward.

Use these questions to choose:

Speed: ElevenAgents lets a team configure and launch an agent from a dashboard, while raw APIs mean building and maintaining the full pipeline.
Control: Raw APIs let you customize every step in code, while ElevenAgents covers common needs through configuration.
Operations: ElevenAgents includes testing and analytics, while raw APIs need you to build your own monitoring.
Team: ElevenAgents suits mixed teams of product, support, and operations staff, while raw APIs suit teams with dedicated engineers.

A practical approach is to validate the use case in ElevenAgents first, then move specific layers to custom code only where you need more control.

Map the AI voice assistant architecture

Once you’ve defined the business goal, map agent functions to the right technologies. An AI voice assistant relies on several key components that work together to turn speech into text, trigger actions, and give a coherent response.

A speech to text model captures verbal utterances, identifies phonemes, and produces transcripts of the conversation.
An AI engine, which ranges from a simple rule-based algorithm or a natural language processing library to a large language model (LLM), acts as the brain of the voice assistant.
A text to speech model synthesizes conversational responses using human-like voices and follows brand guidelines.

ElevenLabs provides Speech to Text and Text to Dialogue voice and audio models that help product and engineering teams architect their AI voice assistants. Our models support nuanced intonation, emotional awareness, and low latency response. You can access the voice models via ElevenAPI.

For AI models, your choice depends heavily on the voice assistant’s goals and requirements. While LLMs, such as OpenAI GPT models, Gemini, and Claude, excel at reasoning, they need additional fine-tuning to meet domain-specific needs. Sometimes, using a simpler neural network or a rule-based parser is more practical.

Beyond voice and AI models, you also need custom software that extracts specific entities from conversations or user requests. For example, when a caller says “I want to talk to Sales Department”, your code must identify “Sales Department” and trigger the appropriate action.

Create API keys and connect speech to text, an LLM, tools, and text to speech

Before you connect to speech and language models. Similarly, you need API keys to make API requests when integrating external service tools into your voice assistant.

To create an ElevenLabs API key, click here.
To use an OpenAI LLM, get your key here.
If you’re connecting to third-party CRMs, ERPs, and other tools, refer to the official documentation to create an API key and find supported APIs.

After getting the API keys, store them in your development environment.

Then, you initiate and access the tools, LLM, and audio models by passing the right arguments to the API.

1. First, connect to ElevenLabs Scribe v2 Realtime to perform automatic speech recognition. ElevenLabs Scribe v2 Realtime allows you to create low-latency voice agents that turn conversations into text in under 150 ms (excluding network and application time).

Below is a Python code snippet that captures mic recording with ElevenLabs Scribe v2 Realtime.

2. Next, you send the transcript to an LLM with strong reasoning capabilities to generate a response. In this example, we use OpenAI GPT-4o mini.

Here’s how you send a transcript to an LLM in your Python code.

3. After receiving the LLM’s output, convert it into a realistic verbal response with ElevenLabs v4 Turbo. Our Text to Speech model is highly emotive, designed for real-time interaction, and has ultra-low latency of ~100 ms (refers to model inference time only. Actual end-to-end latency will vary with factors such as your location and endpoint type used).

The code below shows the agent using a WebSocket to stream voice responses with the ElevenLabs Text to Dialogue API.

Your automatic speech recognition agent should handle multi-turn conversations comfortably. To do so, you need a voice agent capable of knowing when speech starts and stops.

Design conversations that sound natural and stay on task

Conventional interactive voice response (IVR) systems don’t operate beyond the scenarios they’re programmed for. When asked to resolve problems not defined in their rules, these IVRs fail to provide satisfactory responses. Moreover, they don’t always sound natural, which hurts the customer experience.

With advanced AI and voice models, a voice assistant can respond more naturally and handle edge cases. You can architect different scenarios the agent must anticipate and an escalation path for cases it can’t resolve. For example, some customers call to ask about shipping status, while others might be frustrated by defective products and insist on speaking to a CX manager.

Write system instructions, turn-taking rules, and fallback responses

To handle different scenarios, you create specific logic-handling codes and system prompts in the AI voice agent application.

For example, the system prompt below detects frustration and tells the LLM to respond more empathetically.

AI conversations should never feel rushed or interrupted. The voice model must detect when the user has stopped talking and respond in time. ElevenLabs Scribe v2 Realtime lets you activate VAD when listening to microphone input. Unlike conventional VAD, which relies solely on silence duration, ElevenLabs Scribe v2 Realtime uses deep learning algorithms to determine whether a customer has finished speaking.

To enable VAD, set the parameter in your code when interacting with the Speech to Text API.

Next, provide courteous, respectful fallback responses if the agent can’t satisfactorily answer the customer’s request. Similarly, an AI agent might encounter an error while transcribing conversations. Your code should handle technical difficulties and communicate openly to customers.

To provide fallback responses, you use conditional logic in your code. For example, the code snippet below captures error status from the Speech to Text API and triggers appropriate responses.

Build the knowledge and tool layer for reliable answers

LLMs offer strong reasoning capabilities, which enable an AI voice assistant to respond naturally. Conversational response, however, doesn’t always imply accuracy. Without grounding data, LLMs can hallucinate, which causes the voice agent to provide plausible but inaccurate answers.

For example, when you ask GPT-4o mini about a product you recently launched, it can't confidently provide an answer. To do so, you need to connect the LLM to your internal knowledge base with a retrieval-augmented generation (RAG) system. RAG lets the LLM refer to internal product guides and produce an accurate answer.

Add retrieval, API actions, permissions, and human handoffs

To use RAG, you need to create a vector database. A vector database stores documents as numerical representations in a multidimensional space. The following code snippet shows how the voice agent retrieves relevant documents from the vector store before querying the LLM.

Beyond RAG, your AI voice assistant will likely need access to CRM, ERP, and other enterprise systems. This allows the voice agent to surface necessary information, such as customer data, historical transactions, and market trends, as context when crafting responses.

For example, you can integrate ElevenLabs with HubSpot using APIs. This way, you can resolve common issues using the CRM knowledge base, create tickets, and escalate support with full conversation context.

Here’s a code snippet that lets the voice agent create a support ticket in HubSpot.

While AI voice agents can speed up problem resolution, customers must know their options, especially when changes they make could significantly affect their accounts. Unlike text-based chatbots, verbally listing options requires more safeguards. Once they have made a choice, reversing it on a call can be difficult.

In this case, we recommend authenticating the caller and seeking verbal confirmation. For example, if the customer wants to cancel a purchase, your voice agent must explicitly ask, “Are you sure you want to cancel the order?” and wait for an answer.

We show an example below.

How to verify caller identity with an AI voice agent

Before an AI voice agent shares account details or changes an account, it should confirm who it is talking to. A caller’s phone number is a useful signal but does not prove identity. Combine it with other checks:

Match the caller’s number to a customer record in your CRM.
Ask for details only the account holder would know, such as a recent order date.
Validate the answers through a tool call against your records, and never rely on the LLM to judge whether they are correct.
Limit what an unverified caller can do, for example allowing order status checks but not cancellations.

Keep passwords, full payment-card numbers, and authentication codes out of the conversation, as the system prompt above instructs.

AI voice agent with live call transfer and escalation

A live transfer hands the caller to a human agent without making them repeat themselves. In a warm transfer, the voice agent passes the transcript and context to the human before the call connects. Common triggers include:

The caller asks to speak to a person.
The caller sounds frustrated or the conversation becomes emotionally charged.
The agent cannot resolve the request after its fallback responses.
The request needs an action that requires human approval.

In some cases, the AI agent needs to escalate the call to a human agent. Often, the voice agent detects intents indicating frustration or specific words that trigger the transfer. In your code, the call transfer must happen seamlessly and be abstracted from the customers.

Before the switchover, you pass the transcript, customer details, and other supporting context to the live support desk. ElevenLabs provides a transfer-to-number tool that routes calls handled by AI agents to a SIP trunk or Twilio phone number.

The code snippet below shows a Twilio conference handoff.

Build an AI voice assistant without code in ElevenAgents

If you do not need custom code, you can launch the same assistant in ElevenAgents. The steps mirror the architecture above, but you configure each layer in a dashboard instead of writing it.

Create an agent and describe its job in the system prompt, including tone, scope, and when to escalate.
Choose a voice from the Voice Library, or design or clone your own, and set the languages the agent should speak.
Add your knowledge base so the agent can answer from your own documents.
Connect tools and integrations, such as your CRM or help desk, so the agent can take action.
Set up a transfer to a human agent for calls the assistant cannot resolve.
Run simulations and automated tests to check behavior before launch.
Deploy the agent on a phone number or channel, then review success rates, CSAT, and transcripts.

The testing and measurement guidance in the sections below applies whether you build with code or without it.

How to build an AI voice assistant that fits your brand

Organizations can tailor the AI agent’s voice to reflect their brand. A consistent tone throughout the conversation helps build trust. With ElevenLabs, there are several ways to customize your AI agent’s voice.

ElevenLabs Voice Library provides 20,000+ AI voices spoken naturally in 90+ languages. From energetic to calm and confident, you can browse and use any to match your brand tone.
ElevenLabs Voice Design lets you create a unique voice footprint with a text prompt. Describe the pace, tone, emotion, and other verbal characteristics to get the voice you want.
ElevenLabs AI voice cloning creates a digital replica of your voice by analyzing a sample of your verbal recordings.

Besides getting the emotion and nuances right, it’s also important to audit the AI voice’s pronunciation, especially for special names. If needed, create a pronunciation dictionary your voice model can reference.

Test latency, interruptions, accuracy, and safety before launch

Before launching, rigorously test your AI voice agent to ensure it performs reliably across different scenarios.

Latency test: An agent that takes more than 700 ms to respond will sound unnatural to customers. Depending on your setup, network speed, the LLM, voice models, and third-party APIs can affect latency. Ideally, keep conversational latency below 500 ms.
Interruption test: Check if the agent handles mid-conversation interruption. Specifically, the agent should drop output audio frames, stop TTS, and resume capturing the verbal input without losing context.
Turn-taking test: When the user stops speaking, the agent should take over naturally. Assess whether the agent takes over at the right time and whether it interrupts mid-speech. For a detailed guide, read how ElevenLabs handles turn-taking here.
Accuracy test: Any inaccuracy at the STT layer propagates downstream. Ensure the voice assistant can understand speech even in noisy environments. Also, the voice model must consistently transcribe speech in different accents accurately.
Safety test: Ensure guardrails prevent system behaviors that violate brand policy. If an interaction becomes emotionally charged, the agent must facilitate a warm handover to a human agent according to preset guidelines.

Measure outcomes and improve your voice assistant development

Passing internal testing criteria prepares your AI voice agent for real-world deployment. However, it is not the final goal in the development lifecycle. Post-deployment iteration is equally important to ensure your AI voice agent continues to resolve customer requests.

You use feedback from conversational AI metrics to identify gaps in your AI assistant. Then, gradually refine the application logic, API integrations, model choices, and other tech stack components to improve conversational AI performance.

Track containment, resolution, customer satisfaction, and failure modes

Continuously measure your AI assistant against industry benchmarks. Some of the metrics to track include:

Containment: Tracks how many conversations are successfully resolved by the AI agent.
Escalation rate: Measures calls that eventually require human intervention.
Word error rate: An accuracy benchmark indicating how precisely AI transcribes audio.
Hallucination rate: Highlights incidents where LLMs generate inaccurate but plausible answers.
CSAT: Customer feedback indicating how satisfied customers are with the AI voice agent.

Explore more about conversational AI metrics.

Get started with ElevenAPI for AI voice assistants

ElevenAPI simplifies AI voice assistant development. With leading Text to Speech and Speech to Text models, you can create context-aware conversational AI agents that fit right into your enterprise workflow. Our voice and audio models consistently deliver high accuracy and low latency, enabling a natural customer experience.

Get your ElevenAPI API key here and explore our official documentation to start building.