Skip to main content
Voice Agents

Voice Agent

Real-time voice agents over WebRTC with one selected speech-to-speech or STT-to-LLM-to-TTS stack.

Overview

The Voice Agent streams conversations over WebRTC. Choose one configuration for the call:

  • CallMissed-managed pipeline: select one speech-recognition model, one language model, and one speech-generation model and voice.

  • Deepgram-managed pipeline: select a deepgram-voice-* model and its supported recognition and voice settings.

  • Native speech-to-speech: select a GPT Realtime model and voice.

  • Full-duplex speech-to-speech: select gpt-live-1, which listens and speaks at the same time. It behaves differently enough that it has its own page.

An omitted llm_model selects gemma-4-31b on the STT → LLM → TTS pipeline, with thinking off for the fastest replies. If the selected stack cannot start, the call runs on a backup stack instead of failing, and usage records the model that actually served it. If the model stops answering mid-call (an outage or a rate limit), the turn is answered by a backup model instead: sarvam-105b, then gpt-4o. When the agent's model is one of the India-hosted voice LLMs (gemma-4-31b, sarvam-105b and the rest of the india_hosting list in GET /api/v1/bots/config-schema), gemma-4-31b on a second host outside India is also tried: first for gemma-4-31b agents, otherwise just before gpt-4o. A turn sarvam-105b answered is billed at its own rate; a turn gpt-4o or the second gemma-4-31b host answered is billed at your agent's model rate, and usage still shows the model that answered. On an agent with hosting: "india", the second host is never used, and gpt-4o is used only while it is processed in India. Retired voice_fallbacks settings are no longer used.

Architecture

You create a session over REST and receive a connection URL + token. Your client connects with the livekit-client SDK; the CallMissed voice agent joins automatically and handles the speech pipeline. Audio flows over WebRTC.

This is the WebRTC path. For a plain WebSocket you stream raw audio to — no client SDK, no media hop — see the Managed Voice Agent, which runs the same tuned pipeline over wss://api.callmissed.com.

  1. 1

    Browser (livekit-client SDK)

    Captures mic audio and streams it over WebRTC

  2. 2

    Connection

    WebRTC transport that connects your client to the voice agent

  3. 3

    CallMissed voice agent

    Runs the selected speech-to-speech model or STT → LLM → TTS pipeline

  4. 4

    Browser

    Receives synthesized speech back over WebRTC and plays it

One conversational turn

With a GPT Realtime model selected, every turn stays in one speech-to-speech model. With a cascaded model selected (or when the speech-to-speech model is unavailable), every turn streams through STT, LLM, and TTS concurrently to minimize time-to-first-audio:

  1. 1

    STT

    Streams partial transcripts as the user speaks, finalizes on end-of-speech

  2. 2

    LLM

    Generates the reply at high throughput and pushes sentence chunks downstream

  3. 3

    TTS

    Synthesizes each sentence chunk as it arrives — playback starts before generation finishes

Quickstart

1. Create a session:

curl -X POST https://api.callmissed.com/v1/voice/sessions \
  -H "Authorization: Bearer cm_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "system_prompt": "You are a helpful assistant.",
    "voice": "shubh",
    "language": "en-IN",
    "llm_model": "kimi-k2.5"
  }'

Response:

{
  "id": "uuid",
  "ws_url": "wss://…",
  "token": "eyJhbGciOi...",
  "status": "created"
}

Read ws_url from this response and pass it straight to the client — it is issued per session. Do not hardcode it.

2. Connect with the client SDK:

import { Room, RoomEvent, Track } from "livekit-client";

const room = new Room();

room.on(RoomEvent.TrackSubscribed, (track, pub, participant) => {
  if (track.kind === Track.Kind.Audio) {
    const el = track.attach();
    document.body.appendChild(el);
  }
});

room.on(RoomEvent.TranscriptionReceived, (segments, participant) => {
  for (const seg of segments) {
    if (seg.final) {
      const who = participant?.isLocal ? "You" : "Agent";
      console.log(who + ": " + seg.text);
    }
  }
});

await room.connect(session.ws_url, session.token);
await room.localParticipant.setMicrophoneEnabled(true);

The agent joins automatically, greets the user, and responds to speech.

Configuration

FieldTypeDefaultDescription
system_promptstring"You are a helpful voice assistant..."System prompt for LLM
voicestringshubhA speaker of the selected TTS model (bulbul:v3 has 37)
languagestringen-INLanguage code for STT and TTS
llm_modelstringgemma-4-31bOne voice-capable model. An id the agent cannot serve is rejected at create time with 422. deepgram-think-fast and deepgram-phonellm-mini (early access) are processed outside India, so an agent with hosting: "india" cannot use them; see Models
tts_providerstringplan-dependentLegacy TTS selector; use tts_model for per-model selection. Omit it and the server picks by plan: paid plans (starter, pro, enterprise, scale, business, corporate) default to Cartesia sonic-3.6, the free plan to Sarvam bulbul:v3. An explicit value is always honoured.
max_duration_secondsint1800Max session duration (30-3600)

stt_model, tts_model, greeting, variables, bot_id and the rest are covered in the Voice Session API reference.

Backup voices and spoken numbers

Two agent settings, saved in the agent's config with PATCH /api/v1/bots/{bot_id}/config (or the console's Speaking style page). They apply to agents that choose their own models; an agent on a voice tier keeps the tier's voices.

tts_fallbacks

Up to 3 voices to speak through, in order, if the agent's own voice stops working. They are tried before the platform's own backup voice, both when the call starts and in the middle of a call.

{
  "values": {
    "tts_fallbacks": [
      { "tts_model": "sonic-3.6", "voice": "katie" },
      { "tts_model": "bulbul:v3", "voice": "priya" }
    ]
  }
}
  • tts_model is any voice model from the agent's speech models (the tts_model values in GET /api/v1/bots/config-schema). voice is optional; left out, the model's default voice speaks.
  • Each entry must speak the agent's language (for language: "multi", the opening language), and voice must be one of that model's voices. Anything else is refused with 422, and so is a later language change that a saved backup cannot speak.
  • Billing follows the backup voice that spoke. If the agent's voice could not start and a backup voice served the call, the session's metadata.stack_substitution names it as served_tts_model; if the switch happens mid-call, each turn it speaks is billed at that backup voice's rate. When the platform's own backup voice speaks instead, it is billed at your voice's rate.

tts_normalize

true reads numbers the way a person says them: amounts with Indian grouping (₹1,50,000 → "one lakh fifty thousand rupees"), $ amounts, dates (12/03/2026 is read day first), times, percentages, ordinals, ranges and phone numbers (digit by digit). Long digit runs and numbers starting with 0 (order IDs, one-time codes) are read digit by digit.

  • English text gets English words; Hindi text in Devanagari gets Hindi words in Devanagari ("एक लाख पचास हज़ार रुपये").
  • Links, email addresses and digits joined to letters (24x7, 4G) are left as written. The saved transcript keeps the original digits.
  • Off by default. It has no effect on sonic-3.6, bulbul:v3 and gnani-timbre-v3.0, which already read numbers aloud themselves.
  • sonic-3.6 is given the agent's regional language (for example en-IN or hi-IN, not just en or hi), so it reads amounts and dates the way that region does.

Expressive audio tags on Gnani Timbre v3

When an agent speaks with gnani-timbre-v3.0, the agent adds a few audio tags to its own replies: pauses, a sigh, an exhale, a laugh or giggle, or an excited tone. It uses them sparingly, and never when the call is about money owed, a complaint, health or bad news. Other voices never receive these tags (ElevenLabs has its own, below), and you do not need to write them into your prompt.

You control this per agent, in its config (with PATCH /api/v1/bots/{bot_id}/config) or on the console's Speech page:

KeyTypeDefaultWhat it does
tts_audio_tagsbooleantruefalse turns the sounds off: the agent writes plain replies and any tag is removed before it is spoken.
tts_audio_tag_setarray of tag names["pauses", "sigh", "exhales", "laugh", "giggle", "excited"]The sounds the agent may use, from the 13 tags above. Only these are taught to the agent, and any other tag is removed before it is spoken. [] means none.
tts_audio_tag_frequencystring"normal"How often a sound may be used: "rare" (at most one reply in five, one sound), "normal" (one reply in three, up to two sounds) or "often" (any reply, up to three sounds). Enforced on every reply: a sound over the limit is removed and its words are kept. Tones (excited, slow, whispers, shouting) are counted separately from sounds, so one never uses up the other. pauses is silence: capped at the same number per reply, with no rest replies.
tts_speednumber1.0Speaking pace, 0.85–1.15.
{ "values": { "tts_audio_tag_set": ["pauses", "sigh"], "tts_audio_tag_frequency": "rare", "tts_speed": 0.95 } }

Expressive delivery on ElevenLabs

Each ElevenLabs model is taught only what it can perform:

  • eleven_v4, eleven_v4_turbo, eleven_v3, eleven_v3_conversational: the agent adds a few audio tags in square brackets, such as [sighs], [chuckles] or [short pause], with the same restraint as above. Any bracketed cue you have not allowed is removed before it is spoken. For a live call, eleven_v4_turbo and eleven_v3_conversational are the fast, lower-priced choices.
  • eleven_flash_v2_5, eleven_turbo_v2_5, eleven_multilingual_v2, eleven_flash_v2, eleven_turbo_v2: these read bracketed tags aloud, so the agent never gets them and any that appear are removed. Instead, it may pause with <break time="0.5s" />, at most once a reply.

The same keys control it:

KeyTypeDefaultWhat it does
tts_audio_tagsbooleantruefalse turns the sounds (v3/v4) or pauses (the other models) off.
tts_audio_tag_setarray of tag names["short pause", "sighs", "exhales", "chuckles", "laughs", "curious", "excited"]v3/v4 only. The tags the agent may use, from: short pause, long pause, sighs, exhales, inhales deeply, chuckles, laughs, starts laughing, laughs harder, clears throat, curious, excited, happy, sad, whispers, sarcastic, mischievously, appalled, crying, wheezing, snorts. [] means none.
tts_audio_tag_frequencystring"normal"v3/v4 only. Same levels and enforcement as above; tones (curious, excited, happy, sad, whispers, sarcastic, mischievously, appalled, crying) are counted apart from sounds, and short pause / long pause never rest.
tts_speednumber1.0Flash, Turbo and Multilingual v2 only. Speaking pace, 0.7–1.2.

Turn-taking and speech-to-speech tuning

Advanced agent settings, saved in the agent's config with PATCH /api/v1/bots/{bot_id}/config (or the console's Voice page). Leave them unset unless calls cut people off or feel slow; an unset key keeps the default shown. A number outside its range is saved with a warning in the response and used as the nearest end of the range on calls; a value the key does not list is ignored on calls.

Turn-taking (agents with separate listening, thinking and speaking models)

KeyValuesDefaultWhat it does
min_endpointing_delay0.2–1.5 s0.3How long the agent waits after the caller stops before replying.
stt_fast_endpointingtrue, falsefalsesaaras:v4 listening model only. Notices the end of the caller's speech after about 64 ms of silence instead of about 320 ms, so the reply starts roughly a quarter of a second sooner. May cut in on callers who pause mid-sentence.
tool_wait_phraseText, up to 200 charactersunsetA short line the agent says when it starts one of the tool_wait_tools before saying anything else, for example "One moment, I'm sending that now." It plays while the tool runs. Write it in the agent's language. Used only when tool_wait_tools is also set.
tool_wait_toolsList of tool namesunsetWhich tools make the agent say tool_wait_phrase, for example send_template_message or recall_memory.
keep_send_promisestrue, falsetrueWhen the agent tells the caller it is sending something on WhatsApp but calls no send tool, its next reply must make the send. false: the promise is only recorded in the session's metadata.message_check.
min_interruption_words0–51How many words the caller must say before the agent stops talking. 0 stops it at any sound. 2 or more lets short replies like "okay" pass without cutting the agent off, but the caller must say more before the agent yields. On a listening model that returns text only after the caller finishes a phrase (for example saaras:v4), the count is measured as speech time, about 0.4 seconds of the caller's voice per word. Has no effect when allow_interruptions is false.
endpointing_modefixed, dynamicfixedfixed waits the same pause every turn. dynamic learns how long this caller pauses mid-thought and waits that long, between min_endpointing_delay and max_endpointing_delay.
max_endpointing_delay1.0–6.0 s3.0 (2.5 with audio_turn_detector)The longest the agent waits when the caller sounds unfinished. It is the ceiling for dynamic and for audio_turn_detector; with neither, the agent never waits this long.
audio_turn_detectortrue, falsefalseAt each pause, listens to how the caller sounds, not just how long the pause is, and waits longer when they sound unfinished. Only for agents whose language is English or Hindi. Not used with Flux or ink-2 listening models, which do this themselves.
flux_eot_threshold0.5–0.90.7Flux listening models only. How sure the model must be that the caller finished. Higher waits longer and cuts in less.
flux_eager_eot_threshold0.3–0.90.4Flux listening models only. The agent starts preparing its reply once the model is this sure. Lower is faster. Never above flux_eot_threshold: a higher value is lowered to it.

Speech-to-speech models (gpt-realtime family)

For gpt-realtime, gpt-realtime-mini, gpt-realtime-1.5, gpt-realtime-2, gpt-realtime-2.1 and gpt-realtime-2.1-mini. The model decides when the caller has finished, so the turn-taking keys above do not apply.

KeyValuesDefaultWhat it does
realtime_vadserver_vad, semantic_vadserver_vadserver_vad replies after a set pause. semantic_vad replies when the caller's words sound finished.
realtime_eagernesslow, medium, high, autoautoWith semantic_vad only. low waits longest (up to 8 s), high replies fastest, auto is medium.
realtime_vad_threshold0.0–1.00.5With server_vad only. How loud the caller must be to count as speech. Raise it on noisy lines.
realtime_silence_ms200–2000 ms200With server_vad only. The pause before the agent replies.
realtime_noise_reductionnear_field, far_fieldoffCleans the caller's audio before the model hears it. near_field for a phone held to the ear or a headset, far_field for a speakerphone or laptop.
realtime_speed0.25–1.51.0Speaking pace.

Setting realtime_vad_threshold or realtime_silence_ms without realtime_vad uses server_vad. The caller can always interrupt a speech-to-speech agent.

For gpt-live-1 agents with at least one function tool:

KeyValuesDefaultWhat it does
gpt_live_delegation_recoverytrue, falsefalseWhen the model hears a caller's request or answer that needs a tool but never hands it to its backend, check the turn and start the backend work. Each check is billed as a short gpt-6-luna request. See Full-duplex voice.
{
  "values": {
    "realtime_vad": "semantic_vad",
    "realtime_eagerness": "low",
    "realtime_noise_reduction": "near_field"
  }
}

Features

  • Interruption handling — speak while the agent is talking and it stops immediately, listens to you
  • Server-side turn detection — voice-activity detection finds where speech starts and ends; no client-side VAD needed
  • Noise filtering (coming soon) — the noise_suppression agent setting will filter steady background noise such as fans, traffic and line hiss out of the caller's audio. Not available yet: we're testing it on real calls first, and setting it has no effect today
  • Preemptive generation — LLM starts generating before STT fully confirms the transcript
  • Streaming pipeline — each stage streams to the next, no buffering between stages
  • Session management — REST API for creating, listing, deleting sessions and retrieving transcripts
  • Per-model pricing — each model is billed at its own catalogue rate (see the model catalogue), and GET /v1/voice/sessions/{id}/cost itemises a call. A call on your own stack also pays a flat ₹1 a minute platform fee, charged once when the call ends (see Voice Agent Plans); a call on a plan does not
  • Tool calling — built-in tools, your own REST endpoints and MCP servers, mid-conversation. See Voice Agent Tools

On this page