AssemblyAI

Speech-to-text service implementation using AssemblyAI’s real-time transcription API

Overview

AssemblyAISTTService provides real-time speech recognition using AssemblyAI’s WebSocket API with support for interim results, end-of-turn detection, and configurable audio processing parameters for accurate transcription in conversational AI applications.

AssemblyAI STT API Reference

Pipecat’s API methods for AssemblyAI STT integration

Example Implementation

Example with AssemblyAI built-in turn detection

Universal 3.5 Pro Realtime

Universal 3.5 Pro Realtime documentation and features

Universal 3.5 Pro Realtime API Reference

Complete Universal 3.5 Pro Realtime API reference

AssemblyAI Console

Access API keys and transcription features

Installation

To use AssemblyAI services, install the required dependency:

uv add "pipecat-ai[assemblyai]"

Prerequisites

AssemblyAI Account Setup

Before using AssemblyAI STT services, you need:

1. AssemblyAI Account: Sign up at AssemblyAI Console
2. API Key: Generate an API key from your dashboard
3. Configuration: Configure transcription settings and features for your use case

Required Environment Variables

• ASSEMBLYAI_API_KEY: Your AssemblyAI API key for authentication

Configuration

AssemblyAISTTService

api_key · str · required

AssemblyAI API key for authentication.

language · Language · default: "Language.EN" · deprecated

Language code for transcription. Deprecated in v0.0.105. Use settings=AssemblyAISTTService.Settings(...) instead.

api_endpoint_base_url · str · default: "wss://streaming.assemblyai.com/v3/ws"

WebSocket endpoint URL. Override for custom or proxied deployments.

sample_rate · int · default: "16000"

Audio sample rate in Hz.

encoding · str · default: "pcm_s16le"

Audio encoding format.

connection_params · AssemblyAIConnectionParams · default: "None" · deprecated

Connection configuration parameters. Deprecated in v0.0.105. Use settings=AssemblyAISTTService.Settings(...) instead. See Settings below.

vad_force_turn_endpoint · bool · default: "True"

Controls turn detection mode. When True (Pipecat mode, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. VAD stop sends ForceEndpoint as ceiling. No UserStarted/StoppedSpeakingFrame emitted from STT. When False (AssemblyAI turn detection mode): AssemblyAI’s model controls turn endings using built-in turn detection. Uses AssemblyAI API defaults for all parameters unless explicitly set. Emits UserStarted/StoppedSpeakingFrame from STT.

should_interrupt · bool · default: "True"

Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (vad_force_turn_endpoint=False). Only applies when using AssemblyAI’s built-in turn detection.

speaker_format · str | None · default: "None"

Optional format string for speaker labels when diarization is enabled. Use {speaker} for speaker label and {text} for transcript text. Example: "<{speaker}>{text}</{speaker}>" or "{speaker}: {text}". If None, transcript text is not modified.

settings · AssemblyAISTTService.Settings · default: "None"

Runtime-configurable settings for the STT service. See Settings below.

ttfs_p99_latency · float · default: "ASSEMBLYAI_TTFS_P99"

P99 latency from speech end to final transcript in seconds. Override for your deployment.

Settings

Runtime-configurable settings passed via the settings constructor argument using AssemblyAISTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.

Parameter	Type	Default	Description
model	str	"universal-3-5-pro"	STT model identifier. Set to "universal-3-5-pro". (Inherited from base STT settings.)
language	Language | str	Language.EN	Language for speech recognition. (Inherited from base STT settings.)
formatted_finals	bool	True	Whether to enable transcript formatting.
word_finalization_max_wait_time	int	None	Maximum time to wait for word finalization in milliseconds.
end_of_turn_confidence_threshold	float	None	Confidence threshold for end-of-turn detection.
min_turn_silence	int	None	Silence duration (ms) before a speculative end-of-turn check. If terminal punctuation is found, the turn ends; otherwise a partial is emitted and the turn continues. Clamped to 50–10000 ms. Server default is mode-dependent (set by the mode preset).
max_turn_silence	int	None	Maximum silence (ms) before the turn is forced to end, regardless of punctuation.
keyterms_prompt	List[str]	None	A list of words and phrases to improve recognition accuracy for. Maximum 100 terms.
prompt	str	None	A contextual prompt describing what the audio is about — its domain, scenario, or conversation details — so the model better recognizes likely vocabulary. Carries context about your audio, not transcription instructions; formatting or behavioral commands are not supported. Maximum ~1500 characters.
language_code	str	None	Steers transcription toward a specific language on a per-token basis. Pass a single ISO code to bias output toward that language; when unset, no steering is applied and the model code-switches natively across supported languages. Supported codes: en, es, fr, de, it, pt, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh. Distinct from language detection, which only controls whether the detected language is reported.
speaker_labels	bool	None	Whether to enable streaming speaker diarization. When enabled, each turn includes a speaker_label and each final word includes a speaker field for word-level attribution.
vad_threshold	float	None	Confidence threshold (0.0–1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent; increase for noisy environments to reduce false speech detection. Server default is mode-dependent (set by the mode preset).
domain	str	None	Enable a domain-specific model for specialized terminology. Set to "medical-v1" for Medical Mode (improved accuracy on medications, procedures, conditions, and dosages). Supported languages: en, es, de, fr.
continuous_partials	bool	True	Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled, partials covering the full turn transcript are emitted about every 3 seconds while speech continues; when disabled, only one early partial is emitted near turn start. The first partial (at 750 ms) is unaffected.
interruption_delay	int	None	How soon the first partial is emitted, in milliseconds (0–1000). Useful for tuning barge-in responsiveness or emitting earlier partials for LLM inference; larger values are more confident on interruptions, smaller values give faster time to first partial. The server adds a fixed 256 ms, so 0 yields an effective 256 ms and 500 yields 756 ms. Server default is mode-dependent (set by the mode preset).
agent_context	str	None	Your voice agent’s spoken text (TTS reply), used as context for the next user turn — improves accuracy on short or ambiguous replies and spelled-out entities like emails or IDs. Set at connection to seed the agent’s greeting and/or update after each reply; each update replaces the previous value. Maximum ~1500 characters.
previous_context_n_turns	int	None	Advanced. Maximum number of prior conversation entries (user transcripts and any agent_context values) carried forward as context. Range 0–100; set to 0 to disable automatic carryover. Most integrations should leave this unset.
voice_focus	Literal["near-field", "far-field"]	None	Enable Voice Focus to isolate the primary voice and suppress background noise before transcription. Set to "near-field" for close-talking mics (headsets, phones) or "far-field" for distant mics (conference rooms). Off when unset.
voice_focus_threshold	float	None	Controls how aggressively Voice Focus suppresses background audio, from 0.0 (least) to 1.0 (most). Requires voice_focus to be set, otherwise a validation error is returned.
mode	Literal["min_latency", "balanced", "max_accuracy"]	None	Latency and accuracy preset controlling turn-detection and partial-emission defaults. max_accuracy favors quality, min_latency favors speed, and balanced trades off between them. When omitted, the server applies its own preset, which sets the defaults for mode-dependent fields such as interruption_delay, min_turn_silence, vad_threshold, previous_context_n_turns, and continuous_partials.

Usage

Basic Setup

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
    ),
)

With Custom Settings

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        keyterms_prompt=["Pipecat", "AssemblyAI"],
    ),
    vad_force_turn_endpoint=True,
)

With AssemblyAI Built-in Turn Detection

AssemblyAI’s universal-3-5-pro model supports built-in turn detection for more natural conversation flow. When vad_force_turn_endpoint=False, the service automatically requests ExternalUserTurnStrategies, so you don’t need to configure turn strategies manually:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    vad_force_turn_endpoint=False,  # Use AssemblyAI's built-in turn detection
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        # Optional: Tune turn detection timing
        min_turn_silence=100,  # Silence duration (ms) before a speculative end-of-turn check.
        max_turn_silence=1000,  # Maximum silence (ms) before forcing end-of-turn
    ),
)

With Speaker Diarization

Enable speaker identification for multi-party conversations:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        speaker_labels=True,  # Enable speaker diarization
    ),
    speaker_format="{speaker}: {text}",  # Format transcripts with speaker labels
)

With Context Carryover

Context carryover improves transcription by giving the model memory of recent conversation turns. It’s enabled by default for universal-3-5-pro — the service automatically feeds each of the agent’s completed replies to AssemblyAI as carryover context, so no configuration is required. The example below only tunes how many prior turns are retained:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        previous_context_n_turns=4,  # Keep last 4 turns (if None, server default is 3. Set to 0 to disable.)
    ),
)

Context carryover helps with short answers, spelled-out entities (emails, IDs), and similar-sounding words. Set previous_context_n_turns=0 to disable automatic carryover.

With Voice Focus

Voice focus isolates the primary speaker and suppresses background noise:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        voice_focus="near-field",  # Or "far-field" for distant capture
        voice_focus_threshold=0.7,  # 0.0-1.0, higher suppresses more
    ),
)

Use "near-field" for close-talking mics (headsets, handsets) or "far-field" for distant capture (conference rooms, laptop mics).

Methods

update_agent_context()

async def update_agent_context(text: str)

Send the agent’s latest spoken reply to AssemblyAI as carryover context. Improves transcription of the user’s next turn — short answers, spelled-out entities, disambiguation.

Parameters:

• text (str): The agent’s spoken reply text. Clipped to ~1500 characters.

Example:

# After the agent finishes speaking
await stt.update_agent_context("What is your order ID?")

It is automatically invoked each time an assistant turn is completed.

Notes

• Model: Use universal-3-5-pro, AssemblyAI’s flagship streaming model. It supports built-in turn detection, prompting, continuous partials, context carryover, and voice focus.
• Turn detection modes:
  • Pipecat mode (vad_force_turn_endpoint=True, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. The service sends a ForceEndpoint message when VAD detects the user has stopped speaking.
  • AssemblyAI mode (vad_force_turn_endpoint=False, universal-3-5-pro only): AssemblyAI’s model controls turn endings using built-in turn detection. The service emits UserStartedSpeakingFrame and UserStoppedSpeakingFrame and automatically requests ExternalUserTurnStrategies, so you don’t need to configure turn strategies manually. Pass your own user_turn_strategies only to override this.
• Context carryover (universal-3-5-pro only): Seed the agent’s most recent reply, improving transcription of the user’s next turn — short answers, spelled-out entities, disambiguation. update_agent_context() is automatically invoked each time an assistant turn is completed. Control the window size with previous_context_n_turns (0–100, default 3); set to 0 to disable carryover entirely.
• Voice focus (universal-3-5-pro only): Set voice_focus to "near-field" or "far-field" to isolate the primary voice and suppress background noise. Tune suppression strength with voice_focus_threshold (0.0–1.0, higher values suppress more).
• Speaker diarization: Enable speaker_labels=True in Settings to automatically identify different speakers. Final transcripts will include a speaker field (e.g., “Speaker A”, “Speaker B”). Use the speaker_format parameter to format transcripts with speaker labels.
• Prompting: The prompt parameter allows you to guide transcription for specific names, terms, or domain vocabulary. AssemblyAI recommends testing without a prompt first.
• Dynamic settings updates: Most settings can be updated at runtime using STTUpdateSettingsFrame. agent_context is hot-updatable without reconnecting; other settings require a reconnect.

The connection_params= / InputParams / params= pattern is deprecated as of v0.0.105. Use Settings / settings= instead. See the Service Settings guide for migration details.

Event Handlers

AssemblyAI STT supports the standard service connection events, plus turn-level events for conversation tracking:

Event	Description
on_connected	Connected to AssemblyAI WebSocket
on_disconnected	Disconnected from AssemblyAI WebSocket
on_end_of_turn	End of turn detected (fires after final transcript is pushed)

@stt.event_handler("on_connected")
async def on_connected(service):
    print("Connected to AssemblyAI")

@stt.event_handler("on_end_of_turn")
async def on_end_of_turn(service, transcript):
    print(f"Turn ended: {transcript}")

The on_end_of_turn event receives (service, transcript) where transcript is the final transcript text. This event fires after the final transcript is pushed, providing a reliable hook for end-of-turn logic that doesn’t race with TranscriptionFrame. Works in both Pipecat and AssemblyAI turn detection modes.