> ## Documentation Index
> Fetch the complete documentation index at: https://daily-docs-pr-4892.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# AssemblyAI
> Speech-to-text service implementation using AssemblyAI's real-time transcription API
## Overview
`AssemblyAISTTService` provides real-time speech recognition using AssemblyAI's WebSocket API with support for interim results, end-of-turn detection, and configurable audio processing parameters for accurate transcription in conversational AI applications.
Pipecat's API methods for AssemblyAI STT integration
Example with AssemblyAI built-in turn detection
Universal 3.5 Pro Realtime documentation and features
Complete Universal 3.5 Pro Realtime API reference
Access API keys and transcription features
## Installation
To use AssemblyAI services, install the required dependency:
```bash theme={null}
uv add "pipecat-ai[assemblyai]"
```
## Prerequisites
### AssemblyAI Account Setup
Before using AssemblyAI STT services, you need:
1. **AssemblyAI Account**: Sign up at [AssemblyAI Console](https://www.assemblyai.com/dashboard/signup)
2. **API Key**: Generate an API key from your dashboard
3. **Configuration**: Configure transcription settings and features for your use case
### Required Environment Variables
* `ASSEMBLYAI_API_KEY`: Your AssemblyAI API key for authentication
## Configuration
### AssemblyAISTTService
AssemblyAI API key for authentication.
Language code for transcription. *Deprecated in v0.0.105. Use
`settings=AssemblyAISTTService.Settings(...)` instead.*
WebSocket endpoint URL. Override for custom or proxied deployments.
Audio sample rate in Hz.
Audio encoding format.
Connection configuration parameters. *Deprecated in v0.0.105. Use
`settings=AssemblyAISTTService.Settings(...)` instead. See
[Settings](#settings) below.*
Controls turn detection mode. When `True` (Pipecat mode, default): Forces
AssemblyAI to return finals ASAP so Pipecat's turn detection (e.g., Smart
Turn) decides when the user is done. VAD stop sends ForceEndpoint as ceiling.
No UserStarted/StoppedSpeakingFrame emitted from STT. When `False` (AssemblyAI
turn detection mode): AssemblyAI's model controls turn endings using built-in
turn detection. Uses AssemblyAI API defaults for all parameters unless
explicitly set. Emits UserStarted/StoppedSpeakingFrame from STT.
Whether to interrupt the bot when the user starts speaking in AssemblyAI turn
detection mode (`vad_force_turn_endpoint=False`). Only applies when using
AssemblyAI's built-in turn detection.
Optional format string for speaker labels when diarization is enabled. Use
`{speaker}` for speaker label and `{text}` for transcript text. Example:
`"<{speaker}>{text}"` or `"{speaker}: {text}"`. If None, transcript
text is not modified.
Runtime-configurable settings for the STT service. See [Settings](#settings)
below.
P99 latency from speech end to final transcript in seconds. Override for your
deployment.
### Settings
Runtime-configurable settings passed via the `settings` constructor argument using `AssemblyAISTTService.Settings(...)`. These can be updated mid-conversation with `STTUpdateSettingsFrame`. See [Service Settings](/pipecat/fundamentals/service-settings) for details.
| Parameter | Type | Default | Description |
| ---------------------------------- | ---------------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model` | `str` | `"universal-3-5-pro"` | STT model identifier. Set to `"universal-3-5-pro"`. *(Inherited from base STT settings.)* |
| `language` | `Language \| str` | `Language.EN` | Language for speech recognition. *(Inherited from base STT settings.)* |
| `formatted_finals` | `bool` | `True` | Whether to enable transcript formatting. |
| `word_finalization_max_wait_time` | `int` | `None` | Maximum time to wait for word finalization in milliseconds. |
| `end_of_turn_confidence_threshold` | `float` | `None` | Confidence threshold for end-of-turn detection. |
| `min_turn_silence` | `int` | `None` | Silence duration (ms) before a speculative end-of-turn check. If terminal punctuation is found, the turn ends; otherwise a partial is emitted and the turn continues. Clamped to 50–10000 ms. Server default is mode-dependent (set by the `mode` preset). |
| `max_turn_silence` | `int` | `None` | Maximum silence (ms) before the turn is forced to end, regardless of punctuation. |
| `keyterms_prompt` | `List[str]` | `None` | A list of words and phrases to improve recognition accuracy for. Maximum 100 terms. |
| `prompt` | `str` | `None` | A contextual prompt describing what the audio is about — its domain, scenario, or conversation details — so the model better recognizes likely vocabulary. Carries context about your audio, not transcription instructions; formatting or behavioral commands are not supported. Maximum \~1500 characters. |
| `language_code` | `str` | `None` | Steers transcription toward a specific language on a per-token basis. Pass a single ISO code to bias output toward that language; when unset, no steering is applied and the model code-switches natively across supported languages. Supported codes: `en`, `es`, `fr`, `de`, `it`, `pt`, `tr`, `nl`, `sv`, `no`, `da`, `fi`, `hi`, `vi`, `ar`, `he`, `ja`, `zh`. Distinct from language detection, which only controls whether the detected language is reported. |
| `speaker_labels` | `bool` | `None` | Whether to enable streaming speaker diarization. When enabled, each turn includes a `speaker_label` and each final word includes a `speaker` field for word-level attribution. |
| `vad_threshold` | `float` | `None` | Confidence threshold (0.0–1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent; increase for noisy environments to reduce false speech detection. Server default is mode-dependent (set by the `mode` preset). |
| `domain` | `str` | `None` | Enable a domain-specific model for specialized terminology. Set to `"medical-v1"` for Medical Mode (improved accuracy on medications, procedures, conditions, and dosages). Supported languages: `en`, `es`, `de`, `fr`. |
| `continuous_partials` | `bool` | `True` | Whether to emit additional partial transcripts during long turns at a steady \~3 second cadence. When enabled, partials covering the full turn transcript are emitted about every 3 seconds while speech continues; when disabled, only one early partial is emitted near turn start. The first partial (at 750 ms) is unaffected. |
| `interruption_delay` | `int` | `None` | How soon the first partial is emitted, in milliseconds (0–1000). Useful for tuning barge-in responsiveness or emitting earlier partials for LLM inference; larger values are more confident on interruptions, smaller values give faster time to first partial. The server adds a fixed 256 ms, so `0` yields an effective 256 ms and `500` yields 756 ms. Server default is mode-dependent (set by the `mode` preset). |
| `agent_context` | `str` | `None` | Your voice agent's spoken text (TTS reply), used as context for the next user turn — improves accuracy on short or ambiguous replies and spelled-out entities like emails or IDs. Set at connection to seed the agent's greeting and/or update after each reply; each update replaces the previous value. Maximum \~1500 characters. |
| `previous_context_n_turns` | `int` | `None` | Advanced. Maximum number of prior conversation entries (user transcripts and any `agent_context` values) carried forward as context. Range 0–100; set to `0` to disable automatic carryover. Most integrations should leave this unset. |
| `voice_focus` | `Literal["near-field", "far-field"]` | `None` | Enable Voice Focus to isolate the primary voice and suppress background noise before transcription. Set to `"near-field"` for close-talking mics (headsets, phones) or `"far-field"` for distant mics (conference rooms). Off when unset. |
| `voice_focus_threshold` | `float` | `None` | Controls how aggressively Voice Focus suppresses background audio, from `0.0` (least) to `1.0` (most). Requires `voice_focus` to be set, otherwise a validation error is returned. |
| `mode` | `Literal["min_latency", "balanced", "max_accuracy"]` | `None` | Latency and accuracy preset controlling turn-detection and partial-emission defaults. `max_accuracy` favors quality, `min_latency` favors speed, and `balanced` trades off between them. When omitted, the server applies its own preset, which sets the defaults for mode-dependent fields such as `interruption_delay`, `min_turn_silence`, `vad_threshold`, `previous_context_n_turns`, and `continuous_partials`. |
## Usage
### Basic Setup
```python theme={null}
from pipecat.services.assemblyai.stt import AssemblyAISTTService
stt = AssemblyAISTTService(
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
settings=AssemblyAISTTService.Settings(
model="universal-3-5-pro",
),
)
```
### With Custom Settings
```python theme={null}
from pipecat.services.assemblyai.stt import AssemblyAISTTService
stt = AssemblyAISTTService(
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
settings=AssemblyAISTTService.Settings(
model="universal-3-5-pro",
keyterms_prompt=["Pipecat", "AssemblyAI"],
),
vad_force_turn_endpoint=True,
)
```
### With AssemblyAI Built-in Turn Detection
AssemblyAI's `universal-3-5-pro` model supports built-in turn detection for more natural conversation flow. When `vad_force_turn_endpoint=False`, the service automatically requests `ExternalUserTurnStrategies`, so you don't need to configure turn strategies manually:
```python theme={null}
from pipecat.services.assemblyai.stt import AssemblyAISTTService
stt = AssemblyAISTTService(
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
vad_force_turn_endpoint=False, # Use AssemblyAI's built-in turn detection
settings=AssemblyAISTTService.Settings(
model="universal-3-5-pro",
# Optional: Tune turn detection timing
min_turn_silence=100, # Silence duration (ms) before a speculative end-of-turn check.
max_turn_silence=1000, # Maximum silence (ms) before forcing end-of-turn
),
)
```
### With Speaker Diarization
Enable speaker identification for multi-party conversations:
```python theme={null}
from pipecat.services.assemblyai.stt import AssemblyAISTTService
stt = AssemblyAISTTService(
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
settings=AssemblyAISTTService.Settings(
model="universal-3-5-pro",
speaker_labels=True, # Enable speaker diarization
),
speaker_format="{speaker}: {text}", # Format transcripts with speaker labels
)
```
### With Context Carryover
Context carryover improves transcription by giving the model memory of recent conversation turns. **It's enabled by default** for `universal-3-5-pro` — the service automatically feeds each of the agent's completed replies to AssemblyAI as carryover context, so no configuration is required. The example below only tunes how many prior turns are retained:
```python theme={null}
from pipecat.services.assemblyai.stt import AssemblyAISTTService
stt = AssemblyAISTTService(
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
settings=AssemblyAISTTService.Settings(
model="universal-3-5-pro",
previous_context_n_turns=4, # Keep last 4 turns (if None, server default is 3. Set to 0 to disable.)
),
)
```
Context carryover helps with short answers, spelled-out entities (emails, IDs), and similar-sounding words. Set `previous_context_n_turns=0` to disable automatic carryover.
### With Voice Focus
Voice focus isolates the primary speaker and suppresses background noise:
```python theme={null}
from pipecat.services.assemblyai.stt import AssemblyAISTTService
stt = AssemblyAISTTService(
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
settings=AssemblyAISTTService.Settings(
model="universal-3-5-pro",
voice_focus="near-field", # Or "far-field" for distant capture
voice_focus_threshold=0.7, # 0.0-1.0, higher suppresses more
),
)
```
Use `"near-field"` for close-talking mics (headsets, handsets) or `"far-field"` for distant capture (conference rooms, laptop mics).
## Methods
### update\_agent\_context()
```python theme={null}
async def update_agent_context(text: str)
```
Send the agent's latest spoken reply to AssemblyAI as carryover context. Improves transcription of the user's next turn — short answers, spelled-out entities, disambiguation.
**Parameters:**
* `text` (str): The agent's spoken reply text. Clipped to \~1500 characters.
**Example:**
```python theme={null}
# After the agent finishes speaking
await stt.update_agent_context("What is your order ID?")
```
> It is automatically invoked each time an assistant turn is completed.
## Notes
* **Model**: Use `universal-3-5-pro`, AssemblyAI's flagship streaming model. It supports built-in turn detection, prompting, continuous partials, context carryover, and voice focus.
* **Turn detection modes**:
* **Pipecat mode** (`vad_force_turn_endpoint=True`, default): Forces AssemblyAI to return finals ASAP so Pipecat's turn detection (e.g., Smart Turn) decides when the user is done. The service sends a `ForceEndpoint` message when VAD detects the user has stopped speaking.
* **AssemblyAI mode** (`vad_force_turn_endpoint=False`, `universal-3-5-pro` only): AssemblyAI's model controls turn endings using built-in turn detection. The service emits `UserStartedSpeakingFrame` and `UserStoppedSpeakingFrame` and automatically requests `ExternalUserTurnStrategies`, so you don't need to configure turn strategies manually. Pass your own `user_turn_strategies` only to override this.
* **Context carryover** (`universal-3-5-pro` only): Seed the agent's most recent reply, improving transcription of the user's next turn — short answers, spelled-out entities, disambiguation. `update_agent_context()` is automatically invoked each time an assistant turn is completed. Control the window size with `previous_context_n_turns` (0–100, default 3); set to 0 to disable carryover entirely.
* **Voice focus** (`universal-3-5-pro` only): Set `voice_focus` to `"near-field"` or `"far-field"` to isolate the primary voice and suppress background noise. Tune suppression strength with `voice_focus_threshold` (0.0–1.0, higher values suppress more).
* **Speaker diarization**: Enable `speaker_labels=True` in Settings to automatically identify different speakers. Final transcripts will include a speaker field (e.g., "Speaker A", "Speaker B"). Use the `speaker_format` parameter to format transcripts with speaker labels.
* **Prompting**: The `prompt` parameter allows you to guide transcription for specific names, terms, or domain vocabulary. AssemblyAI recommends testing without a prompt first.
* **Dynamic settings updates**: Most settings can be updated at runtime using `STTUpdateSettingsFrame`. `agent_context` is hot-updatable without reconnecting; other settings require a reconnect.
The `connection_params=` / `InputParams` / `params=` pattern is deprecated as
of v0.0.105. Use `Settings` / `settings=` instead. See the [Service Settings
guide](/pipecat/fundamentals/service-settings) for migration details.
## Event Handlers
AssemblyAI STT supports the standard [service connection events](/api-reference/server/events/service-events), plus turn-level events for conversation tracking:
| Event | Description |
| ----------------- | ------------------------------------------------------------- |
| `on_connected` | Connected to AssemblyAI WebSocket |
| `on_disconnected` | Disconnected from AssemblyAI WebSocket |
| `on_end_of_turn` | End of turn detected (fires after final transcript is pushed) |
```python theme={null}
@stt.event_handler("on_connected")
async def on_connected(service):
print("Connected to AssemblyAI")
@stt.event_handler("on_end_of_turn")
async def on_end_of_turn(service, transcript):
print(f"Turn ended: {transcript}")
```
The `on_end_of_turn` event receives `(service, transcript)` where `transcript` is the final transcript text. This event fires after the final transcript is pushed, providing a reliable hook for end-of-turn logic that doesn't race with `TranscriptionFrame`. Works in both Pipecat and AssemblyAI turn detection modes.