> ## Documentation Index
> Fetch the complete documentation index at: https://daily-docs-pr-4892.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# SmolVLM

> Local vision service using HuggingFace SmolVLM models for on-device image description

export const CommunityMaintained = ({maintainer, maintainerUrl, repo}) => <Note>
    <strong>Community-maintained integration.</strong> This service is built and
    maintained by{" "}
    <a href={maintainerUrl} target="_blank" rel="noreferrer">
      {maintainer}
    </a>
    . Pipecat does not test or officially support it. Please report issues and
    request changes on the{" "}
    <a href={repo} target="_blank" rel="noreferrer">
      source repository
    </a>
    . Learn more about{" "}
    <a href="/api-reference/server/services/community-integrations">
      community integrations
    </a>
    .
  </Note>;

<CommunityMaintained maintainer="rahulsolanki001" maintainerUrl="https://github.com/rahulsolanki001" repo="https://github.com/rahulsolanki001/pipecat-smolvlm" />

## Overview

`SmolVlmService` runs HuggingFace [SmolVLM](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct)
vision-language models locally to describe images flowing through a Pipecat
pipeline. The compact, instruction-tuned models run entirely on your own
hardware — no external API calls, no per-image billing, and no data leaving your
infrastructure. It accepts `UserImageRawFrame` input and emits a
`VisionTextFrame` containing the generated description.

<CardGroup cols={2}>
  <Card title="Source Repository" icon="github" href="https://github.com/rahulsolanki001/pipecat-smolvlm">
    Source code, examples, and issues for the SmolVLM integration
  </Card>

  <Card title="PyPI Package" icon="cube" href="https://pypi.org/project/pipecat-smolvlm/">
    The `pipecat-smolvlm` package on PyPI
  </Card>

  <Card title="SmolVLM Models" icon="image" href="https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct">
    Browse the SmolVLM checkpoints on HuggingFace
  </Card>
</CardGroup>

## Installation

This is a community-maintained package distributed separately from `pipecat-ai`:

```bash theme={null}
# Core install
pip install pipecat-smolvlm

# With Flash Attention 2 (CUDA only — significant speedup)
pip install "pipecat-smolvlm[flash-attn]"

# Intel XPU support
pip install "pipecat-smolvlm[xpu]"
```

Model weights are downloaded from HuggingFace on first use and cached in
`~/.cache/huggingface/`.

## Prerequisites

No accounts or API keys are required — inference runs locally. The service
automatically selects the best available device (Intel XPU, CUDA, Apple MPS,
then CPU). Pass `use_cpu=True` to force CPU execution.

## Configuration

<ParamField path="model" type="str" default="None">
  *(Deprecated)* HuggingFace model identifier. Prefer
  `settings=SmolVlmService.Settings(model=...)`.
</ParamField>

<ParamField path="use_cpu" type="bool" default="False">
  Force CPU execution even when a GPU is available. Useful for reproducibility
  or constrained environments.
</ParamField>

<ParamField path="settings" type="SmolVlmService.Settings" default="None">
  Runtime-configurable generation settings. See [Settings](#settings) below.
</ParamField>

### Settings

Runtime-configurable settings passed via the `settings` constructor argument
using `SmolVlmService.Settings(...)`.

| Parameter        | Type            | Default                                 | Description                                        |
| ---------------- | --------------- | --------------------------------------- | -------------------------------------------------- |
| `model`          | `str`           | `"HuggingFaceTB/SmolVLM-256M-Instruct"` | HuggingFace model ID or local path.                |
| `max_new_tokens` | `int`           | `500`                                   | Maximum tokens to generate per image.              |
| `default_prompt` | `str`           | `"Describe the given image."`           | Fallback prompt when the frame carries no text.    |
| `temperature`    | `float \| None` | `None`                                  | Sampling temperature. `None` uses greedy decoding. |
| `do_sample`      | `bool`          | `False`                                 | Enable sampling without a fixed temperature.       |

<Note>
  Available parameters and defaults are defined by the integration. See the
  [source repository](https://github.com/rahulsolanki001/pipecat-smolvlm) for
  the authoritative, up-to-date list.
</Note>

## Usage

```python theme={null}
from pipecat_smolvlm import SmolVlmService

# Default: SmolVLM-256M-Instruct, auto device detection
service = SmolVlmService()

# Custom model and generation settings
service = SmolVlmService(
    settings=SmolVlmService.Settings(
        model="HuggingFaceTB/SmolVLM-500M-Instruct",
        max_new_tokens=256,
        default_prompt="What objects can you see?",
        temperature=0.3,
        do_sample=True,
    )
)

# Force CPU (no GPU required)
service = SmolVlmService(use_cpu=True)
```

Drop `SmolVlmService` into a Pipeline wherever a vision service is expected:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    image_capture_processor,   # converts camera frames → UserImageRawFrame
    SmolVlmService(),
    vision_to_speech,          # VisionTextFrame → TextFrame
    tts_service,
    transport.output(),
])
```

To use a custom per-image prompt, set the `text` field on the incoming
`UserImageRawFrame`; the service falls back to `settings.default_prompt` when it
is empty.

## Compatibility

The package targets `pipecat-ai>=0.0.100`. Check the [source
repository](https://github.com/rahulsolanki001/pipecat-smolvlm) for the latest
tested version and changelog.