---
title: "Build a Video Search Engine in Python"
description: "A video search engine finds a moment inside a clip, not only a filename. You sample frames or keyframes, embed them with a model such as CLIP, index those vectors, rank by similarity, and serve the query over HTTP."
keywords:
  - video search engine
  - video analysis ai
  - python video search
  - CLIP video search
  - semantic video search
  - keyframe search
  - video similarity search
  - build a video search engine
complexity: "intermediate"
estimated_time: "25 min"
url: "https://www.pixeltable.com/use-cases/video-search-engine"
---

# Build a Video Search Engine in Python

A video search engine finds a moment inside a clip, not only a filename. You sample frames or keyframes, embed them with a model such as CLIP, index those vectors, rank by similarity, and serve the query over HTTP.

## Prerequisites

- Basic Python
- A few video files or URLs to insert

## The Problem

A typical video search stack splits ownership: object storage for the files, a database for titles, a job to dump frames, an embedding service, a vector database, and a FastAPI app that tries to keep those copies aligned. Changing the sample rate or the model means touching every joint.

## The Solution

Pixeltable is the unified multimodal backend for that job. One Python file declares the video table, the frame view, the CLIP index, and the HTTP route (TableModel + FastAPIRouter + the pxt CLI). Insert a clip; frames and embeddings follow that row. The same file deploys to Pixeltable Cloud (Limited Beta). It is not an LLM wrapper and not a standalone vector database.

## Implementation

### Declare the video table

Store each clip as a typed Video column plus the metadata you will filter on. Apply the file with pxt schema update; do not call create_table at import.

```python
import pixeltable as pxt

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
```

The catalog records that the column is video. Later columns can extract frames or audio because the type is known, not guessed from a path string.


### Extract frames or keyframes

A view walks each clip and emits one row per still. fps=1 samples once a second. keyframes_only=True emits I-frames. Set only one of fps, num_frames, or keyframes_only.

```python
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.functions.yolox import yolox

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)

# Long archives: frame_iterator(video=Videos.video, keyframes_only=True)
```

Each frame row keeps frame_attrs: index in the stream, time in seconds from the start, and whether the still is a keyframe. Whisper on the soundtrack and YOLOX on the stills are optional video analysis AI columns; search does not require them.


### Embed frames and declare the index

CLIP places stills and sentences in one space. The embedding index is declared on the frame column, so a new clip is indexed when it is inserted.

```python
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator
from pixeltable.functions.yolox import yolox

clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]
```

embedding= registers both CLIP overloads, so a sentence and a still share one index. The index lives on the view that holds the frames. You do not copy vectors into a second database and then write a job to keep them aligned.


### Rank moments by similarity

Embed the question the same way. Nearest frames come back with the parent clip and frame_attrs so the hit is a timestamp, not only a filename.

```python
@pxt.query
def find_moments(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(Videos.title, Frames.frame, Frames.frame_attrs, sim)
    )
```

A sentence query works because CLIP is cross-modal. An image query uses the same index: Frames.frame.similarity(image=still).


### Serve search over HTTP

FastAPIRouter exposes insert and the similarity query from the same file. Apply locally, then publish that file to Pixeltable Cloud (Limited Beta).

```python
from pixeltable.serving import FastAPIRouter

api = FastAPIRouter(name='video-search')
api.add_insert_route(
    Videos,
    path='/videos',
    inputs=[Videos.video, Videos.title],
)
api.add_query_route(path='/search', query=find_moments)

# pxt init
# pxt schema update app.py video_search
# pxt service update app.py video_search
# Cloud (Limited Beta), same file:
# pxt db update pxt://org:db
# pxt schema update app.py pxt://org:db
# pxt service update app.py pxt://org:db
```

The route reads the table. Callers do not talk to a separate vector database. Get started at /get-started.


## Glue stack vs one Python file

| Job | Typical glue stack | One Python file |
| --- | --- | --- |
| Store the clip | Object storage plus a metadata database | A Video column on the table you own |
| Sample frames | A scheduled ffmpeg job between stores | A frame_iterator view on that clip |
| Embed stills | An embedding service you poll and copy out | CLIP as the index function on the frame column |
| Rank similar moments | A separate vector database you keep aligned | EmbeddingIndex on the same frame view |
| Serve search | A FastAPI app that joins those copies | FastAPIRouter declared beside the tables |
| Deploy | Rewrite the joints against hosted services | The same file on Pixeltable Cloud (Limited Beta) |

## Benefits

- One Video column instead of an object store plus a metadata database for the same clip
- Frame extraction and embeddings are columns, not an Airflow job you rerun by hand
- The CLIP index lives on the frame view, so a separate vector database is optional
- FastAPIRouter exposes insert and search from the same file the pxt CLI applies
- The same file deploys to Pixeltable Cloud (Limited Beta)

## Use Cases

- Find a spoken or visual moment in a training library
- Search marketing footage with a sentence or a still
- Review a clip at a timestamp instead of scrubbing the file
- Add detections or a transcript as extra columns on the same rows

## Requirements

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- ffmpeg for video decoding
- torch and transformers for CLIP (openai/clip-vit-base-patch32)
- Optional: openai-whisper and pixeltable-yolox for transcript and detection columns

## Resources

- [How does video search work?](https://www.pixeltable.com/learn/how-video-search-works) - Frames, embeddings, and a timestamped hit
- [Extract video frames](https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames) - fps, num_frames, and keyframes_only on frame_iterator
- [Embedding index](https://docs.pixeltable.com/datastore/embedding-index) - Declare similarity search on the table that holds the pieces
- [ClipFinder: natural-language video moment search](https://www.pixeltable.com/blog/clipfinder-semantic-video-moment-search) - The same TableModel + FastAPIRouter shape as a product
- [Get started](https://www.pixeltable.com/get-started) - Install pxt, write one app.py, apply schema and service

## Frequently asked questions

### What is a keyframe, and where is the frame timestamp? {#faq-1}

A keyframe (I-frame) is a complete picture in the compressed stream. frame_iterator(..., keyframes_only=True) emits only those stills. fps=1 samples at a fixed rate instead. You cannot set fps, num_frames, and keyframes_only together. Each row keeps frame_attrs["index"] (stream index) and frame_attrs["time"] (seconds from the start of the clip), so a hit is a moment, not only a filename.

### Do I still need a separate vector database? {#faq-2}

You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth. Pixeltable declares pxt.EmbeddingIndex on the table that holds the frames.

### Is this an LLM wrapper or a hosted video API? {#faq-3}

Neither. The tables, the CLIP index, and the HTTP route are schema you own in one Python file. A chat model that watches one clip on every question is not an index. A hosted video API that returns JSON and no table leaves you without the rows.

### Can I search with an image as well as a sentence? {#faq-4}

Yes, if the embedding model places text and images in one space. CLIP-style models do. Frames.frame.similarity(string=query) ranks stills against a sentence; Frames.frame.similarity(image=still) ranks against a picture.