What is a keyframe, and where is the frame timestamp?

A keyframe (I-frame) is a complete picture in the compressed stream. frame_iterator(..., keyframes_only=True) emits only those stills. fps=1 samples at a fixed rate instead. You cannot set fps, num_frames, and keyframes_only together. Each row keeps frame_attrs["index"] (stream index) and frame_attrs["time"] (seconds from the start of the clip), so a hit is a moment, not only a filename.

Do I still need a separate vector database?

You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth. Pixeltable declares pxt.EmbeddingIndex on the table that holds the frames.

Is this an LLM wrapper or a hosted video API?

Neither. The tables, the CLIP index, and the HTTP route are schema you own in one Python file. A chat model that watches one clip on every question is not an index. A hosted video API that returns JSON and no table leaves you without the rows.

Can I search with an image as well as a sentence?

Yes, if the embedding model places text and images in one space. CLIP-style models do. Frames.frame.similarity(string=query) ranks stills against a sentence; Frames.frame.similarity(image=still) ranks against a picture.

Intermediate · 25 min

Build a Video Search Engine in Python

A video search engine finds a moment inside a clip, not only a filename. You sample frames or keyframes, embed them with a model such as CLIP, index those vectors, rank by similarity, and serve the query over HTTP.

Get started Quick Start Documentation

The Challenge

A typical video search stack splits ownership: object storage for the files, a database for titles, a job to dump frames, an embedding service, a vector database, and a FastAPI app that tries to keep those copies aligned. Changing the sample rate or the model means touching every joint.

The Solution

Pixeltable is the unified multimodal backend for that job. One Python file declares the video table, the frame view, the CLIP index, and the HTTP route (TableModel + FastAPIRouter + the pxt CLI). Insert a clip; frames and embeddings follow that row. The same file deploys to Pixeltable Cloud (Limited Beta). It is not an LLM wrapper and not a standalone vector database.

Implementation Guide

Step-by-step walkthrough with code examples

Step 1 of 5

Declare the video table

Store each clip as a typed Video column plus the metadata you will filter on. Apply the file with pxt schema update; do not call create_table at import.

python

1import pixeltable as pxt
2
3TableModel = pxt.model_base()
4
5class Videos(TableModel, name='videos'):
6    title: pxt.String
7    video: pxt.Video

The catalog records that the column is video. Later columns can extract frames or audio because the type is known, not guessed from a path string.

Use arrow keys to navigate

Glue stack vs one Python file

Job	Typical glue stack	One Python file
Store the clip	Object storage plus a metadata database	A Video column on the table you own
Sample frames	A scheduled ffmpeg job between stores	A frame_iterator view on that clip
Embed stills	An embedding service you poll and copy out	CLIP as the index function on the frame column
Rank similar moments	A separate vector database you keep aligned	EmbeddingIndex on the same frame view
Serve search	A FastAPI app that joins those copies	FastAPIRouter declared beside the tables
Deploy	Rewrite the joints against hosted services	The same file on Pixeltable Cloud (Limited Beta)

Key Benefits

One Video column instead of an object store plus a metadata database for the same clip

Frame extraction and embeddings are columns, not an Airflow job you rerun by hand

The CLIP index lives on the frame view, so a separate vector database is optional

FastAPIRouter exposes insert and search from the same file the pxt CLI applies

The same file deploys to Pixeltable Cloud (Limited Beta)

Real Applications

Find a spoken or visual moment in a training library

Search marketing footage with a sentence or a still

Review a clip at a timestamp instead of scrubbing the file

Add detections or a transcript as extra columns on the same rows

Prerequisites

Basic Python

A few video files or URLs to insert

Python 3.11+

pip install -U 'pixeltable[serve]'

ffmpeg for video decoding

torch and transformers for CLIP (openai/clip-vit-base-patch32)

Optional: openai-whisper and pixeltable-yolox for transcript and detection columns

Learn More

How does video search work?

Frames, embeddings, and a timestamped hit

Extract video frames

fps, num_frames, and keyframes_only on frame_iterator

Embedding index

Declare similarity search on the table that holds the pieces

ClipFinder: natural-language video moment search

The same TableModel + FastAPIRouter shape as a product

Get started

Install pxt, write one app.py, apply schema and service

Related Guides

Video Intelligence Pipeline: Extract, Enrich, and Search Video at Scale
Modality: Video
Computed column: frame_iterator view, then model columns on each frame
Docs
Production RAG: From Documents to Answers in One System
Modality: Document
Computed column: Chunk view plus an embedding index
Docs
Computer Vision Pipeline: Object Detection, Classification, and Search
Modality: Image
Computed column: Detection computed column on an Image or frame column
Docs

Frequently asked questions

What is a keyframe, and where is the frame timestamp?: A keyframe (I-frame) is a complete picture in the compressed stream. frame_iterator(..., keyframes_only=True) emits only those stills. fps=1 samples at a fixed rate instead. You cannot set fps, num_frames, and keyframes_only together. Each row keeps frame_attrs["index"] (stream index) and frame_attrs["time"] (seconds from the start of the clip), so a hit is a moment, not only a filename.
Do I still need a separate vector database?: You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth. Pixeltable declares pxt.EmbeddingIndex on the table that holds the frames.
Is this an LLM wrapper or a hosted video API?: Neither. The tables, the CLIP index, and the HTTP route are schema you own in one Python file. A chat model that watches one clip on every question is not an index. A hosted video API that returns JSON and no table leaves you without the rows.
Can I search with an image as well as a sentence?: Yes, if the embedding model places text and images in one space. CLIP-style models do. Frames.frame.similarity(string=query) ranks stills against a sentence; Frames.frame.similarity(image=still) ranks against a picture.

Ready to get started?

Install Pixeltable and start building in minutes. One Python file for tables, indexes, and HTTP. Pixeltable Cloud is Limited Beta.

Get started Quick Start Guide View on GitHub