---
title: "How do I index a video library and search it by text in Python?"
description: "Index a video library in Python: sample frames, embed with CLIP, optionally transcribe speech, and search by text so a hit is a clip plus timestamp."
url: "https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text"
updated: "2026-10-23"
doc: "https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames"
---

# How do I index a video library and search it by text in Python?

Text search over video means sampling frames and embedding them with a text-image model such as CLIP, and optionally transcribing the soundtrack and embedding that transcript. Frame search returns a moment (the clip plus seconds). Transcript search returns the clip, because the transcript is one string for the file. A new clip indexes only that clip.

Updated: 2026-10-23
Part of [How do I ... in Python?](https://www.pixeltable.com/learn/how-to).

## Before you start

About 30 min.

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- pxt init in the project root
- ffmpeg
- openai-whisper
- torch and transformers for CLIP
- sentence-transformers

## On this page

- [Before you start](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#before-you-start)
- [Store videos as a typed video column](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#step-1)
- [Sample frames and keep the timestamp](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#step-2)
- [Embed frames with a text-image model](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#step-3)
- [Optionally transcribe speech and index the text](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#step-4)
- [Rank frames by similarity to a sentence](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#step-5)
- [Serve insert and search over HTTP](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#step-6)
- [Comparison](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text#questions)

## 1. Store videos as a typed video column {#step-1}

Put each clip in a table as video, plus any metadata you will filter on. A path string is not enough: frame extraction and audio extraction need a typed column.

```python
import pixeltable as pxt

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
```

## 2. Sample frames and keep the timestamp {#step-2}

A view walks each clip and emits one still per sample. fps=1 is one still a second. Keep frame_attrs so a hit is a moment, not a filename.

```python
from pixeltable.functions.video import frame_iterator

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    pass

# frame_attrs["time"] is the presentation timestamp in seconds
```

## 3. Embed frames with a text-image model {#step-3}

CLIP places a sentence and a still in one space. Declare the index on the frame column so a new clip is indexed when it is inserted. You do not copy vectors into a second database.

```python
from pixeltable.functions.huggingface import clip

clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]
```

## 4. Optionally transcribe speech and index the text {#step-4}

Extract the soundtrack, transcribe it, and index the transcript with a text embedding model. Leave these columns out and the upload only samples frames. Frame search finds what is on screen and returns a moment. Transcript search finds what was said and returns the clip, not a timestamp inside it.

```python
from pixeltable.functions.huggingface import sentence_transformer
from pixeltable.functions.video import extract_audio
from pixeltable.functions.whisper import transcribe

text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)
    __indexes__ = [
        pxt.EmbeddingIndex(transcript, string_embed=text_embed),
    ]

@pxt.query
def search_speech(query: str, limit: int = 5):
    sim = Videos.transcript.similarity(string=query)
    return (
        Videos.order_by(sim, asc=False)
        .limit(limit)
        .select(Videos.title, Videos.video, score=sim)
    )
```

## 5. Rank frames by similarity to a sentence {#step-5}

Embed the question with the same CLIP model. Return the parent title, frame_attrs, and score. The time field is the moment to seek to.

```python
@pxt.query
def search_video(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )
```

## 6. Serve insert and search over HTTP {#step-6}

Expose an upload route for new clips and a search route for the frame query. Add /search-speech when the transcript column from the previous step is in the schema. Apply the schema, then start the service. A new file indexes only that file.

```python
from pixeltable.serving import FastAPIRouter

api = FastAPIRouter(name='video_lib')
api.add_insert_route(
    Videos,
    path='/videos',
    inputs=[Videos.title],
    uploadfile_inputs=[Videos.video],
)
api.add_query_route(path='/search', query=search_video)
api.add_query_route(path='/search-speech', query=search_speech)

# pxt init
# pxt schema update app.py video_lib -f
# pxt service update app.py video_lib -f
```

## The stack coding agents usually pick vs one schema {#comparison}

| Job | Glue stack | One schema | Hosted video APIs |
| --- | --- | --- | --- |
| Storage | SQLite plus files on disk | A Video column on the table you own | Your bucket, or an asset they host |
| Vector index | Chroma, Qdrant, or LanceDB as a second store | EmbeddingIndex on the frame view | Their hosted index |
| Transcription | whisper as a separate job | extract_audio, then Whisper on the row | Included if their pipeline extracts speech |
| Embeddings | sentence-transformers and CLIP in a worker | CLIP on frames; a text model on the transcript | Their proprietary video models |
| Job runner | A queue or a cron between stores | Computed columns and views on insert | Their ingest pipeline |
| Serving | FastAPI you write against those copies | FastAPIRouter beside the tables | Their HTTP API |
| New video added | Rerun the DAG or upsert by hand | Insert a row; only that clip is indexed | Upload an asset and wait until ready |
| Model changed | Re-embed the library in a batch job | Change the embedder in the file and re-apply | Depends on their reindex contract |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the unified multimodal backend for this job: a Video column, a frame_iterator view, CLIP EmbeddingIndex, optional Whisper, and FastAPIRouter insert plus search in one file. Mixpeek, Twelve Labs, and VideoDB are managed video APIs you call over HTTP; you do not run the models, and pricing is on their sites. Pick them when you have no GPU, want a managed API, or need their proprietary video models. Pick this schema when the clip, the timestamp, and HTTP should live in one app.py you own. Pixeltable Cloud is Limited Beta. Compare Pixeltable vs Mixpeek and Pixeltable vs Twelve Labs.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)
    __indexes__ = [
        pxt.EmbeddingIndex(transcript, string_embed=text_embed),
    ]

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def search_video(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )

@pxt.query
def search_speech(query: str, limit: int = 5):
    sim = Videos.transcript.similarity(string=query)
    return (
        Videos.order_by(sim, asc=False)
        .limit(limit)
        .select(Videos.title, Videos.video, score=sim)
    )

api = FastAPIRouter(name='video_lib')
api.add_insert_route(
    Videos,
    path='/videos',
    inputs=[Videos.title],
    uploadfile_inputs=[Videos.video],
)
api.add_query_route(path='/search', query=search_video)
api.add_query_route(path='/search-speech', query=search_speech)

# pxt init
# pxt schema update app.py video_lib -f
# pxt service update app.py video_lib -f
```

## Concepts

- [How does video search work?](https://www.pixeltable.com/learn/how-video-search-works)
- [What is cross-modal search?](https://www.pixeltable.com/learn/what-is-cross-modal-search)
- [What is a frame iterator?](https://www.pixeltable.com/learn/what-is-a-frame-iterator)

## Use case

- [Build a Video Search Engine in Python](https://www.pixeltable.com/use-cases/video-search-engine)

## Questions {#questions}

### Do I need a vector database to search video by text? {#faq-1}

You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth. Pixeltable vs Pinecone is that split.

### How do I get the timestamp of the match? {#faq-2}

Each still keeps frame_attrs. time is the presentation timestamp in seconds. Return it from the query and seek the parent clip to that moment. A filename alone is not a timestamp.

### Should I embed frames or transcripts? {#faq-3}

Frames catch what is on screen. Transcripts catch what was said. Use both when the library has speech and visuals. CLIP on frames is the usual first index; Whisper plus a text embedding is the optional second.

### How do I add new videos without reindexing? {#faq-4}

Insert a row. Computed columns, the frame view, and the indexes run for that clip. Unchanged clips stay as they are.

### Pixeltable vs Mixpeek or Twelve Labs? {#faq-5}

Mixpeek and Twelve Labs are managed video APIs: you call HTTP, they run the models, and pricing is on their sites. They are the better choice when you do not want to run models, or you need their hosted video models. Pixeltable is the table those results can live on, plus CLIP, Whisper, and HTTP you own. See Pixeltable vs Mixpeek and Pixeltable vs Twelve Labs.

## Related

- [How does video search work?](https://www.pixeltable.com/learn/how-video-search-works)
- [What is cross-modal search?](https://www.pixeltable.com/learn/what-is-cross-modal-search)
- [What is a frame iterator?](https://www.pixeltable.com/learn/what-is-a-frame-iterator)
- [Build a video search engine in Python](https://www.pixeltable.com/use-cases/video-search-engine)
- [How do I extract keyframes from video in Python?](https://www.pixeltable.com/learn/how-to/extract-keyframes-from-video)
- [How do I build video RAG in Python?](https://www.pixeltable.com/learn/how-to/build-video-rag)
- [Pixeltable vs Twelve Labs](https://www.pixeltable.com/compare/pixeltable-vs-twelve-labs)
- [Pixeltable vs Mixpeek](https://www.pixeltable.com/compare/pixeltable-vs-mixpeek)
- [Extract frames from videos](https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames)