---
title: "How do I build video RAG in Python?"
description: "Build video RAG in Python: sample frames, embed them, retrieve a timestamped moment, and ground the language model in that presentation timestamp."
url: "https://www.pixeltable.com/learn/how-to/build-video-rag"
updated: "2026-10-14"
doc: "https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames"
---

# How do I build video RAG in Python?

Video RAG retrieves a moment in a clip — a frame, timestamp, or transcript line — and gives that evidence to a language model. It is not captioning one file inside a chat window, and it is not a vector of titles. GitHub examples are a table, a frame view, an index, and a query that returns the time.

Updated: 2026-10-14
Part of [How do I ... in Python?](https://www.pixeltable.com/learn/how-to).

## Before you start

About 25 min.

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- pxt init in the project root
- ffmpeg
- openai-whisper
- torch and transformers for CLIP
- An OpenAI API key for gpt-4o-mini

## On this page

- [Before you start](https://www.pixeltable.com/learn/how-to/build-video-rag#before-you-start)
- [Turn each clip into timestamped stills](https://www.pixeltable.com/learn/how-to/build-video-rag#step-1)
- [Embed the stills in a shared text-and-image space](https://www.pixeltable.com/learn/how-to/build-video-rag#step-2)
- [Retrieve a moment, then generate from that evidence](https://www.pixeltable.com/learn/how-to/build-video-rag#step-3)
- [Comparison](https://www.pixeltable.com/learn/how-to/build-video-rag#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/how-to/build-video-rag#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/how-to/build-video-rag#questions)

## 1. Turn each clip into timestamped stills {#step-1}

A view walks the video and emits one row per still. Keep frame_attrs so a hit is a moment. The sample also transcribes the soundtrack on every insert, which needs openai-whisper installed first.

```python
import pixeltable as pxt
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    pass
```

## 2. Embed the stills in a shared text-and-image space {#step-2}

CLIP places a sentence and a picture in one space. Declare the index on the frame column so a new clip is indexed when it is inserted. You do not copy vectors into a second database.

```python
from pixeltable.functions.huggingface import clip

clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]
```

## 3. Retrieve a moment, then generate from that evidence {#step-3}

Rank stills and return the parent title with frame_attrs. A function joins those rows into the prompt, including each presentation timestamp. The model reads that prompt, not the question alone.

```python
from pixeltable.functions.openai import chat_completions

@pxt.query
def retrieve_moments(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )

@pxt.udf
def build_prompt(question: str, moments: list[dict]) -> str:
    lines = '\n'.join(
        f"{row['title']}: presentation timestamp {row['frame_attrs']['time']} seconds"
        for row in moments
    )
    return (
        'Answer using only these retrieved moments.\n\n'
        f'{lines}\n\n'
        f'Question: {question}'
    )

class Questions(TableModel, name='questions'):
    question: pxt.String
    context = retrieve_moments(question, limit=5)
    prompt = build_prompt(question, context)
    answer = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    ).choices[0].message.content
```

## Video RAG versus the things it is confused with {#comparison}

|  | Video RAG | Not this |
| --- | --- | --- |
| Evidence | A timestamped still, maybe a transcript line | A clip title or a filename |
| Index | Frames and a transcript on the source row | A chat that re-watches the file |
| Model input | The retrieved moment | The entire library on every question |
| Code | A table, a view, an index, a query — on GitHub as Python | A hosted JSON API with no rows |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the unified multimodal backend for video RAG: Video, frame_iterator, CLIP EmbeddingIndex, and a query that returns frame_attrs. The same file is what the GitHub repo's apps look like. A hosted video API that returns JSON and no table leaves you without the rows. Compare Pixeltable vs Twelve Labs when the hosted model is the product.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def retrieve_moments(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )
```

## Concepts

- [What is video RAG?](https://www.pixeltable.com/learn/what-is-video-rag)
- [What is multimodal RAG?](https://www.pixeltable.com/learn/what-is-multimodal-rag)
- [How does video search work?](https://www.pixeltable.com/learn/how-video-search-works)

## Use case

- [Build a Video Search Engine in Python](https://www.pixeltable.com/use-cases/video-search-engine)

## Questions {#questions}

### Where is a video RAG example on GitHub? {#faq-1}

The Pixeltable repository (github.com/pixeltable/pixeltable) is the Python package those apps import. The video search engine use case on this site is the same TableModel shape: frames, CLIP, a query that returns the time. ClipFinder is a product on that shape.

### Is sending the whole file to a chat model video RAG? {#faq-2}

No. That can describe one clip. It does not rank a library, and it does not get cheaper as the library grows. Video RAG retrieves a moment, then generates from that evidence.

### Do I still need a separate vector database? {#faq-3}

You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth.

## Related

- [What is video RAG?](https://www.pixeltable.com/learn/what-is-video-rag)
- [What is multimodal RAG?](https://www.pixeltable.com/learn/what-is-multimodal-rag)
- [How does video search work?](https://www.pixeltable.com/learn/how-video-search-works)
- [Build a video search engine in Python](https://www.pixeltable.com/use-cases/video-search-engine)
- [How do I extract keyframes from video in Python?](https://www.pixeltable.com/learn/how-to/extract-keyframes-from-video)
- [How do I build a multimodal RAG pipeline in Python?](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline)
- [Where VideoRAG stores its indexes](https://www.pixeltable.com/blog/videorag-where-pixeltable-stores-indexes)
- [Pixeltable vs Twelve Labs](https://www.pixeltable.com/compare/pixeltable-vs-twelve-labs)
- [pixeltable/pixeltable on GitHub](https://github.com/pixeltable/pixeltable)