PixeltablePixeltable Logo

How to

How do I build video RAG in Python?

Video RAG retrieves a moment in a clip — a frame, timestamp, or transcript line — and gives that evidence to a language model. It is not captioning one file inside a chat window, and it is not a vector of titles. GitHub examples are a table, a frame view, an index, and a query that returns the time.

Updated · Part of How do I ... in Python?

Before you start

About 25 min.

  • Python 3.11+
  • pip install -U 'pixeltable[serve]'
  • pxt init in the project root
  • ffmpeg
  • openai-whisper
  • torch and transformers for CLIP
  • An OpenAI API key for gpt-4o-mini

1. Turn each clip into timestamped stills

A view walks the video and emits one row per still. Keep frame_attrs so a hit is a moment. The sample also transcribes the soundtrack on every insert, which needs openai-whisper installed first.

import pixeltable as pxt
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe


TableModel = pxt.model_base()


class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)


class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    pass

2. Embed the stills in a shared text-and-image space

CLIP places a sentence and a picture in one space. Declare the index on the frame column so a new clip is indexed when it is inserted. You do not copy vectors into a second database.

from pixeltable.functions.huggingface import clip


clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')


class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

3. Retrieve a moment, then generate from that evidence

Rank stills and return the parent title with frame_attrs. A function joins those rows into the prompt, including each presentation timestamp. The model reads that prompt, not the question alone.

from pixeltable.functions.openai import chat_completions


@pxt.query
def retrieve_moments(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )


@pxt.udf
def build_prompt(question: str, moments: list[dict]) -> str:
    lines = '\n'.join(
        f"{row['title']}: presentation timestamp {row['frame_attrs']['time']} seconds"
        for row in moments
    )
    return (
        'Answer using only these retrieved moments.\n\n'
        f'{lines}\n\n'
        f'Question: {question}'
    )


class Questions(TableModel, name='questions'):
    question: pxt.String
    context = retrieve_moments(question, limit=5)
    prompt = build_prompt(question, context)
    answer = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    ).choices[0].message.content

Video RAG versus the things it is confused with

Video RAG versus the things it is confused with
Video RAGNot this
EvidenceA timestamped still, maybe a transcript lineA clip title or a filename
IndexFrames and a transcript on the source rowA chat that re-watches the file
Model inputThe retrieved momentThe entire library on every question
CodeA table, a view, an index, a query — on GitHub as PythonA hosted JSON API with no rows

Where Pixeltable fits

Pixeltable is the unified multimodal backend for video RAG: Video, frame_iterator, CLIP EmbeddingIndex, and a query that returns frame_attrs. The same file is what the GitHub repo's apps look like. A hosted video API that returns JSON and no table leaves you without the rows. Compare Pixeltable vs Twelve Labs when the hosted model is the product.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator


TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')


class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video


class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]


@pxt.query
def retrieve_moments(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )

Questions

Where is a video RAG example on GitHub?
The Pixeltable repository (github.com/pixeltable/pixeltable) is the Python package those apps import. The video search engine use case on this site is the same TableModel shape: frames, CLIP, a query that returns the time. ClipFinder is a product on that shape.
Is sending the whole file to a chat model video RAG?
No. That can describe one clip. It does not rank a library, and it does not get cheaper as the library grows. Video RAG retrieves a moment, then generates from that evidence.
Do I still need a separate vector database?
You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth.