Skip to content
PixeltablePixeltable Logo

Retrieval

What is multimodal RAG?

Multimodal RAG retrieves pages, frames, or transcript windows, not only plain-text passages, and passes those pieces to a language model at question time. Each retrieved unit keeps a pointer to its source, so the answer can cite a PDF page, a video timestamp, or a spoken line instead of an ungrounded paragraph.

Updated · Part of What is RAG?

How it works

  • Each modality is stored as a typed value, not a filename in a string.
  • Retrieval units are passages, frames, or transcript windows that still point at the source.
  • The model sees the retrieved piece and a pointer to its source, then writes the answer.

What it is not

It is not text-only RAG with images stored as unused attachments.

multimodal RAG: this, and the thing it is confused with

multimodal RAG: this, and the thing it is confused with
ThisNot this
EvidencePage, frame, or transcriptText chunks only
Unused mediaA failure of the pipelineNormal, if images are just attachments
IndexOne per modality you intend to searchA single text index over filenames

Where Pixeltable fits

Pixeltable keeps each modality on the table and retrieves from the view that holds the pieces. The model call is another column or a query. A video search engine is the same shape: frames, a CLIP index, and a query that returns the moment.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator


TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')


class Videos(TableModel, name='videos'):
    video: pxt.Video


class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]


@pxt.query
def retrieve_moments(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(Frames.video, Frames.frame, Frames.frame_attrs, sim)
    )

Questions

Is multimodal RAG just text RAG with images attached?

No. If the image is never retrieved and never shown to the model, it is unused storage. Multimodal RAG retrieves the page, frame, or transcript as evidence.

Can multimodal RAG search video?

Yes. Sample frames or keyframes, embed them, and retrieve a timestamped still. Pixeltable’s video search engine use case is that pipeline; the model call is an extra column or a later prompt.

Do I still need a separate vector database?

You need an embedding index so similar pieces can be found. That index can live on the table that holds the pages or frames. A separate vector database is optional when something else is already the source of truth.

Next step

Install the Pixeltable skill, or sign up for Cloud and run the same file on a hosted database.

Get startedQuick Start