---
title: "What is multimodal RAG?"
description: "Multimodal RAG retrieves pages, frames, or transcripts — not only text chunks — and grounds the model in those pieces, with pointers back to the source file."
url: "https://www.pixeltable.com/learn/what-is-multimodal-rag"
updated: "2026-10-12"
vertical: "Retrieval"
doc: "https://docs.pixeltable.com/datastore/embedding-index"
---

# What is multimodal RAG?

Multimodal RAG retrieves pages, frames, or transcript windows, not only plain-text passages, and passes those pieces to a language model at question time. Each retrieved unit keeps a pointer to its source, so the answer can cite a PDF page, a video timestamp, or a spoken line instead of an ungrounded paragraph.

Updated: 2026-10-12
Part of [What is RAG?](https://www.pixeltable.com/learn/what-is-rag).

## On this page

- [How it works](https://www.pixeltable.com/learn/what-is-multimodal-rag#how-it-works)
- [What it is not](https://www.pixeltable.com/learn/what-is-multimodal-rag#what-it-is-not)
- [Comparison](https://www.pixeltable.com/learn/what-is-multimodal-rag#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/what-is-multimodal-rag#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/what-is-multimodal-rag#questions)

## How it works {#how-it-works}


- Each modality is stored as a typed value, not a filename in a string.
- Retrieval units are passages, frames, or transcript windows that still point at the source.
- The model sees the retrieved piece and a pointer to its source, then writes the answer.

## What it is not {#what-it-is-not}

It is not text-only RAG with images stored as unused attachments.

## multimodal RAG: this, and the thing it is confused with {#comparison}

|  | This | Not this |
| --- | --- | --- |
| Evidence | Page, frame, or transcript | Text chunks only |
| Unused media | A failure of the pipeline | Normal, if images are just attachments |
| Index | One per modality you intend to search | A single text index over filenames |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable keeps each modality on the table and retrieves from the view that holds the pieces. The model call is another column or a query. A video search engine is the same shape: frames, a CLIP index, and a query that returns the moment.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Videos(TableModel, name='videos'):
    video: pxt.Video

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def retrieve_moments(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(Frames.video, Frames.frame, Frames.frame_attrs, sim)
    )
```

## Questions {#questions}

### Is multimodal RAG just text RAG with images attached? {#faq-1}

No. If the image is never retrieved and never shown to the model, it is unused storage. Multimodal RAG retrieves the page, frame, or transcript as evidence.

### Can multimodal RAG search video? {#faq-2}

Yes. Sample frames or keyframes, embed them, and retrieve a timestamped still. Pixeltable’s video search engine use case is that pipeline; the model call is an extra column or a later prompt.

### Do I still need a separate vector database? {#faq-3}

You need an embedding index so similar pieces can be found. That index can live on the table that holds the pages or frames. A separate vector database is optional when something else is already the source of truth.

## In the blog

- [Powering Multimodal LLMs: Pixeltable MCP Servers for Rich Data Context](https://www.pixeltable.com/blog/pixeltable-mcp-servers)
- [VideoRAG: Where Pixeltable Stores Frames and Indexes](https://www.pixeltable.com/blog/videorag-where-pixeltable-stores-indexes)

## How-to guides

- [How do I build video RAG in Python?](https://www.pixeltable.com/learn/how-to/build-video-rag)
- [How do I build a multimodal RAG pipeline in Python?](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline)
- [How do I build a company knowledge base for AI from docs, Slack, meeting recordings and videos?](https://www.pixeltable.com/learn/how-to/build-a-multimodal-knowledge-base)

## Related

- [Documentation](https://docs.pixeltable.com/datastore/embedding-index)
- [What are multimodal embeddings?](https://www.pixeltable.com/learn/what-are-multimodal-embeddings)
- [Best tools to build multimodal RAG in 2027](https://www.pixeltable.com/learn/best-multimodal-rag-tools)
- [Production Multimodal RAG with Pixeltable: Dev to Deployment](https://www.pixeltable.com/blog/multimodal-rag-production)
- [Video search engine in Python](https://www.pixeltable.com/use-cases/video-search-engine)
- [Production RAG](https://www.pixeltable.com/use-cases/production-rag-implementation)
- [Multimodal application](https://www.pixeltable.com/use-cases/multimodal-ai-application-development)