---
title: "How do I build a multimodal RAG pipeline in Python?"
description: "Build a multimodal RAG pipeline in Python: chunk documents, index images with CLIP, retrieve passages or stills, and ground the language model."
url: "https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline"
updated: "2026-10-14"
doc: "https://docs.pixeltable.com/howto/cookbooks/agents/pattern-rag-pipeline"
---

# How do I build a multimodal RAG pipeline in Python?

A multimodal RAG pipeline retrieves pages, frames, or images — not only text chunks — and passes those pieces to a language model at question time. Vision RAG is the image half of that job: embed stills, rank them, and ground the answer in the retrieved picture. The index is one part of the pipeline, not a second database.

Updated: 2026-10-14
Part of [How do I ... in Python?](https://www.pixeltable.com/learn/how-to).

## Before you start

About 30 min.

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- pxt init in the project root
- spacy and tiktoken for sentence,token_limit splitting
- torch for CLIP
- An OpenAI API key for gpt-4o-mini

## On this page

- [Before you start](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline#before-you-start)
- [Store documents and images as typed columns](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline#step-1)
- [Chunk PDFs and index stills in one schema](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline#step-2)
- [Retrieve evidence, then generate from those rows](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline#step-3)
- [Comparison](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline#questions)

## 1. Store documents and images as typed columns {#step-1}

A multimodal table holds a PDF and a picture as columns the engine understands, not paths in a string. Retrieval units later point back at those rows.

```python
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import clip, sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Docs(TableModel, name='docs'):
    title: pxt.String
    document: pxt.Document

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None
```

## 2. Chunk PDFs and index stills in one schema {#step-2}

Split each document into sentence windows with a token limit. Index passage text for meaning. Index images with CLIP so a sentence can retrieve a picture — that is vision RAG on the same backend.

```python
class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='sentence,token_limit',
        limit=300,
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]
```

## 3. Retrieve evidence, then generate from those rows {#step-3}

A query ranks passages and stills. A function joins the passage text and still captions into one prompt. The language model reads that prompt, so the answer uses the retrieved rows.

```python
from pixeltable.functions.openai import chat_completions

@pxt.query
def retrieve_passages(question: str, limit: int = 5):
    sim = Chunks.text.similarity(string=question)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(text=Chunks.text, title=Docs.title, score=sim)
    )

@pxt.query
def retrieve_stills(question: str, limit: int = 5):
    sim = Images.image.similarity(string=question)
    return (
        Images.order_by(sim, asc=False)
        .limit(limit)
        .select(caption=Images.caption, score=sim)
    )

@pxt.udf
def build_prompt(question: str, passages: list[dict], stills: list[dict]) -> str:
    passage_lines = '\n'.join(f"{row['title']}: {row['text']}" for row in passages)
    still_lines = '\n'.join(row['caption'] or 'untitled still' for row in stills)
    return (
        'Answer using only the retrieved passages and still captions.\n\n'
        f'Passages:\n{passage_lines}\n\n'
        f'Stills:\n{still_lines}\n\n'
        f'Question: {question}'
    )

class Questions(TableModel, name='questions'):
    question: pxt.String
    passages = retrieve_passages(question, limit=5)
    stills = retrieve_stills(question, limit=5)
    prompt = build_prompt(question, passages, stills)
    answer = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    ).choices[0].message.content
```

## Text RAG, vision RAG, and a multimodal pipeline {#comparison}

|  | Text-only RAG | Vision RAG | Multimodal pipeline |
| --- | --- | --- | --- |
| Units | Passages | Stills or frames | Passages and stills on one schema |
| Index | A text embedding | CLIP on the image column | One index per modality you search |
| Evidence | A quote | A picture | The piece the question needs |
| Unused media | Normal if images are attachments | A failure of the pipeline | A failure of the pipeline |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the unified multimodal backend for a multimodal RAG tutorial in one Python file: Document and Image columns, a document_splitter view, CLIP on stills, and @pxt.query retrieval. Agentic RAG is the same store with retrieval exposed as a tool. A separate vector database is optional.

```python
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import clip, sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Docs(TableModel, name='docs'):
    document: pxt.Document

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=300
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

class Images(TableModel, name='images'):
    image: pxt.Image
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]
```

## Concepts

- [What is multimodal RAG?](https://www.pixeltable.com/learn/what-is-multimodal-rag)
- [What is agentic RAG?](https://www.pixeltable.com/learn/what-is-agentic-rag)
- [What is a multimodal database?](https://www.pixeltable.com/learn/what-is-a-multimodal-database)

## Use case

- [Production RAG: From Documents to Answers in One System](https://www.pixeltable.com/use-cases/production-rag-implementation)

## Questions {#questions}

### Is vision RAG a separate product from multimodal RAG? {#faq-1}

No. Vision RAG is multimodal RAG whose retrieved units are images or frames. Text RAG retrieves passages. A multimodal pipeline declares both indexes on the tables that hold the files.

### Do I need LangChain to wire retrieval to the model? {#faq-2}

No. Retrieval is a query on the table. The model call is a computed column or a later prompt. LangGraph can still run an agent loop that calls that query as a tool; see Pixeltable vs LangChain.

### Do I still need a separate vector database? {#faq-3}

You need an embedding index so similar pieces can be found. That index can live on the chunk view and the image table. A separate vector database is optional when something else is already the source of truth.

## Related

- [What is multimodal RAG?](https://www.pixeltable.com/learn/what-is-multimodal-rag)
- [What is agentic RAG?](https://www.pixeltable.com/learn/what-is-agentic-rag)
- [What is a multimodal database?](https://www.pixeltable.com/learn/what-is-a-multimodal-database)
- [Production RAG implementation](https://www.pixeltable.com/use-cases/production-rag-implementation)
- [How do I chunk PDFs for RAG in Python?](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag)
- [How do I embed images for search in Python?](https://www.pixeltable.com/learn/how-to/embed-images-for-search)
- [Pixeltable vs LangChain](https://www.pixeltable.com/compare/pixeltable-vs-langchain)
- [Pixeltable vs Pinecone](https://www.pixeltable.com/compare/pixeltable-vs-pinecone)
- [Build a RAG pipeline](https://docs.pixeltable.com/howto/cookbooks/agents/pattern-rag-pipeline)