Is vision RAG a separate product from multimodal RAG?

No. Vision RAG is multimodal RAG whose retrieved units are images or frames. Text RAG retrieves passages. A multimodal pipeline declares both indexes on the tables that hold the files.

Do I need LangChain to wire retrieval to the model?

No. Retrieval is a query on the table. The model call is a computed column or a later prompt. LangGraph can still run an agent loop that calls that query as a tool; see Pixeltable vs LangChain.

Do I still need a separate vector database?

You need an embedding index so similar pieces can be found. That index can live on the chunk view and the image table. A separate vector database is optional when something else is already the source of truth.

How to

How do I build a multimodal RAG pipeline in Python?

A multimodal RAG pipeline retrieves pages, frames, or images — not only text chunks — and passes those pieces to a language model at question time. Vision RAG is the image half of that job: embed stills, rank them, and ground the answer in the retrieved picture. The index is one part of the pipeline, not a second database.

Updated 2026-10-14 · Part of How do I ... in Python?

Before you start

About 30 min.

Python 3.11+
pip install -U 'pixeltable[serve]'
pxt init in the project root
spacy and tiktoken for sentence,token_limit splitting
torch for CLIP
An OpenAI API key for gpt-4o-mini

1. Store documents and images as typed columns

A multimodal table holds a PDF and a picture as columns the engine understands, not paths in a string. Retrieval units later point back at those rows.

import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import clip, sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Docs(TableModel, name='docs'):
    title: pxt.String
    document: pxt.Document

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None

2. Chunk PDFs and index stills in one schema

Split each document into sentence windows with a token limit. Index passage text for meaning. Index images with CLIP so a sentence can retrieve a picture — that is vision RAG on the same backend.

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='sentence,token_limit',
        limit=300,
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

3. Retrieve evidence, then generate from those rows

A query ranks passages and stills. A function joins the passage text and still captions into one prompt. The language model reads that prompt, so the answer uses the retrieved rows.

from pixeltable.functions.openai import chat_completions

@pxt.query
def retrieve_passages(question: str, limit: int = 5):
    sim = Chunks.text.similarity(string=question)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(text=Chunks.text, title=Docs.title, score=sim)
    )

@pxt.query
def retrieve_stills(question: str, limit: int = 5):
    sim = Images.image.similarity(string=question)
    return (
        Images.order_by(sim, asc=False)
        .limit(limit)
        .select(caption=Images.caption, score=sim)
    )

@pxt.udf
def build_prompt(question: str, passages: list[dict], stills: list[dict]) -> str:
    passage_lines = '\n'.join(f"{row['title']}: {row['text']}" for row in passages)
    still_lines = '\n'.join(row['caption'] or 'untitled still' for row in stills)
    return (
        'Answer using only the retrieved passages and still captions.\n\n'
        f'Passages:\n{passage_lines}\n\n'
        f'Stills:\n{still_lines}\n\n'
        f'Question: {question}'
    )

class Questions(TableModel, name='questions'):
    question: pxt.String
    passages = retrieve_passages(question, limit=5)
    stills = retrieve_stills(question, limit=5)
    prompt = build_prompt(question, passages, stills)
    answer = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    ).choices[0].message.content

Text RAG, vision RAG, and a multimodal pipeline

Text RAG, vision RAG, and a multimodal pipeline
	Text-only RAG	Vision RAG	Multimodal pipeline
Units	Passages	Stills or frames	Passages and stills on one schema
Index	A text embedding	CLIP on the image column	One index per modality you search
Evidence	A quote	A picture	The piece the question needs
Unused media	Normal if images are attachments	A failure of the pipeline	A failure of the pipeline

Where Pixeltable fits

Pixeltable is the unified multimodal backend for a multimodal RAG tutorial in one Python file: Document and Image columns, a document_splitter view, CLIP on stills, and @pxt.query retrieval. Agentic RAG is the same store with retrieval exposed as a tool. A separate vector database is optional.

import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import clip, sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Docs(TableModel, name='docs'):
    document: pxt.Document

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=300
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

class Images(TableModel, name='images'):
    image: pxt.Image
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

Questions

Is vision RAG a separate product from multimodal RAG?: No. Vision RAG is multimodal RAG whose retrieved units are images or frames. Text RAG retrieves passages. A multimodal pipeline declares both indexes on the tables that hold the files.
Do I need LangChain to wire retrieval to the model?: No. Retrieval is a query on the table. The model call is a computed column or a later prompt. LangGraph can still run an agent loop that calls that query as a tool; see Pixeltable vs LangChain.
Do I still need a separate vector database?: You need an embedding index so similar pieces can be found. That index can live on the chunk view and the image table. A separate vector database is optional when something else is already the source of truth.