---
title: "Multimodal RAG reference architecture"
description: "A multimodal RAG architecture: PDFs and images on one schema, chunking, indexes, retrieval, then generation. Named alternatives per slot."
url: "https://www.pixeltable.com/reference-architecture/multimodal-rag"
updated: "2026-10-28"
doc: "https://docs.pixeltable.com/howto/cookbooks/agents/pattern-rag-pipeline"
---

# Multimodal RAG reference architecture

Multimodal RAG retrieves passages, stills, or frames, then generates from those rows. The index is one piece. Object storage, a parser, a vector engine, and an LLM API are slots you can keep or fold into one TableModel file.

Updated: 2026-10-28
Docs: https://docs.pixeltable.com/howto/cookbooks/agents/pattern-rag-pipeline

A multimodal RAG system has five jobs: hold the files, turn them into retrieval units, embed those units, rank them for a question, and generate from the ranked rows. Vision RAG is the image half of that job. Text RAG is the passage half. A multimodal pipeline declares both.
This page is the architecture, not a tutorial. The Python how-to is How do I build a multimodal RAG pipeline in Python. The docs cookbook is Build a RAG pipeline. Use this page to name each slot and to keep Qdrant, pgvector, or S3 when that engine is already the contract.

## Multimodal RAG: files become retrieval units, then a grounded answer {#diagram}

```mermaid
flowchart TB
  pdfs[PDFs] --> docs[Document column]
  imgs[Images] --> images[Image column]
  docs --> chunks[document_splitter view]
  chunks --> textIdx[Text EmbeddingIndex]
  images --> clipIdx[CLIP EmbeddingIndex]
  textIdx --> retrieve["@pxt.query rank"]
  clipIdx --> retrieve
  retrieve --> prompt[Prompt from retrieved rows]
  prompt --> answer[LLM answer]
  docs -.->|keep if already the contract| s3[S3 or R2]
  textIdx -.->|keep if already the serving index| qdrant[Qdrant / pgvector / Pinecone]
```

## Role of each component

- **Source files.** PDFs, images, and later frames or transcripts. They stay addressable so a hit can cite a page or a still, not only a vector.
- **Typed store.** A Document column and an Image column, or an object store plus a path table. The store must know the type so splitters and embedders can read it.
- **Splitter.** Turns one PDF into passage rows tied to the parent file. sentence,token_limit is the documented default. Page metadata needs separators that include page.
- **Embedding indexes.** Nearest-neighbor search on passage text and on stills. CLIP places a sentence and a picture in one space so a question can retrieve either.
- **Retriever.** A query that ranks rows and returns the text or caption the model is allowed to see. Agentic RAG exposes the same query as a tool.
- **Generator.** An LLM call whose prompt is built from the retrieved rows. The model does not search the corpus itself.

## Honest alternatives per slot {#alternatives}

| Slot | Role | Pixeltable | Named alternatives |
| --- | --- | --- | --- |
| Object store | Bytes for PDFs and images | Media columns under PIXELTABLE_HOME locally; Cloud hosts the same app file | Amazon S3: The bucket is already the system of record, or another fleet reads it; Cloudflare R2: You want S3-compatible object storage with no egress fee on that vendor; MinIO: You need an S3 API inside your network; GCS or Azure Blob: That cloud is already where the files live |
| Document parser | Text and layout from PDFs | pxt.Document plus document_splitter on a view | Unstructured: You need layout-aware parsing before the rows exist; LlamaParse: Complex PDFs with tables that a splitter alone misreads; Apache Tika: Office formats in an existing JVM pipeline |
| Vector index | Nearest-neighbor search over passages and stills | pxt.EmbeddingIndex on the chunk view and the image column | Qdrant: You already run a dedicated ANN service with collections and filters; pgvector: Postgres is already the source of truth and you want vectors next to SQL; Pinecone: A hosted ANN SLA is the product, not the table; Weaviate: You need BM25 plus dense fusion in one engine; LanceDB: You want a Lance file layout and an embed-on-write API |
| Orchestration | Chunk, embed, and reindex on insert | Computed columns and iterator views; insert updates the affected rows | Airflow: You already schedule unrelated DAGs across many systems; Prefect or Temporal: The job is long-running workers with their own retry policy |
| LLM API | Generate from retrieved rows | chat_completions as a computed column, or a later prompt you own | OpenAI, Anthropic, Gemini, Groq: Any of these as the model; Pixeltable is not the model; Ollama or vLLM: The weights run in your process or cluster |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the unified multimodal backend for this architecture: Document and Image columns, a document_splitter view, CLIP on stills, and @pxt.query retrieval in one Python file. Keep S3 when the bucket is already the system of record (load media from S3). Keep Qdrant, pgvector, Pinecone, Weaviate, or LanceDB when that engine is the serving index. Pixeltable vs Pinecone, Pixeltable vs Weaviate, and Pixeltable vs LanceDB are that split. A separate vector database is optional when the index can live on the tables that hold the files.

```python
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.functions.openai import chat_completions

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Docs(TableModel, name='docs'):
    title: pxt.String
    document: pxt.Document

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=300
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def retrieve_passages(question: str, limit: int = 5):
    sim = Chunks.text.similarity(string=question)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(text=Chunks.text, title=Docs.title, score=sim)
    )

@pxt.query
def retrieve_stills(question: str, limit: int = 5):
    sim = Images.image.similarity(string=question)
    return (
        Images.order_by(sim, asc=False)
        .limit(limit)
        .select(caption=Images.caption, score=sim)
    )

@pxt.udf
def build_prompt(question: str, passages: list[dict], stills: list[dict]) -> str:
    passage_lines = '\n'.join(f"{row['title']}: {row['text']}" for row in passages)
    still_lines = '\n'.join(row['caption'] or 'untitled still' for row in stills)
    return (
        'Answer using only the retrieved passages and still captions.\n\n'
        f'Passages:\n{passage_lines}\n\n'
        f'Stills:\n{still_lines}\n\n'
        f'Question: {question}'
    )

class Questions(TableModel, name='questions'):
    question: pxt.String
    passages = retrieve_passages(question, limit=5)
    stills = retrieve_stills(question, limit=5)
    prompt = build_prompt(question, passages, stills)
    answer = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    ).choices[0].message.content

# pxt init
# pxt schema update app.py multimodal_rag -f
```

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. [Get started](https://www.pixeltable.com/get-started) covers the CLI.

## Questions {#questions}

### Do I need Qdrant or pgvector if I use Pixeltable? {#faq-1}

You need an embedding index so similar passages and stills can be ranked. That index can live on the chunk view and the image table. Keep Qdrant, pgvector, Pinecone, Weaviate, or LanceDB when that engine is already the serving contract or you need BM25 fusion, namespaces, or a hosted ANN SLA.

### Where do the files live if I already use S3? {#faq-2}

Keep S3 (or R2, GCS, MinIO) as the object store and insert the URLs into typed columns. Pixeltable caches media locally under PIXELTABLE_HOME when it needs to read a file. The docs cookbook is Load media from S3.

### Is this a replacement for LangChain? {#faq-3}

No. Retrieval is a query on the table. LangGraph can still run an agent loop that calls that query as a tool. Pixeltable vs LangChain is that split: data plane versus agent loop.

## Related

- [How do I build a multimodal RAG pipeline in Python?](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline)
- [How do I chunk PDFs for RAG in Python?](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag)
- [What is multimodal RAG?](https://www.pixeltable.com/learn/what-is-multimodal-rag)
- [Production RAG implementation](https://www.pixeltable.com/use-cases/production-rag-implementation)
- [RAG demo notebook](https://www.pixeltable.com/notebooks/rag-demo)
- [Examples hub: PDF RAG](https://www.pixeltable.com/examples)
- [Pixeltable vs Pinecone](https://www.pixeltable.com/compare/pixeltable-vs-pinecone)
- [Pixeltable vs LanceDB](https://www.pixeltable.com/compare/pixeltable-vs-lancedb)
- [Build a RAG pipeline (docs)](https://docs.pixeltable.com/howto/cookbooks/agents/pattern-rag-pipeline)
- [Split documents into chunks (docs)](https://docs.pixeltable.com/howto/cookbooks/text/doc-chunk-for-rag)

This is the architecture. Step-by-step Python is on [how-to guides](https://www.pixeltable.com/learn/how-to) and [examples](https://www.pixeltable.com/examples).