Reference architecture

Multimodal RAG reference architecture

Multimodal RAG retrieves passages, stills, or frames, then generates from those rows. The index is one piece. Object storage, a parser, a vector engine, and an LLM API are slots you can keep or fold into one TableModel file.

Updated 2026-10-28 · Matching docs cookbook

A multimodal RAG system has five jobs: hold the files, turn them into retrieval units, embed those units, rank them for a question, and generate from the ranked rows. Vision RAG is the image half of that job. Text RAG is the passage half. A multimodal pipeline declares both.

This page is the architecture, not a tutorial. The Python how-to is How do I build a multimodal RAG pipeline in Python. The docs cookbook is Build a RAG pipeline. Use this page to name each slot and to keep Qdrant, pgvector, or S3 when that engine is already the contract.

Multimodal RAG: files become retrieval units, then a grounded answer

Role of each component

Source files: PDFs, images, and later frames or transcripts. They stay addressable so a hit can cite a page or a still, not only a vector.
Typed store: A Document column and an Image column, or an object store plus a path table. The store must know the type so splitters and embedders can read it.
Splitter: Turns one PDF into passage rows tied to the parent file. sentence,token_limit is the documented default. Page metadata needs separators that include page.
Embedding indexes: Nearest-neighbor search on passage text and on stills. CLIP places a sentence and a picture in one space so a question can retrieve either.
Retriever: A query that ranks rows and returns the text or caption the model is allowed to see. Agentic RAG exposes the same query as a tool.
Generator: An LLM call whose prompt is built from the retrieved rows. The model does not search the corpus itself.

Honest alternatives per slot

Keep the named engine when it is already the contract. Pixeltable fills the slot when you want that job on the same table as the files.

Named alternatives for Multimodal RAG reference architecture
Slot	Role	Pixeltable	Named alternatives
Object store	Bytes for PDFs and images	Media columns under PIXELTABLE_HOME locally; Cloud hosts the same app file	Amazon S3. The bucket is already the system of record, or another fleet reads it Cloudflare R2. You want S3-compatible object storage with no egress fee on that vendor MinIO. You need an S3 API inside your network GCS or Azure Blob. That cloud is already where the files live
Document parser	Text and layout from PDFs	pxt.Document plus document_splitter on a view	Unstructured. You need layout-aware parsing before the rows exist LlamaParse. Complex PDFs with tables that a splitter alone misreads Apache Tika. Office formats in an existing JVM pipeline
Vector index	Nearest-neighbor search over passages and stills	pxt.EmbeddingIndex on the chunk view and the image column	Qdrant. You already run a dedicated ANN service with collections and filters pgvector. Postgres is already the source of truth and you want vectors next to SQL Pinecone. A hosted ANN SLA is the product, not the table Weaviate. You need BM25 plus dense fusion in one engine LanceDB. You want a Lance file layout and an embed-on-write API
Orchestration	Chunk, embed, and reindex on insert	Computed columns and iterator views; insert updates the affected rows	Airflow. You already schedule unrelated DAGs across many systems Prefect or Temporal. The job is long-running workers with their own retry policy
LLM API	Generate from retrieved rows	chat_completions as a computed column, or a later prompt you own	OpenAI, Anthropic, Gemini, Groq. Any of these as the model; Pixeltable is not the model Ollama or vLLM. The weights run in your process or cluster

Where Pixeltable fits

Pixeltable is the unified multimodal backend for this architecture: Document and Image columns, a document_splitter view, CLIP on stills, and @pxt.query retrieval in one Python file. Keep S3 when the bucket is already the system of record (load media from S3). Keep Qdrant, pgvector, Pinecone, Weaviate, or LanceDB when that engine is the serving index. Pixeltable vs Pinecone, Pixeltable vs Weaviate, and Pixeltable vs LanceDB are that split. A separate vector database is optional when the index can live on the tables that hold the files.

import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.functions.openai import chat_completions

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Docs(TableModel, name='docs'):
    title: pxt.String
    document: pxt.Document

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=300
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def retrieve_passages(question: str, limit: int = 5):
    sim = Chunks.text.similarity(string=question)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(text=Chunks.text, title=Docs.title, score=sim)
    )

@pxt.query
def retrieve_stills(question: str, limit: int = 5):
    sim = Images.image.similarity(string=question)
    return (
        Images.order_by(sim, asc=False)
        .limit(limit)
        .select(caption=Images.caption, score=sim)
    )

@pxt.udf
def build_prompt(question: str, passages: list[dict], stills: list[dict]) -> str:
    passage_lines = '\n'.join(f"{row['title']}: {row['text']}" for row in passages)
    still_lines = '\n'.join(row['caption'] or 'untitled still' for row in stills)
    return (
        'Answer using only the retrieved passages and still captions.\n\n'
        f'Passages:\n{passage_lines}\n\n'
        f'Stills:\n{still_lines}\n\n'
        f'Question: {question}'
    )

class Questions(TableModel, name='questions'):
    question: pxt.String
    passages = retrieve_passages(question, limit=5)
    stills = retrieve_stills(question, limit=5)
    prompt = build_prompt(question, passages, stills)
    answer = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    ).choices[0].message.content

# pxt init
# pxt schema update app.py multimodal_rag -f

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. Get started covers the CLI.

Questions

Do I need Qdrant or pgvector if I use Pixeltable?

You need an embedding index so similar passages and stills can be ranked. That index can live on the chunk view and the image table. Keep Qdrant, pgvector, Pinecone, Weaviate, or LanceDB when that engine is already the serving contract or you need BM25 fusion, namespaces, or a hosted ANN SLA.

Where do the files live if I already use S3?

Keep S3 (or R2, GCS, MinIO) as the object store and insert the URLs into typed columns. Pixeltable caches media locally under PIXELTABLE_HOME when it needs to read a file. The docs cookbook is Load media from S3.

Is this a replacement for LangChain?

No. Retrieval is a query on the table. LangGraph can still run an agent loop that calls that query as a tool. Pixeltable vs LangChain is that split: data plane versus agent loop.