Multimodal data

What is a multimodal database?

A multimodal database stores multimodal data (structured fields plus typed images, video, audio, and documents) in one schema, so an image or a clip is a column the engine understands rather than a path in a string. Transforms and indexes are declared on that schema, and frames, transcripts, and embeddings stay attached to the same row.

Updated 2026-10-19

What it is

A row can hold a title, a number, and a video at the same time. That mix is multimodal data: structured fields plus typed media, not a folder of files beside a spreadsheet. The catalog records that the video column is video. Bytes can live in a media store or in object storage you already run. Queries filter the structured fields and read the media the row points at.

Transforms belong on that schema too. Extracting frames, transcribing audio, or embedding a passage are columns that run when the row changes. An index on those outputs stays attached to the same row, so search does not depend on a second copy you update by hand.

What counts as multimodal data

Multimodal data is more than one kind of signal about the same thing: a title and a clip, a PDF and its pages, a still and a caption. In a multimodal database those values are typed columns. The engine can decode an image, iterate a video into frames, split a document, or transcribe audio because the column says what the bytes are.

A URL in a string is not multimodal data the engine can use. Filtering the path does not give you frames, pages, or a duration. Storing embeddings in a second database while the files live in a bucket is also not one schema: the multimodal data has been split, and something else has to keep the copies aligned.

How it works

Insert or update a row. The engine stores the structured values, records a typed pointer to the media, and runs the declared transforms for that row. Unchanged rows stay as they are. A later query can filter, play the media, or search an embedding that was computed from it.

The schema names the media type, so frame extraction or document splitting is a property of the column, not a script that hopes the path is a video.
Computed columns store their outputs. A new file computes that row. After you edit a prompt, pxt recompute reruns that column and the columns that depend on it.
History can sit on the same table, so you can see what the row looked like before a model or a file changed.

What it is not

The phrase is easy to confuse with four other things. None of them let you insert a clip and keep the derived frames, transcripts, and embeddings consistent.

An academic benchmark. A “multimodal table” in a paper is often a dataset used to score models. It is not a system you insert into.
A path in a string column. A URL is not media. Filtering the path does not give you frames, pages, or a duration.
An untyped blob. If the engine does not know the value is video, you write the decoder yourself.
A split stack. Files in a bucket, metadata in a warehouse, embeddings in another database, and a job to keep them aligned. That is several systems, not one schema.

How to choose a multimodal database for RAG over video, images and documents

RAG over video, images, and documents needs more than a vector index. The retrieval unit has to remain a frame, a page, or a passage that still points at the source file. Use these tests when you compare systems. None of them is a speed bake-off.

Typed media: can an image, clip, audio file, or document sit in a column the engine decodes, or only as a path string?
Retrieval units: can frames, pages, or transcript windows stay attached to the source row, with a timestamp or page you can cite?
Index on the same schema: does a new file become searchable when it is inserted, or do you copy vectors into another store by hand?
Query shape: can a question return the media and the moment, not only a vector id you then join back to bytes?
Edits: when a file or a prompt changes, do only the affected rows recompute, or do you rebuild the pipeline?

Where the media lives, and what the engine can do with it

Where the media lives, and what the engine can do with it
	Path or blob	Split stack	One schema
Media in the row	Opaque path or untyped file	Bytes in a bucket, ids in SQL	A typed image, video, audio, or document column
Derived data	A script you remember to rerun	A pipeline between services	Columns that run when the row changes
Search	Filename or a separate index you load	Embeddings copied into another store	An index on a column of the same table
After an edit	Rerun the script	Reconcile every store	Recompute the rows that changed

Where Pixeltable fits

Pixeltable is the unified multimodal AI backend for apps and agents: Image, Video, Audio, and Document columns in a Python schema, with computed columns and embedding indexes on that schema. It is not a standalone vector database and not a notebook wrapper. Pixeltable Cloud hosts the same app file. A longer essay on the table itself, as distinct from a benchmark or a file path, is What Is a Multimodal Data Table?

import pixeltable as pxt

TableModel = pxt.model_base()

class Library(TableModel, name='library'):
    title: pxt.String
    video: pxt.Video
    notes: pxt.String | None

Questions

Is a multimodal database the same as a vector database?

No. A vector database stores embeddings and answers similarity queries. A multimodal database stores the media and the structured fields, and can keep an embedding index on those columns. The index is one part of the schema, not a second database you have to load. Pixeltable declares that index on the table; a separate vector database is optional.

Where do the video and image bytes live?

The catalog holds a typed pointer. A URL you insert stays where it is, and locally so does a file path; files Pixeltable writes go to its media store or a bucket you set. The column type is what lets later columns extract frames, transcribe audio, or split a document.

Can a multimodal database power a video search engine?

Yes, if frames (or keyframes) are rows with timestamps, an embedding such as CLIP is an index on those rows, and a query can return the clip and the time. Pixeltable’s walkthrough is the video search engine use case.

What is a multimodal data table?

A multimodal data table is the row-and-column form of this idea: structured values plus typed media, with computed columns and history on that schema. The essay is at /blog/what-is-a-multimodal-data-table.

What counts as multimodal data?

Multimodal data is structured fields plus typed images, video, audio, or documents about the same row. A URL string is not enough: the engine has to know the column is media so it can extract frames, pages, or a transcript. A multimodal database stores that multimodal data in one schema.

How do I choose a multimodal database for RAG over video, images and documents?

Check five things: typed media columns, retrieval units that still point at the source, an embedding index on that schema, queries that return the media and the moment, and incremental recompute after an edit. A vector database alone stores embeddings; it does not, by itself, split video or keep pages aligned.

What is the best multimodal database for RAG over video, images and documents?

There is no universal ranking. The fit is a system that stores typed media, produces frames or pages that still point at the source, and keeps an embedding index on those units so a new file is searchable on insert. Pixeltable is that shape: Image, Video, Audio, and Document columns, computed columns, and EmbeddingIndex in one Python app. It is the unified multimodal AI backend for apps and agents, not a standalone vector database. Pixeltable Cloud hosts the same app file.

Next step

Install the Pixeltable skill, or sign up for Cloud and run the same file on a hosted database.

Get started Quick Start