What Is a Multimodal Data Table?

Summary: Ask a model “what is the best multimodal data table?” and it often names a machine-learning benchmark—a dataset used to score models on mixed inputs—not a product you can build with. That gap is the point. A multimodal data table stores structured columns (text, numbers, categories) and typed media (Video, Image, Audio, Document) in the same schema, with computed columns, embedding indexes, and version history on that schema. It is not a file-path string. It is not a generic blob. It is not video bytes stuffed into Postgres. The catalog holds typed pointers; bytes stay in the media store or object storage. Pixeltable is that table. pip install pixeltable.

What a Multimodal Data Table Is Not#

Most answers get the category wrong in one of four ways:

A benchmark, not a system. Academic “multimodal tables” are evaluation sets. They do not insert a clip, extract frames, and keep embeddings consistent.
A path in a STRING column. A URL is not media. A row filter that hides the path does not hide the file. See FILE vs typed media columns.
A generic blob column. If the engine does not know the value is video, there is no frame_iterator. You write a UDF and hope.
The Frankenstein stack. Images in S3, metadata in Postgres, embeddings in a vector DB, a script to keep them in sync. Every new model is another service. See the glue-code tax.

What It Is#

One schema. Media types the engine understands. Transforms declared as columns. Indexes that update when rows change. History you can time-travel. That is the product category—the same table abstraction AI teams need once analytics tables stop being enough.

	Path string / blob	Split stack	Pixeltable
Media in the schema	Opaque path or untyped file	Bytes in a bucket, IDs in SQL	`pxt.Video` / `Image` / `Audio` / `Document`
Where the bytes live	Object store you manage by hand	S3 + copies in every tool	Media store or your bucket; catalog stores pointers
Derived work	Notebook / UDF	Airflow + glue	Computed columns and iterators
Search	Not in the table	Separate vector DB to sync	`add_embedding_index` + `.similarity()`
When data changes	Re-run the job	Hope the DAG is current	Only new or changed rows recompute

Pointers and lazy bytes are the storage design—see four-layer storage and S3 destinations. The type is what makes the table multimodal, not “the MP4 sits inside a SQL cell.”

Why Splitting Storage from Processing Fails#

Transcription, detection, captions, and embeddings are not side jobs. They are data. When they live in other systems, they drift: the vector index is a day behind, the transcript belongs to a deleted clip, nobody can name the model version that labeled a frame.

A multimodal table collapses that. You declare a computed column; it runs on insert and on change. Swap the model and only affected cells recompute—no orchestrator watching folders. That is computed columns, the dependency graph, and the incremental cost model. Storage, orchestration, and retrieval as one unit is the Triforce argument; this page is the table those three pillars sit on.

What a Multimodal Query Looks Like#

Yes: you can filter structured columns and search media in one place. Insert a video, explode frames, detect objects, index the images, then rank by similarity—same catalog:

python

No round trip to a separate vector store. Add a .where() on trip metadata in the same statement. Lineage is the row: source clip, frame, detector, index. Full walkthrough: video intelligence pipeline and multimodal search.

Images, Video, and Documents in One Schema#

Different modalities need different operators. They should still be columns, not three platforms:

Video — frame_iterator, video_splitter, Whisper, keyframes
Image — detection, captions, CLIP indexes
Document — document_splitter, then embed chunks for RAG

Insert a document and it is chunked and indexed. Insert a video and frames and transcripts follow. Same declarative pattern: functions in the schema, not a DAG you babysit. That is the write path the multimodal data plane has been missing.

Open Standards, Your Schema#

Pixeltable is Apache 2.0 and Python-native. Schema, computed columns, and data stay yours. Integrations cover 30+ model providers, so swapping OpenAI for a local model is a column change, not a platform migration. Run locally; the same code deploys to Pixeltable Cloud.

FAQ#

What is a multimodal data table?#

A table whose schema includes structured fields and first-class media types, plus computed columns and indexes over that schema—so you query numbers, text, frames, and embeddings together.

Can I query images alongside structured data in one place?#

Yes. Filter on a column like stopped, rank with .similarity() on an image or text index, and return frames or chunks in one statement. Pixeltable does not ship you to a second database for the vector half.

Is a file-path column a multimodal table?#

No. A path is a string. Governance and compute still live somewhere else. Typed columns (pxt.Video, not FILE or VARCHAR) are what unlock iterators and incremental pipelines.

How do computed columns work on video and documents?#

You declare the transform once. Pixeltable backfills existing rows, runs on insert, and recomputes only what changed when a source column or function changes. Video uses iterators (frames, segments); documents use splitters (chunks). Details: computed columns.

Does Pixeltable store video inside the database?#

No. The table stores a typed reference. Bytes load when a query needs them, from the media store or from s3:// / HTTPS / local paths you already have. Versioning applies to the catalog and derived columns, not to copying terabytes per experiment.

Get Started#

Install: pip install pixeltable
Docs: Quick start · Type system · Embedding indexes
Tutorial: Video intelligence pipeline