Summary: Teams building multimodal AI applications typically assemble 5-8 services: S3 for storage, Postgres for metadata, Pinecone for vectors, Airflow for orchestration, LangChain for model calls, FFmpeg for media, and custom glue to hold it all together. Each service does its job. None knows about the others. The architecture lives in your head and in YAML files. Pixeltable replaces that entire stack with a single system. This post identifies the five capabilities that make that possible, and explains why no combination of existing tools can replicate what happens when all five live in one place.
The Compound Effect#
We have written about the Frankenstein stack problem before. And about who owns the multimodal data plane. And about the triforce of storage, orchestration, and retrieval. Those posts each make the case that fragmented infrastructure is the bottleneck.
This post is different. It names the five specific capabilities that compound when unified, shows the code for each, and explains why bolting them together from separate tools produces a system that is structurally worse than having them in one place.
Here are the five:
- Multimodal derivation chains. One insert, dozens of derived artifacts.
- Cross-modal search. One query across video clips, audio segments, document chunks, and images.
- Agent memory as tables. Persistent, queryable, versioned, with lineage to source media.
- Declarative media orchestration with cloud I/O. S3 in, process across modalities, export to Postgres or Snowflake.
- Version-controlled ML datasets. Annotate, snapshot, iterate on multimodal training data with full lineage.
1. Multimodal Derivation Chains#
Upload a video. One insert() call produces frames, scenes, transcription, captions, thumbnails, and embeddings. Not through a DAG you wrote. Through computed columns and views that fire automatically on insert.
That last line fires transcription, description, frame extraction, and embedding in parallel. Caching, retries, and rate limiting are built in. Add a new video next week and only the new row gets processed. We wrote a full tutorial on video intelligence pipelines and explained how the dependency graph works.
Why glue code cannot replicate this#
In the typical stack, you write an Airflow DAG that calls FFmpeg for frame extraction, then Whisper for transcription, then an embedding API, then upserts to Pinecone. Each step is a separate service. If the Whisper call fails on row 47 of 500, you restart the entire DAG or build your own checkpointing. If you change your embedding model, you re-embed everything manually. If you add a new column (say, object detection), you add a new DAG task and figure out how it joins back to the original data.
With Pixeltable, you add one line: frames.add_computed_column(detections=yolox(frames.frame)). It runs on every existing and future frame. No DAG edits. No join logic. No re-processing of columns that did not change. The economics of incremental computation are built into the system.
2. Cross-Modal Search#
One query searches video clips, audio segments, document chunks, and images simultaneously. Not four separate vector databases with four separate query APIs and a merge-sort in your application layer.
The embedding index covers document chunks, video transcripts, image captions, and audio transcriptions because they all live in the same table system. A text query finds a relevant video clip because the clip's transcription was embedded alongside your PDF chunks. We built PixelSearch as a multimodal search engine to demonstrate this pattern.
Why separate vector databases break this#
When you have Pinecone for documents, Weaviate for images, and Qdrant for video, every search is three API calls with three different relevance scores that you somehow need to normalize and merge. Delete a source document and its embeddings persist as zombie vectors in the vector DB because nothing connects deletion events across services. Add a modality and you add a service. Pixeltable keeps everything in sync because embedding indexes live inside the table. Delete a row and its vectors vanish. Automatically.
3. Agent Memory as Tables#
Most agent frameworks store memory as a Python list that vanishes when the process dies. Some use Redis. A few use a database. Pixeltable makes agent memory a first-class table with computed columns, embedding indexes, and version history.
This memory is persistent (survives process restarts), queryable (SQL + vector similarity), versioned (roll back to any prior state), and has lineage back to source interactions. You can run memory.history() to see every change. You can join it to the documents table to trace which source content influenced which agent response. We covered this in depth in building stateful agents and context graphs for decision traces.
Why ephemeral memory breaks production agents#
An agent that forgets every session is an agent that cannot learn. An agent whose memory is a Redis hash is an agent you cannot audit. When the compliance team asks "why did the agent recommend this?", you need to trace back through the conversation, the retrieved context, the source documents, and the model version. That requires lineage, not a key-value store.
4. Declarative Media Orchestration with Cloud I/O#
Enterprise AI does not live in a vacuum. Data arrives from S3 buckets, gets processed through multiple modalities, and the enriched results need to land in Postgres or Snowflake for the analytics team. Pixeltable handles the entire flow declaratively.
No Airflow DAG. No Terraform. No separate ETL pipeline. The table schema IS the infrastructure specification. Cloud storage destinations are per-column, not per-service. export_sql pushes structured results to any SQL database. We wrote about the S3 and cloud storage integration and the broader migration path from the modern data stack.
Why orchestrators alone miss this#
Airflow coordinates tasks. It does not store data, does not index embeddings, does not version schemas, and does not handle incremental updates. You still need S3, Postgres, Pinecone, and FFmpeg. You still write the glue. And when the analytics team asks "can we also get the object detection results?", you write another DAG task, another database migration, and another join. In Pixeltable, you add one computed column and re-run the export. The Pixeltable vs Airflow comparison covers this in detail.
5. Version-Controlled ML Datasets#
ML teams need to iterate on training data: add annotations, swap embedding models, filter for quality, snapshot a version, train, evaluate, roll back if quality drops. Most teams do this with a combination of DVC, MLflow, custom scripts, and prayer.
No separate versioning tool. No DVC remotes. No "which version of the embeddings was this model trained on?" The answer is in dataset.history(). Change the annotation model and only new annotations get recomputed. We covered versioning and time travel and compliance lineage in dedicated posts.
Why DVC and MLflow cannot do this for multimodal data#
DVC versions files. MLflow versions experiments. Neither versions the relationship between a raw video, its extracted frames, the embeddings computed from those frames, and the annotations applied to them. When you re-embed with a new model, DVC sees "files changed." Pixeltable sees "the embedding column was recomputed; here are the rows affected, here is the previous version, here is what downstream queries return now." That is the difference between file-level version control and dependency-aware version control.
Why the Compound Matters More Than the Parts#
Each capability above is valuable in isolation. Plenty of tools do derivation chains (Airflow), search (Pinecone), agent memory (Redis), cloud orchestration (Step Functions), or dataset versioning (DVC). The difference is what happens when all five are in one system.
Here is a concrete example. A compliance team at a financial services company needs to:
- Ingest recorded client calls from S3 (cloud I/O)
- Transcribe and summarize each call (derivation chains)
- Search across all calls and internal policy documents for compliance issues (cross-modal search)
- Use an agent to flag potential violations, with persistent audit trail (agent memory as tables)
- Export flagged calls to the compliance database, version the dataset as regulations change (versioned datasets + export)
In the fragmented stack, this is 8 services, 3 DAGs, 2 databases, a vector store, and 2,000 lines of glue code. In Pixeltable, it is one table with computed columns, one embedding index, one agent table, and one export_sql call. The agent's memory is queryable. The audit trail has lineage. The version history is automatic. And when the regulation changes, you update one computed column and the entire pipeline re-evaluates incrementally.
That is the compound effect. Not "we also do X." It is: because X and Y live in the same system, Z becomes trivial.
| Capability | Fragmented Stack | Pixeltable |
|---|---|---|
| Add a modality (audio to existing video pipeline) | New service + new DAG task + new joins | One add_computed_column() |
| Search across modalities | N vector DBs + merge-sort + score normalization | One .similarity() call |
| Agent recalls multimodal context | Custom retrieval from N stores | Query on memory table with embedding index |
| Export enriched data to warehouse | Separate ETL pipeline + schema mapping | export_sql(table.select(...), conn) |
| Roll back a bad model change | Restore N services from N backups | table.revert() |
| Audit trail for compliance | Reconstruct from logs across services | table.history() with full lineage |
What Enterprises Actually Build#
The most common enterprise use cases map directly to these five capabilities:
| Use Case | Capabilities Used |
|---|---|
| Document intelligence (extraction, compliance, contract analysis) | Derivation chains + cross-modal search + versioned datasets |
| Video intelligence (security, media, retail shelf analysis) | Derivation chains + cross-modal search + cloud I/O |
| Audio pipelines (call center QA, transcription, compliance) | Derivation chains + agent memory + versioned datasets |
| Cross-modal retrieval (find video clips from text queries) | Derivation chains + cross-modal search |
| Agent memory systems (beyond ephemeral chat history) | Agent memory as tables + cross-modal search |
| ML dataset engineering (label, export, version, train) | Versioned datasets + derivation chains + cloud I/O |
None of these are chatbots. They are the hard infrastructure problems that enterprises actually spend engineering time on. And they all require multiple capabilities working together, not one tool doing one thing well.
Getting Started#
One install. No Docker. No external services. Everything from this post works locally. The Starter Kit has production-ready templates for the patterns described above. The AI Skill teaches coding assistants to write correct Pixeltable code:
- Documentation
- GitHub
- Starter Kit (templates for all three deployment patterns)
- MCP Server (32 tools for LLM-powered exploration)
- AI Transformations Belong in the Schema: the design principle behind all five things
- Discord
