Summary: Every era of data has produced an infrastructure owner. Relational data got Oracle. Structured analytics got Snowflake. Batch processing got Databricks. But the fastest-growing category of enterprise data (multimodal AI workloads involving video, audio, images, and documents) has no infrastructure owner. The tools exist, but they don't compose. This post explores why the gap exists, why current tools can't close it, and what the multimodal data plane actually needs to look like.
Every Era of Data Gets an Owner#
The history of data infrastructure follows a pattern. A new category of workload emerges, existing tools get stretched beyond their design, and eventually a purpose-built system captures the category:
- Relational data (1980s-90s): Oracle and then Postgres became the systems of record for transactional workloads.
- Structured analytics (2010s): Snowflake separated storage from compute and owned the cloud data warehouse.
- Batch processing (2010s): Databricks productized Spark and became the default for large-scale data engineering and ML training pipelines.
Each of these systems succeeded because they didn't just solve one problem. They provided operational integrity across the full lifecycle. Snowflake doesn't just store data; it handles access control, versioning, compute scaling, and governance. Databricks doesn't just run Spark jobs; it manages notebooks, lineage, model registries, and feature stores.
Now look at multimodal AI. Video, audio, images, and documents are the fastest-growing data categories in enterprise AI. Every company building AI applications is touching this data. And yet: there is no equivalent infrastructure owner.
The Multimodal Gap#
It's not that tools don't exist. The problem is that they don't compose into a unified system that provides operational integrity. Here's the current landscape:
- Vector databases (Pinecone, Weaviate, Qdrant) store embeddings and do similarity search. But they only handle one derived artifact: the embedding. They know nothing about the raw video, the frames, the transcripts, or the pipeline that produced the embedding. They're indexes, not systems of record.
- Orchestrators (Airflow, Prefect, Temporal) coordinate task execution. But they coordinate existing systems. They don't unify them. You still need separate storage, separate compute, separate state management. The orchestrator is the glue, not the foundation.
- LLM frameworks (LangChain, LlamaIndex) make it easy to wire up retrieval and generation. But they're stateless by design. They don't persist data, don't version anything, don't handle incremental updates. Every run starts from scratch.
- Object storage (S3, GCS) stores the raw bytes. But it's a dumb filesystem: no schema, no transforms, no lineage, no retrieval.
- Relational databases (Postgres, MySQL) handle metadata. But they can't natively process a video, extract frames, transcribe audio, or manage embeddings. Media is an opaque blob.
The result? Every team building a multimodal AI application ends up stitching together 5-8 services: S3 for media, Postgres for metadata, Pinecone for vectors, Airflow for orchestration, LangChain for retrieval, FFmpeg for video processing, Whisper for transcription, all held together by custom glue code. This is the Frankenstein stack, and it's the norm.
Why Video Breaks Everything#
If you want to understand why the multimodal data plane needs an owner, look at video. Video is the hardest data type to wrangle at scale, because a single video generates every other data type:
- Audio: you extract the audio track, chunk it into samples, and transcribe it. Now you have text.
- Images: you extract keyframes, generating hundreds or thousands of images per video.
- Metadata: you run object detection, scene classification, face recognition, or scene detection on those frames. Now you have structured annotations.
- Embeddings: you embed the frames, the audio chunks, and the transcript segments into vector space for similarity search.
- LLM outputs: you feed all of this into an LLM for summarization, analysis, or agentic workflows.
One input, five derived data types, each requiring different processing, different storage, and different retrieval patterns. And you need lineage between all of them: which transcript came from which audio segment, which came from which clip, which came from which source video.
Teams building video pipelines today (in media, security, healthcare, autonomous vehicles) are stitching together 5+ services to handle this. It's fragile, hard to iterate on, and nearly impossible to put into production without a dedicated infrastructure team.
What the Owner of This Space Needs to Provide#
If you study what made Snowflake and Databricks succeed, it wasn't just features. It was operational integrity under change. The system handles the hard things so the user doesn't have to think about them.
The multimodal data plane needs the same. Specifically:
- Native multimodal types. Video, Audio, Image, and Document can't be opaque blobs. They need to be first-class column types with native operations: extract frames from a video column, transcribe an audio column, chunk a document column. These need to be data operations, not pipeline steps.
- Declarative transforms. Instead of writing imperative code to process each row, you should declare what you want computed. Computed columns that trigger automatically when data arrives, with incremental updates when upstream data changes.
- Unified storage and retrieval. One system for raw data, transformations, embeddings, and vector search. No synchronization between services. No manual index maintenance.
- Lineage and versioning. Full provenance: which rows used which model version? Which embeddings are stale? Which customer interaction came from which LLM call? This isn't a nice-to-have. It's a requirement in healthcare, finance, and security.
- Operational integrity. Retries, caching, rate limiting, type safety, error handling: all built in, not bolted on. One line of code should handle what currently takes hundreds of lines of glue.
This isn't a tooling problem. It's an architecture problem. You can't fix it by adding another tool to the stack. You need to rebase the stack itself.
Why Existing Players Can't Retrofit This#
The natural question is: why won't Snowflake or Databricks just add multimodal support?
They're trying. Snowflake has Cortex. Databricks has Unity Catalog with image support. But their architectures were designed for different workloads:
- Snowflake is OLAP: optimized for analytical reads over structured data. Multimodal AI workloads are OLTP: transactional writes, incremental recomputation, real-time retrieval. Different engine, different consistency model, different access patterns.
- Databricks is batch-first. Spark processes data in bulk. But multimodal AI pipelines need to process data as it arrives, maintain state between runs, and recompute incrementally when a prompt or model changes. Bolting this onto a batch system creates the worst of both worlds.
- Vector databases started as single-purpose indexes. Adding storage, orchestration, and lineage means rebuilding as a general-purpose system, something they weren't designed for.
- LLM frameworks are libraries, not infrastructure. They coordinate systems; they don't unify them. The gap between a Python library and a production data system is architectural, not incremental.
AI Makes It Easy to Ship, Not to Maintain#
There's a related shift happening that makes this even more urgent. With AI coding assistants and no-code builders, it's never been easier to ship an AI application. You can vibe-code a multimodal prototype in an afternoon.
But shipping isn't the hard part. Maintaining is. When your prototype becomes production, you need versioning, lineage, incremental updates, caching, rate limiting, error handling, and retrieval, all the things that glue code can't provide reliably.
This is why the infrastructure layer matters more than the intelligence layer. AI democratized the ability to write code, but it didn't democratize the ability to build robust data infrastructure. The infrastructure layer needs to make the right decisions on behalf of the user (for data management, orchestration, and retrieval) so that what ships can also sustain.
The Path Forward#
The multimodal data plane will get an owner. The workloads are too large, the pain is too acute, and the pattern from prior eras is too clear.
What will the answer look like? We think it starts where the developer is: a pip install that works locally, with a working multimodal app in minutes. Not a platform you sign up for. Not 5 cloud services you stitch together. A single system that unifies storage, orchestration, and retrieval under one table abstraction.
That's why we're building Pixeltable: declarative, multimodal-native data infrastructure where video, audio, images, and documents are first-class types, not opaque blobs. Where computed columns replace ETL pipelines. Where embedding indexes and RAG retrieval are built in, not bolted on. And where lineage, versioning, and auditability are native, because industries like healthcare and security can't afford anything less.
Structured data got its owner. Batch processing got its owner. Multimodal AI data is next.
Want to see what this looks like in practice? Start with the 10-Minute Tour, explore the video processing cookbooks, or dive into the RAG pipeline guide.
