Summary: Building multimodal AI is an experimental process. Which chunking strategy? Which embedding model? What retrieval threshold? What prompt? These decisions compound, and you can only make good ones with fast iteration and comparable results. But when your data lives across 5+ disconnected services, nothing is captured consistently. There's no versioning across the pipeline and no way to compare runs. This post shows what a real experimentation loop looks like when storage, orchestration, and retrieval are unified.
The Hidden Cost of Fragmented Infrastructure#
When we talk to teams building multimodal AI applications, the first complaint is always about complexity: too many services, too much glue code, the Frankenstein Stack.
But the deeper you dig, the real pain isn't operational. It's that they can't iterate.
Building a multimodal AI application isn't a build-once-deploy-forever exercise. It's an experimental process with dozens of decisions:
- How should I chunk these documents: by sentence, by token count, with overlap?
- Which embedding model gives the best retrieval quality for my domain?
- What similarity threshold balances recall vs. precision?
- Which prompt gets the most useful LLM output?
- Does adding object detection to my video frames improve search quality?
Each of these decisions compounds. The chunking strategy affects the embeddings, which affect retrieval, which affects the context the LLM sees, which affects output quality. You can only make good decisions with fast iteration and comparable results.
Why Iteration Is Broken Today#
In the typical multimodal stack (S3 for media, Postgres for metadata, Pinecone for vectors, Airflow for orchestration, LangChain for retrieval), the experimentation loop is fundamentally broken:
1. No Versioning Across the Pipeline#
Each system versions independently, if at all. S3 has object versioning. Postgres has transactions. Pinecone has no built-in versioning. There's no way to say "show me the state of everything at 3pm Tuesday" (the version of your chunks, the embedding model used, the retrieval results, and the LLM output) as a coherent snapshot.
2. No Way to Compare Runs#
If you change your chunking strategy, you re-run your entire pipeline and eyeball the results. There's no queryable system that lets you say: "show me retrieval quality with 300-token chunks vs. 500-token chunks, same queries, same embedding model." You'd have to build that comparison infrastructure yourself.
3. Full Reruns on Every Change#
Changed the embedding model? Re-embed everything. Changed the chunking? Re-chunk, re-embed, re-index. Changed a prompt? Re-run every LLM call. The economics are brutal: most of the compute is wasted reprocessing data that didn't change.
4. Infrastructure Eats the Experiment Budget#
The time your team spends writing sync scripts, debugging stale indexes, and managing pipeline failures is time they're not spending on the experiments that actually improve your product. The infrastructure becomes the bottleneck, not the AI.
The Experimentation Loop in a Unified System#
When storage, orchestration, and retrieval are unified, the experimentation loop transforms. Here's what it actually looks like in Pixeltable:
Step 1: Build Your Baseline#
Start with a table, some computed columns, and an embedding index. This is your baseline: everything is stored, versioned, and queryable from the start.
Everything is now stored, versioned, and queryable. Check the current version:
Step 2: Test Before You Commit#
This is where it gets powerful. Before changing your pipeline, you can test on a sample: nothing is stored, and API calls are parallelized:
The key insight: sample(n=5).select(...) is ephemeral. It doesn't modify your table. It doesn't store results, so each run calls the model again for the rows it samples.
Step 3: Commit When You're Happy#
Once you've validated the improvement, commit it as a computed column. It computes every row, the 5 sampled ones included, since the sample stored nothing. From then on, only new rows call the model:
Step 4: Swap Models and Compare#
Want to try a different embedding model? Drop the old index and add a new one. Pixeltable embeds every existing chunk with the new model, then keeps the index current as rows arrive:
Step 5: Compare Against Any Prior Version#
Every change is automatically versioned. You can query any prior state:
Why This Only Works in a Unified System#
Every step above is trivial individually. The reason it's transformative is that it all happens in the same system:
- Sample, test, commit: works because storage and compute are unified. The sample query, the UDF test, and the committed column all operate on the same data through the same interface.
- Model swapping: works because the embedding index is part of the table, not a separate service. Drop and re-add is atomic, not a distributed operation across Pinecone + custom sync scripts.
- Time travel: works because versioning spans the entire pipeline, not just one layer. You're comparing the full state (chunks, embeddings, model outputs), not just one piece.
- Incremental updates: work because the system tracks the full dependency graph. It knows which embeddings depend on which chunks, and only recomputes what's affected.
- Stored results: work because results are persisted automatically. Once a column is committed, queries and new inserts never call the model again for rows already computed. The 5-row sample stores nothing, so committing computes those rows too.
In a separated stack, each of these would require custom engineering. Together, they form a fast, natural experimentation loop where the infrastructure gets out of your way.
The Multimodal Dimension: Where This Gets Even More Powerful#
The experimentation loop becomes even more valuable with multimodal data. Consider a video analysis pipeline:
You just went from basic frame extraction to a multi-modal searchable video pipeline, with every decision versioned and reversible. In a separated stack, each step would have been a separate service integration.
Real-World Impact#
This isn't theoretical. Teams using Pixeltable's experimentation loop report transformative changes:
"Pixeltable has transformed our computer vision workflow. Before Pixeltable, our engineers spent 80% of their time on data plumbing. Now, they can focus on what truly matters: building better models and delivering value. We've reduced our infrastructure code by 90%, cut compute costs by 70%, and dramatically accelerated model iteration cycles."
"Pixeltable has transformed how our ML engineers spend their time. We have a highly multimodal use case, and Pixeltable gives us full visibility into the input data, models, and incremental steps of our system's pipeline."
Because everything is declarative and versioned, anyone on the team can reproduce the full workflow without rebuilding anything. No tribal knowledge. No "it works on my machine."
The Loop, Summarized#
| Step | Separated Stack | Pixeltable |
|---|---|---|
| Build baseline | Configure 5+ services, write glue code | Define tables, computed columns, indexes |
| Test a change | Re-run entire pipeline on a fork | table.sample(n=5).select(...) |
| Commit | Update scripts, re-run, hope sync works | add_computed_column(): every row computed and stored |
| Compare | Manual diffing across services | Query current vs. prior version |
| Roll back | Restore backups across 3+ systems | table.revert() |
| Share results | Write docs, export CSVs, screenshot dashboards | Colleague queries the same versioned table |
Stop Fighting Infrastructure. Start Experimenting.#
The teams that win in multimodal AI aren't the ones with the most GPUs or the fanciest models. They're the ones that can iterate fastest: trying more ideas, comparing results, and converging on what works.
That's only possible when your infrastructure gets out of the way. When every decision is versioned, every result is queryable, and every change propagates incrementally, you stop fighting infrastructure and start experimenting.
Iterate on your data, not your infrastructure. pip install pixeltable.
Ready to see the experimentation loop in action? Start with the 10-Minute Tour, explore the iterative workflow cookbook, or try the embedding indexes notebook. Everything is open source under Apache 2.0.
Further Reading#
- AI Transformations Belong in the Schema: the architectural mechanism that makes this experimentation loop possible
- The Economics of Incremental AI: why incremental recomputation matters for cost
- Schema-Driven Infrastructure: the broader thesis on schema as infrastructure specification
