Summary: Microsoft just open-sourced pg_durable, a PostgreSQL extension that runs durable, fault-tolerant workflows inside the database—no external orchestrator, no queue, no status-table glue. Its own README frames it as "part of our mission to bring compute close to data." That is not a Postgres-only idea. It is the same principle Pixeltable is built on, one layer up: instead of durable execution for SQL-shaped work, Pixeltable brings declarative, incremental compute close to multimodal data—video, audio, images, documents, embeddings, and model calls that SQL was never meant to express. pg_durable is a credible validation signal for where AI infrastructure is heading.
What pg_durable actually is#
pg_durable adds durable execution to PostgreSQL. You define a workflow as a graph of SQL steps; the extension checkpoints each step as it runs. If the database crashes, restarts, or a step fails, execution resumes from the last durable checkpoint instead of forcing you to reconstruct state by hand. As InfoQ reported, the goal is to let developers "avoid stitching together cron jobs, background workers, message queues, and external orchestrators."
The interface is a SQL DSL. Composable operators (~> for sequencing, |=> for binding a result to a variable) wire steps together, and df.start() kicks off an instance:
Architecturally it is intentionally minimal: a Postgres extension built with pgrx plus a background worker, with no external control plane. The worker hosts a Rust orchestration runtime (duroxide) backed by a PostgreSQL state provider (duroxide-pg) that persists instances, history, and work queues in dedicated schemas. Operational visibility comes from ordinary Postgres tables like df.instances, using the same auth and backup model as your data. It is currently in preview.
Source: microsoft/pg_durable · InfoQ coverage
Why this matters beyond Postgres#
Strip away the SQL specifics and pg_durable is making an architectural bet: the system that owns your data should also own the execution state of the work that touches it. Retries, progress tracking, checkpointing, and audit trails belong next to the data—not scattered across cron schedules, queue consumers, worker fleets, and bespoke status tables.
This is the durable-execution paradigm that Temporal and Cloudflare popularized as standalone runtimes, now collapsed into the database. When a vendor the size of Microsoft ships this as an open-source extension—and builds it into a managed cloud Postgres offering—it is a strong signal that "move the orchestration to where the state already lives" has gone from contrarian to mainstream.
We have made the same argument from the AI side for a while: the Frankenstein stack of S3 + Postgres + a vector DB + an orchestrator + a framework + glue code is not a foundation, it is technical debt with a diagram. pg_durable attacks one slice of that debt for relational workloads. The question for AI teams is what attacks it for multimodal workloads.
The same idea, one layer up: multimodal AI#
pg_durable is deliberately SQL-shaped. Its own "when not to use it" guidance is refreshingly honest: skip it when you need "arbitrary application logic that does not map cleanly to SQL steps," a non-HTTP SDK, "rich in-memory control flow," or workflows that "span many heterogeneous systems."
That list is an almost perfect description of a modern AI application. Consider the embedding pipeline pg_durable itself cites as a motivating example—chunk documents, call an embedding API, upsert into pgvector. In a real multimodal app that workflow is rarely just SQL:
- You ingest documents, video, audio, and images, not just rows.
- You run model inference—transcription, captioning, object detection, LLM calls—via Python SDKs, not HTTP-from-SQL.
- You chunk and iterate over media (frames, audio segments, document sections) into derived views.
- You maintain embedding indexes and run similarity search, then keep them in sync as source data changes.
SQL cannot natively process a video, extract keyframes, or transcribe audio—media is an opaque blob to a relational engine. That gap is exactly why the multimodal data plane has no owner, and it is where Pixeltable lives. Pixeltable applies the same "compute close to data" principle, but its unit of compute is the declarative computed column over multimodal tables, not a SQL step.
Side by side: durable SQL steps vs. declarative multimodal columns#
Take the embedding workflow both systems care about. In pg_durable you express it as durable SQL steps:
In Pixeltable you declare the outcome—a chunked view and an embedding index—and the engine computes, persists, and incrementally maintains it as new documents arrive:
The two systems rhyme but operate on different terrain:
| Dimension | pg_durable | Pixeltable |
|---|---|---|
| Unit of work | Durable SQL step in a function graph | Declarative computed column / view |
| Data types | Relational rows (plus pgvector) | Video, audio, images, documents, JSON, embeddings |
| How compute is defined | Imperative SQL steps with operators (~>, |=>) | Declarative outcome; engine derives execution |
| Recovery model | Checkpoint + deterministic replay | Incremental recompute of only changed cells |
| Model inference | HTTP from SQL (df.http()) | Native Python UDFs and 25+ provider integrations |
| Retrieval | pgvector queries you maintain | Built-in embedding indexes + similarity search |
| Language surface | SQL | Python |
| Where state lives | PostgreSQL tables | Pixeltable tables (versioned, with lineage) |
Where they are complementary, not competing#
This is not a "pg_durable is bad" argument—quite the opposite. pg_durable is a clean, well-scoped tool for teams whose state already lives in Postgres and whose workflows are SQL-shaped: ingest pipelines, scheduled maintenance, fan-out aggregation, durable execution per row or batch. For that, keeping execution in the database is exactly right.
Pixeltable targets the workloads pg_durable explicitly defers: arbitrary model code, multimodal transforms, rich Python control flow, and pipelines that span media and providers. The two can coexist in one architecture—pg_durable orchestrating relational steps, Pixeltable owning the multimodal data plane—because they share the same conviction that compute belongs next to data. The difference is which data, and how much of it SQL can reach.
Why the trend favors a multimodal data plane#
Once you accept "compute close to data," the properties that make pg_durable attractive for SQL become table stakes for AI infrastructure too—and they are exactly what Pixeltable provides for multimodal workloads:
- Incremental computation: insert new media and only the affected cells recompute, instead of rerunning whole pipelines. See automatic dependency tracking.
- Versioning and lineage: every transformation is tracked and reproducible, so you can debug and roll back like you can with time travel.
- Per-row error containment: one failed API call or bad row does not poison the batch—failures land in error columns you can inspect and retry, a pattern we cover in rate limiting and graceful failure handling.
- Durable state as table state: agent and pipeline history live in queryable tables, the same idea behind treating the session log as the system of record and schema-driven infrastructure.
It is the same shift that moved teams off general-purpose DAG schedulers for AI work, as we argued in Pixeltable vs Airflow: stop coordinating external systems and start declaring outcomes where the data already is.
Conclusion#
pg_durable is a meaningful release, and its framing matters as much as its code: a major vendor declaring that durable execution belongs inside the system of record, close to the data. That is the trend. Pixeltable is the multimodal expression of the same principle—declarative, incremental, versioned compute over the video, audio, images, documents, and embeddings that SQL cannot reach. When the industry moves your direction, the right response is to make the case clearly.
To see what "compute close to multimodal data" looks like in practice:
- Start with the Pixeltable docs and
pip install pixeltable - Read Who Owns the Multimodal Data Plane?
- See declarative, incremental multimodal pipelines in action
- Explore agents as data for durable agent state
