---
title: "The Unified Multimodal Backend Agents Build With"
date: "2026-09-16"
dateModified: "2026-09-16"
author: "Pierre Brunelle"
tags:
  - Multimodal AI
  - TableModel
  - AI Agents
  - Unified Backend
  - Declarative Schema
  - Pixeltable Cloud
  - Computed Columns
description: "Pixeltable is the database, orchestration, and serving layers in one Python file. Images, video, audio, and documents live in tables. A transform is a computed column. An index is a declaration, and so is an HTTP route."
url: "https://www.pixeltable.com/blog/unified-multimodal-backend-agents-build-with"
---

# The Unified Multimodal Backend Agents Build With

**Summary:** Pixeltable is the unified multimodal backend agents build with. Database, orchestration, and serving live in one Python file. Images, video, audio, and documents are column types. A transform is a computed column. An index is a declaration, and so is an HTTP route. Insert a row and everything below it runs. Object storage, a vector database, an orchestrator, and the endpoint code that copies between them collapse into `app.py`. `pip install pixeltable`.

 
## Four Products, One File

 
The tax was never writing a column. It was the joints before you could write one.

 
Bytes in an object store. IDs in a warehouse. A DAG that re-runs a world because it cannot see which cells changed. Embeddings in a vector database that nobody invalidates when the source clip dies. A FastAPI file you babysit so the demo has a URL. Each piece is fine. The set is a second job. That is the [Frankenstein stack](/blog/deconstructing-ai-frankenstein-stack). We spent 2026 deleting it: [Never Fuck With Your Backend Anymore](/blog/never-fuck-with-your-backend-anymore).

 
What remains is a habit. You declare tables, computed columns, indexes, and routes in one file. You apply it. You insert a row. The work runs. You serve the same file. That is the backend a coding agent can keep — not a shopping list of services it has to invent every session.

 
## The File Agents Write

 
`pxt service example --out app.py` writes this. Delete what you do not need. Annotations are stored columns. Assignments are computed columns: they run on insert and on update. A `@pxt.udf` in the same file is a function those columns can call. Production tables often use `uuid7()` as a generated primary key; the example uses an integer you insert so the first `curl` is obvious.

 
```bash
pip install 'pixeltable[serve]'
pxt init
pxt service example --out app.py
pxt schema update app.py my_app
pxt service update app.py my_app
```

 
```python
import pixeltable as pxt
import pixeltable.functions as pxtf
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()

@pxt.udf
def excerpt(text: str, n: int = 12) -> str:
 return text if len(text) <= n else f'{text[:n]}...'

class Docs(TableModel, name='docs'):
 doc_id: pxt.Int
 title: pxt.String
 body: pxt.String | None
 title_upper = pxtf.string.upper(title)
 summary = excerpt(title)

ingest = FastAPIRouter(name='ingest')
ingest.add_insert_route(
 Docs, path='/docs', inputs=[Docs.doc_id, Docs.title, Docs.body], outputs=[Docs.title_upper, Docs.summary]
)
ingest.add_compute_route(Docs, path='/titles', inputs=[Docs.title], outputs=[Docs.title_upper])
```

 
`title: pxt.String` is a value you provide. `title_upper = ...` is computed. `POST /docs` inserts and returns the computed columns. `POST /titles` computes without storing a row. Insert a row. The assignments run. That is the whole orchestration story.

 
## Three Verbs That Are Not Interchangeable

 
Agents mix these. The file does not.

 
`pxt schema update` creates the catalog `my_app` and its tables. It does not start HTTP. `pxt service update` starts HTTP. It does not create tables. On Cloud, `pxt db update` creates or updates the hosted database — image, workers, disk. It does not insert rows. `pxt service run` is local only. It cannot target `pxt://`.

 
The port is assigned. Read it back rather than hardcoding it:

 
```bash
URL=$(pxt service list --json | jq -r '.[0].endpoint')
curl -X POST "$URL/docs" \
 -H 'Content-Type: application/json' \
 -d '{"doc_id": 1, "title": "Hello", "body": "world"}'
# {"title_upper":"HELLO","summary":"Hello"}
```

 
## Media Is a Column

 
The same file holds `pxt.Image`, `pxt.Video`, `pxt.Audio`, or `pxt.Document`. A computed column over one of them is another assignment: frames, transcripts, captions, chunks. That is a [multimodal data table](/blog/what-is-a-multimodal-data-table), not a `FILE` path the engine does not understand. Flattening a Gong call into a SQL tag is not beyond SQL. It is also not a system of record for the audio. The post-AI stack still draws a warehouse. Media needs a typed table: [The Post-AI Data Stack Needs a Multimodal Table](/blog/post-ai-data-stack-needs-multimodal-table).

 
An embedding index is a declaration on the class (`__indexes__`). A query is `@pxt.query`. HTTP for that query is a route on the same router. You do not stand up a second product to keep the index honest. Insert is the pipeline.

 
## Same File on Cloud

 
Create an API key in the Cloud dashboard, set `PIXELTABLE_API_KEY`, name the database in `pixeltable.toml`, then target it by URI. A `@pxt.udf` in that same `app.py` is in the image `pxt db update` builds. Promotion is the file you already ran, not a rewrite into a warehouse dialect. That is why the [local-cloud loop](/blog/why-local-cloud-loop-matters) is the lifecycle, not a staging cluster.

 
```bash
pxt db update pxt://org:mydb
pxt schema update app.py pxt://org:mydb
pxt service update app.py pxt://org:mydb
```

 
Order matters. Schema first against a hosted database that does not exist yet is a 404 on the UDF. Database, then schema, then service.

 
## What Agents Copy vs Invent

 
Hand the agent [get-started.md](/get-started.md). That playbook installs the package, the [Pixeltable Skill](https://docs.pixeltable.com/overview/building-pixeltable-with-llms), and MCP. The capability file is [skill.md](https://docs.pixeltable.com/skill.md). How you design for that customer — install via `npx`, evaluate via a machine-readable overview, buy by writing `app.py` — is [Designing Software for LLMs as Customers](/blog/designing-software-for-llms-as-customers).

 
```bash
npx skills add pixeltable/pixeltable-skill
```

 
The skill writes a `TableModel` in `app.py`. If the agent writes `create_table` in application code, names the file `schema.py`, or writes the removed command `pxt serve`, the installed skill is stale. Reinstall it. Notebooks and tests still use `pxt.create_table()`. An app puts tables in `app.py` and creates them with `pxt schema update`. Do not mix both styles for the same tables.

 
`pxt service example` is the generic file. [`uvx pixeltable-new`](https://github.com/pixeltable/pixeltable-new) copies one app from the starter kit — the acquisition artifact we learned to ship so agents do not default to LangChain plus a vector DB: [What We Learned Shipping Application Templates for AI Agents](/blog/shipping-application-templates-for-ai-agents). The default copy is a chat app; pass `agent` as the last argument to `pxt schema update`. `--video` copies video search; pass `videointel`.

 
```bash
uvx pixeltable-new myapp
cd myapp
uv sync
pxt schema update app.py agent
pxt service update app.py agent
```

 
Inserting into the knowledge table needs no API key. The `/ask` route needs `ANTHROPIC_API_KEY`.

 
To mount the routes on an existing FastAPI app, `app.include_router(...)`. To skip endpoints, run `pxt schema update`, insert from Python, then `export_sql` if you need a warehouse bridge. Self-hosting is the same schema without the hosted target.

 
## What This Is Not

 
It is not no-code. If you do not want a schema, Pixeltable is the wrong product. The point is that the schema *is* the backend. The persona version of shrinking that surface — Lovable for people who do not write backends, Pixeltable for people who do — is [Making Multimodal More Lovable](/blog/making-multimodal-more-lovable). It is also not a signup-wall remover that provisions Postgres, Redis, auth, and hosting — that job is [Cohesivity](/compare/pixeltable-vs-cohesivity); the query page is [Backend for AI agents](/backend-for-ai-agents).

 
It is not “don’t use FastAPI.” `FastAPIRouter` is FastAPI derived from the catalog. You do not write the handlers. You still get HTTP.

 
It is not a warehouse replacement. Keep Snowflake or Databricks for Card TPV and NRR. Do not start the video graph in a generic `FILE` column.

 
And it is not “use our agent.” Use yours. Pixeltable is the catalog your agent writes to. MCP and HTTP are adapters on that catalog, not a second semantic language.

 
## People Also Ask

 
**Is TableModel required?** For new apps, yes — it is the public schema. The notebook form (`create_table`, `add_computed_column`) still exists for one-off scripts. Do not mix both styles for the same tables.

 
**Do I still need Pinecone, Airflow, and an object store?** Not as a default architecture. Media, computed transforms, embedding indexes, and HTTP live in one system. You can still point media at your bucket. You should not need a sync job to keep three products from drifting.

 
**How do I go to Cloud?** Same `app.py`. Set `PIXELTABLE_API_KEY`, declare `pxt://org:db`, then `pxt db update` → `pxt schema update` → `pxt service update`. Hosted HTTP is on the website dashboard. `pxt service run` is local only.

 
**Can I mount this on an existing FastAPI app?** Yes. `app.include_router(...)` the routers this file declares. To skip HTTP entirely, apply the schema and insert from Python.

 
**The agent wrote `create_table` / `pxt serve` / `schema.py`.** The skill is stale. Reinstall `npx skills add pixeltable/pixeltable-skill` and hand it [get-started.md](/get-started.md).

 
## Keep the File, Delete the Stack

 
The backend is one schema you keep, not a stack you assemble. Agents can keep a file. They cannot keep a Frankenstein.

 
Declare the class. Apply it. Insert a row. Serve the same file.

 

 - **[Pixeltable on GitHub](https://github.com/pixeltable/pixeltable)**: `pip install pixeltable`

 - **[Get started](/get-started.md)**: CLI, Skill, and the one-file loop

 - **[skill.md](https://docs.pixeltable.com/skill.md)**: the capability file the skill follows

 - **[Never Fuck With Your Backend Anymore](/blog/never-fuck-with-your-backend-anymore)**: the 2026 recap — what disappeared

 - **[Making Multimodal More Lovable](/blog/making-multimodal-more-lovable)**: the sibling essay on persona and surface

 - **[Why the Local-Cloud Loop Matters](/blog/why-local-cloud-loop-matters)**: the sibling essay on promotion

 - **[The Post-AI Data Stack Needs a Multimodal Table](/blog/post-ai-data-stack-needs-multimodal-table)**: warehouse stays SQL; media needs a typed table

 - **[Pixeltable documentation](https://docs.pixeltable.com)**