Coding agents are good at writing a backend. The trouble starts the day after: a new column on a table that already holds data, a model provider that times out halfway through a batch, a deploy, a request that came back wrong. In the usual stack each of those touches a different system, and the agent can edit code in all of them but can see the result of almost none.
This post takes one multimodal app through that whole life, using pxt commands an agent can run and read, and at the end one short script. The app is a claims desk: a customer sends a photo and a voice note, and the app transcribes the note, checks the photo, routes the claim and writes a reply. Every output below is from recorded runs of the app, on my laptop and then on Pixeltable Cloud. The requests to the agent are staged, and the outage is simulated.
Why the day after the build is harder#
A multimodal backend is usually an API service, a queue and workers, a preprocessing job, an embedding job, a vector database, a metadata database, a status table and a retry script. Each piece is fine on its own. The trouble is in between. When the agent changes a prompt, which rows are stale? When OpenAI fails for twenty minutes, which cells need a retry, and who wrote that bookkeeping?
Pixeltable keeps that state in the tables. A column is either stored data or a computation over other columns, so the system knows what depends on what, what failed and what each version held. That is what lets the commands below answer the agent's questions in plain text.
The app: one file#
I asked the agent for a claims desk: "transcribe the voice note, check the photo, route the claim, write a reply." It wrote one app.py. The class below is an excerpt of that file, not a runnable program: the helper functions, QUESTIONS and MODEL are defined above it in the file.
The annotations are stored columns: the product name, the photo, the voice note. The assignments are computed columns. seen is a CLIP query that names the product in the photo, photo_ok and route are plain Python functions, and the rest are model calls. triage is one call that answers four questions about the photo and the transcript. The same file declares the HTTP routes:
The file adds two query routes the same way: /similar returns the stored claims whose transcripts are closest to a text, and /review lists the claims that did not route to approve, highest severity first.
Build and serve: two commands#
The recorded schema update is the one that added the products table and its CLIP index. An earlier version of the file had already created claims, so it shows as updated:
A POST with a blender photo and a voice note ("The jar on my blender cracked down the side the second time I used it, and now it leaks all over the counter.") came back with the stored row in 9.6 s, routed to approve. The route answers once every computed column of the row is stored, so the agent's first read of the result is also the database's.
Change a table that already holds data#
Then the second request: "Add a column with what is wrong in each claim, and make the reply one sentence." The agent added two lines to the class and, before applying anything, asked for the plan:
"0 destructive" is the line an agent should check before every apply. pxt schema update then added the column and computed it for the five stored claims: four demo claims a seed script filed, and the blender claim from the POST. The transcripts and replies were not run again, because nothing they read had changed.
The prompt change is the subtle one. The reply prompt lives in a function body, so the catalog does not change: pxt schema update reports "catalog is up to date" but reloads app.py for the commands that follow, and then the agent asks for the rerun explicitly:
Three columns, because reply_text and voice_reply read reply. Nothing upstream of it reran.
I had stopped the local service before this change, so the recording has no request against the changed table. A service still running on the old file needs pxt service update app.py claims -f after each of the two edits: its insert route answers 409 once the table has columns it does not know, and its process keeps the old prompt for new rows. pxt recompute rewrites stored rows only.
Recover from a provider outage#
Next, a claim arrives while OpenAI is unreachable. I simulated that by pointing the import script's OpenAI base URL at a closed port. The script inserts with on_error='ignore', so the row is kept with its errors:
One recompute of the first failed column filled every cell that depended on it. One thing we learned recording this: the HTTP insert route aborts on a model error and stores nothing, so a client that must not lose a request retries it, or the app takes it through a Python insert like the one above.
Every change is a version#
The column, the reply rerun, the outage's claim and its fix are four versions (I dropped the timestamp and deletes columns here). pxt revert claims/claims --steps N -f undoes the last N of them. It counts back from the newest version and takes no version number, and it is the undo an agent needs when the change it just made was the wrong one.
Deploy the same app.py to Pixeltable Cloud#
The file the agent changed on my laptop went to Pixeltable Cloud as it was. The project folder added a requirements.txt, and pxt db diff --add-config-entry wrote the database's entry in pixeltable.toml. Then the same commands, pointed at a pxt:// database:
pxt db update builds the database's image from the project and uploads the project. The first build took 950 s; two later runs with the image already built took 143 s and 251 s. The OpenAI key went in as a secret scoped to that one database (pxt secret set). A service reads its secrets when it starts, so the agent sets the secret before pxt service update, as this run did, or runs pxt service restart after.
The POST used an API key granted access to this one service and nothing else (pxt key create hero-ship-post2 --grant access:pxt://pixeltable:claims/services/claims), deleted right after with pxt key delete hero-ship-post2:
The reply is one sentence, because the prompt change went up with the file.
Read what happened: logs and spans#
Last, the agent reads the request from the server's side:
The version the insert made, then the insert, then the request. A service that fails to start puts its traceback in the same log.
A hosted service also keeps sampled traces, and pxt service spans prints them with each span indented under its parent. These are four of that request's spans:
So 7.1 s at the gateway, 6.9 s of it in the insert that runs the computed columns. The trace stops there: in this build the insert is one span, with no span per model call, so it tells the agent which layer was slow and not which column. Spans are read only from hosted services.
Take the data somewhere else#
The last step has no pxt command. It is one library call in a script the agent writes and runs, here copying five text columns of the hosted table into a SQLite file:
db_connect_str is a SQLAlchemy connection string; SQLite is the only target I ran. The script prints nothing, so the agent reads the result from the target database. I exported text columns only, not the photo or the voice note.
What the agent reads at each step#
| Step | Command | What the output tells the agent |
|---|---|---|
| Build | pxt schema update | Which tables were created or updated |
| Serve, and again after a change | pxt service update | Which routes were added, and where they answer |
| Plan a change | pxt schema diff | What will change, and whether anything is destructive |
| Rerun | pxt recompute | Which columns reran, on how many rows |
| Recover | pxt errors, pxt recompute --errors-only | Every failed cell with its error, then what was filled |
| Undo | pxt history, pxt revert | Each version and what it changed, then the last ones undone |
| Deploy | pxt db update | Whether the hosted database's generation is done |
| Observe | pxt service logs, pxt service spans | Requests, inserts and startup errors, then where a request's time went |
| Export | export_sql() in a script | Nothing: the agent queries the target database |
Where this is the wrong tool#
If you don't want a schema, Pixeltable is the wrong tool: everything above follows from the columns being declared. It also won't judge your prompts for you. The agent can rerun a reply in one command, but deciding that the new reply is better is still evaluation, and evaluation is still on you. And a hosted database is a real build: the first image took about sixteen minutes, which you notice when an agent is waiting on it.
For the same app from a different angle, ClaimBot covers the claim row itself, and What Is Pixeltable? walks the local-to-Cloud path in more detail. Configure Pixeltable Cloud covers pixeltable.toml, secrets and deploy order.
Try it with your agent#
Then ask your agent for an app. If you have no account, pxt new gives your agent a free hosted trial database. Claim it within 48 hours or it is deleted; once claimed, the organization is on the Community plan. The commands and outputs above were recorded on Pixeltable 0.7.24.dev7, a development build from before the 0.7.24 release.
