--- title: "Can a Coding Agent Maintain a Multimodal AI Backend? Recorded Runs, From Build to Export" date: "2026-11-12" dateModified: "2026-11-12" author: "Pierre Brunelle" tags: - AI Agents - Coding Agents - Multimodal AI - Pixeltable Cloud - CLI - Computed Columns description: "A coding agent can write a backend. Keeping it running is harder: changing a column, refilling failed model calls, deploying it, reading its logs and traces, exporting its data. Here is one claims desk app taken through all of it with pxt commands, every output from recorded runs." url: "https://www.pixeltable.com/blog/coding-agent-maintains-multimodal-backend" --- # Can a Coding Agent Maintain a Multimodal AI Backend? Recorded Runs, From Build to Export Coding agents are good at writing a backend. The trouble starts the day after: a new column on a table that already holds data, a model provider that times out halfway through a batch, a deploy, a request that came back wrong. In the usual stack each of those touches a different system, and the agent can edit code in all of them but can see the result of almost none. This post takes one multimodal app through that whole life, using `pxt` commands an agent can run and read, and at the end one short script. The app is a claims desk: a customer sends a photo and a voice note, and the app transcribes the note, checks the photo, routes the claim and writes a reply. Every output below is from recorded runs of the app, on my laptop and then on Pixeltable Cloud. The requests to the agent are staged, and the outage is simulated. ## Why the day after the build is harder A multimodal backend is usually an API service, a queue and workers, a preprocessing job, an embedding job, a vector database, a metadata database, a status table and a retry script. Each piece is fine on its own. The trouble is in between. When the agent changes a prompt, which rows are stale? When OpenAI fails for twenty minutes, which cells need a retry, and who wrote that bookkeeping? Pixeltable keeps that state in the tables. A column is either stored data or a computation over other columns, so the system knows what depends on what, what failed and what each version held. That is what lets the commands below answer the agent's questions in plain text. ## The app: one file I asked the agent for a claims desk: "transcribe the voice note, check the photo, route the claim, write a reply." It wrote one `app.py`. The class below is an excerpt of that file, not a runnable program: the helper functions, `QUESTIONS` and `MODEL` are defined above it in the file. ```python class Claims(TableModel, name="claims"): id = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True) product: pxt.String photo: pxt.Image voice_note: pxt.Audio seen = closest_product(photo) photo_ok = shows_product(seen, product) transcript = openai.transcriptions(voice_note, model="gpt-4o-transcribe").text.astype(pxt.String | None) flagged = openai.moderations(transcript).results[0].flagged triage = openai.decisions( [ { "role": "user", "content": [ {"type": "input_text", "text": "Product: " + product + ". Customer: " + transcript}, {"type": "input_image", "image_url": photo}, ], } ], questions=QUESTIONS, model=MODEL, ) damaged = triage.answers[0].probability consistent = triage.answers[1].probability cause = triage.answers[2].choice severity = triage.answers[3].score route = route_claim(damaged, consistent, flagged) reply = openai.responses(reply_prompt(product, transcript, route), model="gpt-4o-mini") reply_text = reply.output[0].content[0].text.astype(pxt.String | None) voice_reply = openai.speech(reply_text, model="gpt-4o-mini-tts", voice="alloy") ``` The annotations are stored columns: the product name, the photo, the voice note. The assignments are computed columns. `seen` is a CLIP query that names the product in the photo, `photo_ok` and `route` are plain Python functions, and the rest are model calls. `triage` is one call that answers four questions about the photo and the transcript. The same file declares the HTTP routes: ```python api = FastAPIRouter(name="claims", prefix="/api") api.add_insert_route( Claims, path="/claims", uploadfile_inputs=[Claims.photo, Claims.voice_note], inputs=[Claims.product], outputs=[Claims.id, Claims.transcript, Claims.cause, Claims.severity, Claims.route, Claims.reply_text], ) ``` The file adds two query routes the same way: `/similar` returns the stored claims whose transcripts are closest to a text, and `/review` lists the claims that did not route to `approve`, highest severity first. ## Build and serve: two commands The recorded schema update is the one that added the `products` table and its CLIP index. An earlier version of the file had already created `claims`, so it shows as updated: ```bash $ pxt schema update app.py claims -f unchanged claims/demo_inputs created claims/products updated claims/claims $ pxt service update app.py claims -f + claims applied http://127.0.0.1:54963 route 'POST /api/claims' will be added [additive] route 'POST /api/similar' will be added [additive] route 'GET /api/review' will be added [additive] Plan: 1 start, 0 restart, 0 unchanged, 0 extra ``` A POST with a blender photo and a voice note ("The jar on my blender cracked down the side the second time I used it, and now it leaks all over the counter.") came back with the stored row in 9.6 s, routed to `approve`. The route answers once every computed column of the row is stored, so the agent's first read of the result is also the database's. ## Change a table that already holds data Then the second request: "Add a column with what is wrong in each claim, and make the reply one sentence." The agent added two lines to the class and, before applying anything, asked for the plan: ```bash $ pxt schema diff app.py claims = claims/demo_inputs no change = claims/products no change ~ claims/claims update + column 'issue' will be added safe + column 'issue_call' will be added safe Plan: 0 create, 1 update, 2 unchanged, 0 extra | 0 destructive ``` "0 destructive" is the line an agent should check before every apply. `pxt schema update` then added the column and computed it for the five stored claims: four demo claims a seed script filed, and the blender claim from the POST. The transcripts and replies were not run again, because nothing they read had changed. The prompt change is the subtle one. The reply prompt lives in a function body, so the catalog does not change: `pxt schema update` reports "catalog is up to date" but reloads `app.py` for the commands that follow, and then the agent asks for the rerun explicitly: ```bash $ pxt recompute claims/claims reply -f recomputed claims.reply, claims.reply_text, claims.voice_reply on claims/claims: 5 rows updated ``` Three columns, because `reply_text` and `voice_reply` read `reply`. Nothing upstream of it reran. I had stopped the local service before this change, so the recording has no request against the changed table. A service still running on the old file needs `pxt service update app.py claims -f` after each of the two edits: its insert route answers 409 once the table has columns it does not know, and its process keeps the old prompt for new rows. `pxt recompute` rewrites stored rows only. ## Recover from a provider outage Next, a claim arrives while OpenAI is unreachable. I simulated that by pointing the import script's OpenAI base URL at a closed port. The script inserts with `on_error='ignore'`, so the row is kept with its errors: ```bash $ uv run python import_claims.py claims Inserted 1 row with 14 errors across 14 columns (claims.transcript, claims.flagged, ..., claims.issue) in 12.45 s (0.08 rows/s) $ pxt errors claims/claims {id: '01a1c769-...'} transcript APIConnectionError Connection error. {id: '01a1c769-...'} flagged APIConnectionError Connection error. ... $ pxt recompute claims/claims transcript --errors-only -f recomputed claims.triage, claims.reply_text, claims.issue, ... on claims/claims: 1 row updated ``` One recompute of the first failed column filled every cell that depended on it. One thing we learned recording this: the HTTP insert route aborts on a model error and stores nothing, so a client that must not lose a request retries it, or the app takes it through a Python insert like the one above. ## Every change is a version ```bash $ pxt history claims/claims version change_type inserts updates errors schema_change 17 data 0 1 0 16 data 1 0 14 15 data 0 5 0 14 schema 0 5 0 Added: issue_call, issue ``` The column, the reply rerun, the outage's claim and its fix are four versions (I dropped the timestamp and deletes columns here). `pxt revert claims/claims --steps N -f` undoes the last N of them. It counts back from the newest version and takes no version number, and it is the undo an agent needs when the change it just made was the wrong one. ## Deploy the same app.py to Pixeltable Cloud The file the agent changed on my laptop went to [Pixeltable Cloud](https://docs.pixeltable.com/cloud) as it was. The project folder added a `requirements.txt`, and `pxt db diff --add-config-entry` wrote the database's entry in `pixeltable.toml`. Then the same commands, pointed at a `pxt://` database: ```bash $ pxt db update pxt://pixeltable:claims -f claims generation 1 done $ pxt schema update app.py pxt://pixeltable:claims -f created pxt://pixeltable:claims/demo_inputs created pxt://pixeltable:claims/products created pxt://pixeltable:claims/claims $ pxt service update app.py pxt://pixeltable:claims -f + claims applied https://pixeltable-claims.svc.pxt.run/claims ``` `pxt db update` builds the database's image from the project and uploads the project. The first build took 950 s; two later runs with the image already built took 143 s and 251 s. The OpenAI key went in as a secret scoped to that one database (`pxt secret set`). A service reads its secrets when it starts, so the agent sets the secret before `pxt service update`, as this run did, or runs `pxt service restart` after. The POST used an API key granted access to this one service and nothing else (`pxt key create hero-ship-post2 --grant access:pxt://pixeltable:claims/services/claims`), deleted right after with `pxt key delete hero-ship-post2`: ```bash $ curl -H 'X-api-key: ...' -F product=blender -F photo=@media/request-photo.jpeg \ -F voice_note=@media/request-voice_note.mp3 https://pixeltable-claims.svc.pxt.run/claims/api/claims HTTP 200 in 7.374653s {"id":"01a1d163-9b46-7702-961e-a59cf89a8ed8","transcript":"The jar on my blender cracked down the side the second time I used it, and now it leaks all over the counter.","cause":"manufacturing defect","severity":1.98,"route":"approve","reply_text":"Your blender replacement is approved and will ship within 3 days."} ``` The reply is one sentence, because the prompt change went up with the file. ## Read what happened: logs and spans Last, the agent reads the request from the server's side: ```bash $ pxt service logs pxt://pixeltable:claims/claims --since 5m ... INFO pixeltable.catalog.table_version: TableVersion claims: new version 1 ... INFO pixeltable: Inserted 1 row with 0 errors in 6.91 s (0.14 rows/s) ... INFO uvicorn.access: ... "POST /claims/api/claims HTTP/1.1" 200 ``` The version the insert made, then the insert, then the request. A service that fails to start puts its traceback in the same log. A hosted service also keeps sampled traces, and `pxt service spans` prints them with each span indented under its parent. These are four of that request's spans: ```bash $ pxt service spans pxt://pixeltable:claims/claims --since 10m trace ad5f0b6e71b6399c48caea37352f103f ...T20:12:08.154 7118.5 ms pixeltable.gateway ...T20:12:08.190 7081.8 ms POST /api/claims ...T20:12:08.362 6909.2 ms pixeltable.insert ...T20:12:08.389 6867.8 ms pixeltable.store.insert_rows ``` So 7.1 s at the gateway, 6.9 s of it in the insert that runs the computed columns. The trace stops there: in this build the insert is one span, with no span per model call, so it tells the agent which layer was slow and not which column. Spans are read only from hosted services. ## Take the data somewhere else The last step has no `pxt` command. It is one library call in a script the agent writes and runs, here copying five text columns of the hosted table into a SQLite file: ```python import pixeltable as pxt from pixeltable.io import export_sql claims = pxt.get_table('pxt://pixeltable:claims/claims') export_sql( claims.select(claims.product, claims.transcript, claims.route, claims.reply_text, claims.issue), 'claims', db_connect_str='sqlite:///claims.db', ) ``` ```bash $ sqlite3 claims.db "select product, route, issue from claims" blender|approve|Defective jar material. ``` `db_connect_str` is a SQLAlchemy connection string; SQLite is the only target I ran. The script prints nothing, so the agent reads the result from the target database. I exported text columns only, not the photo or the voice note. ## What the agent reads at each step | Step | Command | What the output tells the agent | | --- | --- | --- | | Build | pxt schema update | Which tables were created or updated | | Serve, and again after a change | pxt service update | Which routes were added, and where they answer | | Plan a change | pxt schema diff | What will change, and whether anything is destructive | | Rerun | pxt recompute | Which columns reran, on how many rows | | Recover | pxt errors, pxt recompute --errors-only | Every failed cell with its error, then what was filled | | Undo | pxt history, pxt revert | Each version and what it changed, then the last ones undone | | Deploy | pxt db update | Whether the hosted database's generation is done | | Observe | pxt service logs, pxt service spans | Requests, inserts and startup errors, then where a request's time went | | Export | export_sql() in a script | Nothing: the agent queries the target database | ## Where this is the wrong tool If you don't want a schema, Pixeltable is the wrong tool: everything above follows from the columns being declared. It also won't judge your prompts for you. The agent can rerun a reply in one command, but deciding that the new reply is better is still evaluation, and evaluation is still on you. And a hosted database is a real build: the first image took about sixteen minutes, which you notice when an agent is waiting on it. For the same app from a different angle, [ClaimBot](/blog/claimbot-multimodal-fnol-triage) covers the claim row itself, and [What Is Pixeltable?](/blog/what-is-pixeltable-one-app-py-local-to-cloud) walks the local-to-Cloud path in more detail. [Configure Pixeltable Cloud](/blog/configure-pixeltable-cloud) covers `pixeltable.toml`, secrets and deploy order. ## Try it with your agent ```bash npx skills add pixeltable/pixeltable-skill ``` Then ask your agent for an app. If you have no account, `pxt new` gives your agent a free hosted trial database. [Claim it within 48 hours](/new) or it is deleted; once claimed, the organization is on the Community plan. The commands and outputs above were recorded on Pixeltable 0.7.24.dev7, a development build from before the 0.7.24 release. ## Frequently asked questions ### Can a coding agent maintain a multimodal AI backend? Yes, when every operation is a command the agent can run and whose output it can read. In Pixeltable the backend is one app.py, and building, changing, recovering from failed model calls, deploying and reading logs and traces are pxt commands: pxt schema diff, pxt schema update, pxt recompute, pxt errors, pxt history, pxt db update, pxt service update, pxt service logs and pxt service spans. ### How does an agent add a column to an AI pipeline that already has data? It adds the computed column to app.py, runs pxt schema diff to see the plan (the recorded run printed "0 destructive"), then pxt schema update, which computes the new column for every stored row without rerunning the columns that did not change. A service already running answers 409 on its insert route until pxt service update restarts it. ### What happens to rows when a model provider is down? An insert from Python with on_error="ignore" keeps the row and stores the error in each failed cell. pxt errors lists them, and pxt recompute with --errors-only on the first failed column fills them and everything that depends on it. An HTTP insert route aborts on a model error and stores nothing. ### Does the app.py change to run on Pixeltable Cloud? No. The same file deploys with pxt db update, then pxt schema update and pxt service update against a pxt:// database URI. The recorded run added only a requirements.txt and a database entry in pixeltable.toml.