Skip to content
PixeltablePixeltable Logo
All Stories
2026-11-12•13 min read

Pydantic AI · Harness · AI Agents · Python · Multimodal AI

Let your Pydantic AI agent search your media files with Pixeltable

Give a Pydantic AI agent read-only search over images, audio, video, and documents in Pixeltable, plus persistent notes across runs in a second Pixeltable table, using pydantic-ai-harness 0.66.0.

Alison Presmanes Hill

Alison Presmanes Hill

Pixeltable Team

View as Markdown

Join our live webinar on November 19. Laís Carvalho from Pydantic and I will walk through the agent in this post live, on Thursday, November 19, at 11:30am ET. Register for the webinar.

With the Pixeltable capability and memory store in Pydantic AI Harness 0.66.0, you can now build an agentic system that searches your media files (images, audio, video, and documents) and keeps notes across runs.

Two panels. Searching media files today: your agent calls a search tool you write, which reads a vector database; separately, a worker calls models on each new media file and an indexing script writes the embeddings to the database, and a file the worker hasn't reached yet can be missing from the agent's answers. With Pydantic AI Harness and Pixeltable: your agent, with the Pixeltable capability and the Pixeltable memory store, searches and keeps notes in Pixeltable tables that hold your media files, what's made from them (descriptions, transcripts, and a search index, made when you add a file), and the agent's notes.
Figure 1: Searching media files with a separate worker and vector database versus Pixeltable tables and Harness 0.66.0 capabilities.

Why passing files in the prompt stops working#

Pydantic AI agents can take an image, audio clip, video, or PDF in the prompt when you already know which file belongs there. An application holding thousands of photos, recordings, or PDFs has a different problem: the agent has to find the right files by what they show or say, and keep finding new uploads as they arrive.

In the usual stack, a file lands in object storage, a background worker writes descriptions or transcripts, a sync script loads a separate vector database, and any file the worker has not reached yet is missing from the agent's answers.

Open-source Pixeltable combines the table, derived columns, and embedding index in one file. You declare a TableModel with media columns, computed columns, and embedding indexes, and Pixeltable updates them when rows are inserted. Harness 0.66.0 adds two capabilities: Pixeltable gives the agent four read-only catalog tools (list_tables, describe_table, query_table, and similarity_search) that return rows without exposing media file paths, and Memory(PixeltableMemoryStore(...)) stores agent memory in a Pixeltable table in the same catalog.

How the marketplace agent is wired#

In the marketplace example app, sellers upload three photos and a price per listing, and buyers ask a Pydantic AI agent for recommendations. Two Pixeltable tables back the agent: market.listings holds the photos, extracted metadata, and image embedding index, and market.memory stores buyer preferences between separate runs.

Three groups. Your Pydantic AI agent: an agent that can use any model Pydantic AI supports, the Pixeltable capability with its four read-only tools, and the memory capability. Your Pixeltable tables: a table of media files, with what you define computed on insert and a search index, and a notes table. Models you choose: a model that describes, transcribes, or captions each file, and an embedding model, local or hosted.
Figure 2: The agent connects through two capabilities to Pixeltable tables; the chat, extraction, and embedding models are configurable.

The agent lives in agent.py. Below is an excerpt from that file (not a standalone script: the top of agent.py imports MODEL, Listings, and TableModel from app.py, defines the Picks output schema, binds the catalog with TableModel.bind_all('market'), and attaches an output_validator that verifies returned IDs and prices against market.listings):

python

The agent works with any model provider Pydantic AI supports. In the marketplace app, MODEL is 'gemini-3.8-flash', and the model calls that extract titles, categories, conditions, and photo embeddings run when a listing is inserted rather than when the agent searches. Filtering rows with query_table calls no model at all.

Search images by what they show#

When an embedding index sits directly on an image column, the agent searches what is inside the photo rather than relying on manually written captions or tags. This excerpt from app.py shows the indexed image column on Listings (TableModel = pxt.model_base(), pxtf, EMBEDDING_MODEL = 'gemini-embedding-2', and the derived Gemini columns including title and condition sit in the rest of the file):

python

When a buyer asks for reading light under $150, the agent inspects the table schema with describe_table, runs similarity_search on photo_1 with search phrases of its own, filters candidates, and returns structured JSON (listing IDs, titles, and prices) so the frontend loads photos by ID without ever passing image files through tool context:

Two windows. Your Pydantic AI agent: the request "I need better light for reading at night, under $150", then four tool calls (describe_table on market.listings, two similarity_search calls with the agent's own words, such as "reading light lamp nightstand desk lamp", and query_table for category "home"), then the agent's answer as JSON: two picks, each a listing ID, a title, and a price, a Black Adjustable LED Desk Lamp at $45 and a Vintage Brass Banker's Lamp at $65. Your app: the same two listings with their photos, which your app loads by listing ID. A note says the agent never gets the photos.
Figure 3: One buyer request recorded in the marketplace app. The agent makes two similarity_search calls and returns listing IDs and prices; the app fetches the photos by ID.

You choose the embedding function per column. Local models such as sentence-transformers and CLIP from Hugging Face run with no API key. Hosted models from OpenAI, Gemini, Voyage AI, or Twelve Labs use your API key, and each similarity_search call embeds its query string once. A single user request can make several similarity_search calls (Figure 3 shows two), which means multiple embedding API calls on hosted indexes.

Pixeltable's pxtf.gemini.embed_content and pxtf.twelvelabs.embed functions also accept audio and video columns directly. For timestamped search over longer clips, Pydantic's video example takes a text-based route: it transcribes 30-second segments of audio, captions one frame every 10 seconds, and indexes that text so search results include frame timestamps.

New files become searchable once insert finishes#

Pixeltable maintains computed columns and embedding indexes incrementally on insert. When you add a row, Pixeltable computes every derived column and updates the index as part of that insert operation, with no external cron job or indexing queue to trigger.

How long an insert takes depends on the model calls and media transforms defined on the table. Adding a listing in the marketplace app runs Gemini extraction on three photos, bounding-box cropping with Pillow, speech synthesis and an 11-second ffmpeg video, plus the embedding call on photo_1. app.py runs POST /listings as a background job because those calls take around 15 seconds in typical runs (23 seconds for the recorded insert behind Figure 4). When I posted a 19th listing (a floor lamp at $120) against the running app, once Gemini finished writing the details and the embedding index updated, running the same buyer query again returned the new floor lamp.

You add a photo, a recording, or a document, and a new row goes into your table. Inside a box labeled "Pixeltable computes what you defined, on insert", four lanes run: a model describes the file; built-in iterators cut audio, pull frames, or split documents, and a model transcribes or captions the pieces; built-in and your own functions make new media; and your embedding model adds the file to the search index.
Figure 4: What Pixeltable computes on insert. The diagram label says "Searchable on the agent's next request" from a 23-second listing insert; strictly, a new row becomes searchable once all insert-time work completes.

Because computed media columns live in the same table as the search index, the same insert also generated the app's user-facing assets in that row: the cropped thumbnail from the custom crop_to_box UDF and the narrated listing video:

Two windows. What the seller added: three photos of a brass arched floor lamp and a price of $120. What the insert made, in the same row: Gemini's title, Brass Arched Floor Lamp with Fabric Drum Shade, with condition "good" and category "home"; the first photo cropped to the lamp, by the app's own function; and a frame of a narrated video of all three photos with the title and price as a caption, made by built-in functions with Gemini's voice.
Figure 5: Seller inputs for the 19th listing on the left, and the derived title, condition, cropped image, and narrated video stored in the same Pixeltable row on the right.

Filter on structured fields a model wrote#

similarity_search ranks listings by visual or semantic closeness, while query_table applies equality filters. Filtering on model-written descriptions only stays reliable if those values come from a constrained schema rather than free text that mixes "like new", "like-new", and "barely used".

In app.py, Pixeltable asks Gemini for JSON matching a Pydantic model and stores details = response.parsed.astype(pxt.Json[ListingDetails]), promoting fields such as condition into typed columns indexed with a BtreeIndex (an insert with a missing field or invalid enum value fails cleanly rather than storing dirty strings):

python

Because condition is restricted to four literal strings, the agent can call query_table on condition == 'like new' without guessing phrasing variants.

Keep notes across separate runs with PixeltableMemoryStore#

PixeltableMemoryStore implements Harness's MemoryStore interface on top of a Pixeltable table (market.memory here), alongside the built-in in-memory, file, SQLite, and Postgres backends. On a write, the agent calls write_memory; on the first request of any later run, Memory injects an excerpt of the saved notes into context even when there is no shared conversation history. Because writes use compare-and-set versions and operation receipts, a retried tool call replays instead of duplicating a note (see the technical walkthrough for receipt details and version history behavior: deleted memories stay in prior table versions unless you drop the table).

I ran agent.py as two separate processes with no chat history between them. In run 1, telling the agent "I only buy things in like-new condition. Remember that." triggered a write_memory call. In run 2, asking "What do you have for my reading corner?" still matched the $120 brass floor lamp first on visual similarity, but the agent applied the saved preference with query_table and returned only the three like-new lamps:

Two windows. Your Pydantic AI agent: in run 1, the request "I only buy things in like-new condition. Remember that." and a write_memory call that saves "Buyer only buys items in like-new condition." In run 2, a separate process, the request "What do you have for my reading corner?", a similarity_search whose first match is a Brass Arched Floor Lamp in "good" condition, a query_table for condition "like new", and an answer of 3 picks, all like new. Your app: the desk lamp at $45, the mushroom table lamp at $95, and the salt lamp at $25, each like new, with the floor lamp at $120 grayed out as left out.
Figure 6: A preference stored via write_memory in process 1 filters search results in process 2, omitting the good floor lamp even though it ranked first by similarity.

Where this is the wrong tool#

If your agent only inspects a few known files passed directly in a request, use Pydantic AI's native multimodal inputs instead of setting up a Pixeltable table. If you only want persistent agent notes and have no media tables or embedding indexes beside them, use Harness's built-in FileStore, SqliteMemoryStore, or PostgresMemoryStore. If you do not want a typed TableModel schema, or you already operate a vector database and ingest pipeline you plan to keep, Pixeltable is the wrong tool.

Run the example or wire your own agent#

To run the complete marketplace demo yourself (requires Python 3.12 to 3.14, uv, ffmpeg, and GEMINI_API_KEY set in your environment), clone the repository, apply the schema and start the local FastAPI service, seed the sample listings using the printed service URL, and query the agent:

bash

If you are adding the integration directly to your own Pydantic AI project instead of cloning the demo, install the extra alongside a model provider package (plus pixeltable[serve] if you want Pixeltable's FastAPI serving routes):

bash

The same app.py runs hosted on Pixeltable Cloud via pxt db update when you are ready to move off a local machine.

See it live on November 19. Laís Carvalho from Pydantic and I will walk through this agent at 11:30am ET. Register for the webinar.

Resources#

Frequently asked questions

Does the Pixeltable capability let a Pydantic AI agent write to my tables or read raw media file paths?

No. The Pixeltable capability gives the agent four read-only tools: list_tables, describe_table, query_table, and similarity_search. It omits media, array, and binary columns so raw local file paths stay out of tool outputs, and it refuses version handles. Your application loads or serves media files by row ID. Agent notes go into a separate memory table through PixeltableMemoryStore.

When can the agent find a newly added image, video, audio file, or document?

Once the insert finishes. Computed columns, iterators, and embedding indexes run as part of inserting the row into Pixeltable. If an insert calls external models or transcodes video, the new file does not appear in search results until that insert work completes.

Does every agent search call an external embedding API?

Only when your embedding index uses a hosted embedding model such as Gemini or OpenAI. Local models such as sentence-transformers and CLIP run on your machine with no API call. With a hosted model, each similarity_search tool call embeds its query string once, and a single user request can trigger multiple similarity_search calls. Filtering stored columns with query_table makes no model call.

When should I use Pydantic AI native media inputs or Harness file memory instead of Pixeltable?

If your agent inspects a small, known set of files per request, pass them directly with Pydantic AI native media inputs. If you only need agent memory with no media tables or embedding indexes beside it, use the in-memory, file, SQLite, or Postgres store in Pydantic AI Harness. If you do not want a typed table schema, Pixeltable is the wrong tool.

Declarative. Multimodal. Incremental.

One Python file for tables, indexes, and HTTP. Start locally, or sign up for Cloud.

Get started

Docs quickstart · 10-minute tutorial · GitHub · Discord

Read Next

Which tools do AI coding agents pick? A benchmark of 96 runs

AI Agents

Which tools do AI coding agents pick? A benchmark of 96 runs

Four AI engines, three of them search-grounded, 8 prompts, 96 runs. FastAPI in all 96. Qdrant 46 vs pgvector 33. Orchestration split by prompt and by engine. Citations are not the same as picks.

Read article
AI Agent Architecture: A Practical Guide to Building Agents with State Management

AI Agents

AI Agent Architecture: A Practical Guide to Building Agents with State Management

Master AI agent architecture and state management in complex AI agent systems. Learn how to build effective AI agents with Pixeltable's declarative infrastructure, covering memory, tools, orchestration patterns, and multimodal capabilities.

Read article
Can a Coding Agent Maintain a Multimodal AI Backend? Recorded Runs, From Build to Export

AI Agents

Can a Coding Agent Maintain a Multimodal AI Backend? Recorded Runs, From Build to Export

A coding agent can write a backend. Keeping it running is harder: changing a column, refilling failed model calls, deploying it, reading its logs and traces, exporting its data. Here is one claims desk app taken through all of it with pxt commands, every output from recorded runs.

Read article