---
title: "Let your Pydantic AI agent search your media files with Pixeltable"
date: "2026-11-12"
dateModified: "2026-11-12"
author: "Alison Presmanes Hill"
tags:
  - Pydantic AI
  - Harness
  - AI Agents
  - Python
  - Multimodal AI
description: "Give a Pydantic AI agent read-only search over images, audio, video, and documents in Pixeltable, plus persistent notes across runs in a second Pixeltable table, using pydantic-ai-harness 0.66.0."
url: "https://www.pixeltable.com/blog/let-your-pydantic-ai-agent-search-your-media-files"
---

# Let your Pydantic AI agent search your media files with Pixeltable

**Join our live webinar on November 19.** [Laís Carvalho](https://www.linkedin.com/in/laisbsc/) from Pydantic and I will walk through the agent in this post live, on Thursday, November 19, at 11:30am ET. [Register for the webinar](https://pydantic.dev/events/pixeltable-type-safe-media).

 
With the [Pixeltable](https://github.com/pixeltable/pixeltable) capability and memory store in [Pydantic AI Harness](https://pydantic.dev/docs/ai/harness/) 0.66.0, you can now build an agentic system that searches your media files (images, audio, video, and documents) and keeps notes across runs.

 
 [Image: Two panels. Searching media files today: your agent calls a search tool you write, which reads a vector database; separately, a worker calls models on each new media file and an indexing script writes the embeddings to the database, and a file the worker hasn't reached yet can be missing from the agent's answers. With Pydantic AI Harness and Pixeltable: your agent, with the Pixeltable capability and the Pixeltable memory store, searches and keeps notes in Pixeltable tables that hold your media files, what's made from them (descriptions, transcripts, and a search index, made when you add a file), and the agent's notes.]
 *Figure 1: Searching media files with a separate worker and vector database versus Pixeltable tables and Harness 0.66.0 capabilities.*
 

 
## Why passing files in the prompt stops working

 
Pydantic AI agents can take an image, audio clip, video, or PDF in the prompt when you already know which file belongs there. An application holding thousands of photos, recordings, or PDFs has a different problem: the agent has to find the right files by what they show or say, and keep finding new uploads as they arrive.

 
In the usual stack, a file lands in object storage, a background worker writes descriptions or transcripts, a sync script loads a separate vector database, and any file the worker has not reached yet is missing from the agent's answers.

 
Open-source Pixeltable combines the table, derived columns, and embedding index in one file. You declare a [`TableModel`](/learn/what-is-a-tablemodel) with media columns, [computed columns](/learn/what-is-a-computed-column), and [embedding indexes](/learn/what-is-an-embedding-index), and Pixeltable updates them when rows are inserted. Harness 0.66.0 adds two capabilities: `Pixeltable` gives the agent four read-only catalog tools (`list_tables`, `describe_table`, `query_table`, and `similarity_search`) that return rows without exposing media file paths, and `Memory(PixeltableMemoryStore(...))` stores [agent memory](/learn/what-is-agent-memory) in a Pixeltable table in the same catalog.

 
## How the marketplace agent is wired

 
In the [marketplace example app](https://github.com/apreshill/pixeltable-marketplace), sellers upload three photos and a price per listing, and buyers ask a Pydantic AI agent for recommendations. Two Pixeltable tables back the agent: `market.listings` holds the photos, extracted metadata, and image embedding index, and `market.memory` stores buyer preferences between separate runs.

 
 [Image: Three groups. Your Pydantic AI agent: an agent that can use any model Pydantic AI supports, the Pixeltable capability with its four read-only tools, and the memory capability. Your Pixeltable tables: a table of media files, with what you define computed on insert and a search index, and a notes table. Models you choose: a model that describes, transcribes, or captions each file, and an embedding model, local or hosted.]
 *Figure 2: The agent connects through two capabilities to Pixeltable tables; the chat, extraction, and embedding models are configurable.*
 

 
The agent lives in `agent.py`. Below is an excerpt from that file (not a standalone script: the top of `agent.py` imports `MODEL`, `Listings`, and `TableModel` from `app.py`, defines the `Picks` output schema, binds the catalog with `TableModel.bind_all('market')`, and attaches an `output_validator` that verifies returned IDs and prices against `market.listings`):

 
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Memory, Pixeltable
from pydantic_ai_harness.memory import PixeltableMemoryStore

agent = Agent(
 f'google:{MODEL}',
 output_type=Picks,
 instructions=(
 'You help buyers on a secondhand marketplace. To search the listings, call similarity_search on photo_1 '
 "with words for what the buyer wants. Recommend only listings that fit the buyer's request, "
 'with their real id, title, and price. '
 'An empty list is a fine answer: say so in the note when nothing fits.'
 ),
 capabilities=[
 Pixeltable(['market.listings']),
 Memory(PixeltableMemoryStore(table_name='market.memory')),
 ],
)
```

 
The agent works with any model provider Pydantic AI supports. In the marketplace app, `MODEL` is `'gemini-3.8-flash'`, and the model calls that extract titles, categories, conditions, and photo embeddings run when a listing is inserted rather than when the agent searches. Filtering rows with `query_table` calls no model at all.

 
## Search images by what they show

 
When an [embedding index](/learn/what-is-an-embedding-index) sits directly on an image column, the agent searches what is inside the photo rather than relying on manually written captions or tags. This excerpt from `app.py` shows the indexed image column on `Listings` (`TableModel = pxt.model_base()`, `pxtf`, `EMBEDDING_MODEL = 'gemini-embedding-2'`, and the derived Gemini columns including `title` and `condition` sit in the rest of the file):

 
```python
class Listings(TableModel, name='listings'):
 id = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)
 seller: pxt.String
 price: pxt.Float
 photo_1: pxt.Image
 photo_2: pxt.Image
 photo_3: pxt.Image
 ...

 __indexes__ = [
 pxt.EmbeddingIndex(photo_1, embedding=pxtf.gemini.embed_content.using(model=EMBEDDING_MODEL)),
 pxt.BtreeIndex(condition),
 ]
```

 
When a buyer asks for reading light under $150, the agent inspects the table schema with `describe_table`, runs `similarity_search` on `photo_1` with search phrases of its own, filters candidates, and returns structured JSON (listing IDs, titles, and prices) so the frontend loads photos by ID without ever passing image files through tool context:

 
 [Image: Two windows. Your Pydantic AI agent: the request "I need better light for reading at night, under $150", then four tool calls (describe_table on market.listings, two similarity_search calls with the agent's own words, such as "reading light lamp nightstand desk lamp", and query_table for category "home"), then the agent's answer as JSON: two picks, each a listing ID, a title, and a price, a Black Adjustable LED Desk Lamp at $45 and a Vintage Brass Banker's Lamp at $65. Your app: the same two listings with their photos, which your app loads by listing ID. A note says the agent never gets the photos.]
 *Figure 3: One buyer request recorded in the marketplace app. The agent makes two `similarity_search` calls and returns listing IDs and prices; the app fetches the photos by ID.*
 

 
You choose the embedding function per column. Local models such as `sentence-transformers` and CLIP from Hugging Face run with no API key. Hosted models from OpenAI, Gemini, Voyage AI, or Twelve Labs use your API key, and each `similarity_search` call embeds its query string once. A single user request can make several `similarity_search` calls (Figure 3 shows two), which means multiple embedding API calls on hosted indexes.

 
Pixeltable's `pxtf.gemini.embed_content` and `pxtf.twelvelabs.embed` functions also accept audio and video columns directly. For timestamped search over longer clips, Pydantic's [video example](https://github.com/pydantic/pydantic-ai/blob/main/src/pydantic_ai_harness/examples/pixeltable_video.py) takes a text-based route: it transcribes 30-second segments of audio, captions one frame every 10 seconds, and indexes that text so search results include frame timestamps.

 
## New files become searchable once insert finishes

 
Pixeltable maintains computed columns and embedding indexes incrementally on insert. When you add a row, Pixeltable computes every derived column and updates the index as part of that insert operation, with no external cron job or indexing queue to trigger.

 
How long an insert takes depends on the model calls and media transforms defined on the table. Adding a listing in the marketplace app runs Gemini extraction on three photos, bounding-box cropping with Pillow, speech synthesis and an 11-second `ffmpeg` video, plus the embedding call on `photo_1`. `app.py` runs `POST /listings` as a background job because those calls take around 15 seconds in typical runs (23 seconds for the recorded insert behind Figure 4). When I posted a 19th listing (a floor lamp at $120) against the running app, once Gemini finished writing the details and the embedding index updated, running the same buyer query again returned the new floor lamp.

 
 [Image: You add a photo, a recording, or a document, and a new row goes into your table. Inside a box labeled "Pixeltable computes what you defined, on insert", four lanes run: a model describes the file; built-in iterators cut audio, pull frames, or split documents, and a model transcribes or captions the pieces; built-in and your own functions make new media; and your embedding model adds the file to the search index.]
 *Figure 4: What Pixeltable computes on insert. The diagram label says "Searchable on the agent's next request" from a 23-second listing insert; strictly, a new row becomes searchable once all insert-time work completes.*
 

 
Because computed media columns live in the same table as the search index, the same insert also generated the app's user-facing assets in that row: the cropped thumbnail from the custom `crop_to_box` UDF and the narrated listing video:

 
 [Image: Two windows. What the seller added: three photos of a brass arched floor lamp and a price of $120. What the insert made, in the same row: Gemini's title, Brass Arched Floor Lamp with Fabric Drum Shade, with condition "good" and category "home"; the first photo cropped to the lamp, by the app's own function; and a frame of a narrated video of all three photos with the title and price as a caption, made by built-in functions with Gemini's voice.]
 *Figure 5: Seller inputs for the 19th listing on the left, and the derived title, condition, cropped image, and narrated video stored in the same Pixeltable row on the right.*
 

 
## Filter on structured fields a model wrote

 
`similarity_search` ranks listings by visual or semantic closeness, while `query_table` applies equality filters. Filtering on model-written descriptions only stays reliable if those values come from a constrained schema rather than free text that mixes "like new", "like-new", and "barely used".

 
In `app.py`, Pixeltable asks Gemini for JSON matching a Pydantic model and stores `details = response.parsed.astype(pxt.Json[ListingDetails])`, promoting fields such as `condition` into typed columns indexed with a `BtreeIndex` (an insert with a missing field or invalid enum value fails cleanly rather than storing dirty strings):

 
```python
class ListingDetails(BaseModel):
 title: str = Field(description='A listing title, under 60 characters')
 category: Literal['cameras', 'bikes', 'home', 'bags', 'instruments', 'other']
 condition: Literal['new', 'like new', 'good', 'fair']
 color: str
 description: str = Field(description='Two or three sentences for buyers, about only what the photos show')
 pitch: str = Field(description='Narration for a 10-second listing video: two short spoken sentences, 15 to 25 words')
```

 
Because `condition` is restricted to four literal strings, the agent can call `query_table` on `condition == 'like new'` without guessing phrasing variants.

 
## Keep notes across separate runs with PixeltableMemoryStore

 
`PixeltableMemoryStore` implements Harness's `MemoryStore` interface on top of a Pixeltable table (`market.memory` here), alongside the built-in in-memory, file, SQLite, and Postgres backends. On a write, the agent calls `write_memory`; on the first request of any later run, `Memory` injects an excerpt of the saved notes into context even when there is no shared conversation history. Because writes use compare-and-set versions and operation receipts, a retried tool call replays instead of duplicating a note (see the [technical walkthrough](/blog/pydantic-ai-harness-pixeltable) for receipt details and version history behavior: deleted memories stay in prior table versions unless you drop the table).

 
I ran `agent.py` as two separate processes with no chat history between them. In run 1, telling the agent "I only buy things in like-new condition. Remember that." triggered a `write_memory` call. In run 2, asking "What do you have for my reading corner?" still matched the $120 brass floor lamp first on visual similarity, but the agent applied the saved preference with `query_table` and returned only the three like-new lamps:

 
 [Image: Two windows. Your Pydantic AI agent: in run 1, the request "I only buy things in like-new condition. Remember that." and a write_memory call that saves "Buyer only buys items in like-new condition." In run 2, a separate process, the request "What do you have for my reading corner?", a similarity_search whose first match is a Brass Arched Floor Lamp in "good" condition, a query_table for condition "like new", and an answer of 3 picks, all like new. Your app: the desk lamp at $45, the mushroom table lamp at $95, and the salt lamp at $25, each like new, with the floor lamp at $120 grayed out as left out.]
 *Figure 6: A preference stored via `write_memory` in process 1 filters search results in process 2, omitting the `good` floor lamp even though it ranked first by similarity.*
 

 
## Where this is the wrong tool

 
If your agent only inspects a few known files passed directly in a request, use Pydantic AI's native multimodal inputs instead of setting up a Pixeltable table. If you only want persistent agent notes and have no media tables or embedding indexes beside them, use Harness's built-in `FileStore`, `SqliteMemoryStore`, or `PostgresMemoryStore`. If you do not want a typed `TableModel` schema, or you already operate a vector database and ingest pipeline you plan to keep, Pixeltable is the wrong tool.

 
## Run the example or wire your own agent

 
To run the complete marketplace demo yourself (requires Python 3.12 to 3.14, [uv](https://docs.astral.sh/uv/), `ffmpeg`, and `GEMINI_API_KEY` set in your environment), clone the repository, apply the schema and start the local FastAPI service, seed the sample listings using the printed service URL, and query the agent:

 
```bash
git clone https://github.com/apreshill/pixeltable-marketplace.git
cd pixeltable-marketplace
uv sync
source .venv/bin/activate
pxt schema update app.py market
pxt service update app.py market
pxt service list
python scripts/seed.py <url>
python agent.py 'I need better light for reading at night, under $150'
```

 
If you are adding the integration directly to your own Pydantic AI project instead of cloning the demo, install the extra alongside a model provider package (plus `pixeltable[serve]` if you want Pixeltable's FastAPI serving routes):

 
```bash
pip install 'pydantic-ai-harness[pixeltable]>=0.66.0' 'pydantic-ai-slim[google]'
```

 
The same `app.py` runs hosted on [Pixeltable Cloud](https://docs.pixeltable.com/cloud) via `pxt db update` when you are ready to move off a local machine.

 
**See it live on November 19.** [Laís Carvalho](https://www.linkedin.com/in/laisbsc/) from Pydantic and I will walk through this agent at 11:30am ET. [Register for the webinar](https://pydantic.dev/events/pixeltable-type-safe-media).

 
## Resources

 

 - [pixeltable-marketplace repository](https://github.com/apreshill/pixeltable-marketplace) (full `app.py`, `agent.py`, sample photos, and seed scripts)

 - [Give a Pydantic AI agent memory and read access to your tables](/blog/pydantic-ai-harness-pixeltable) (step-by-step snippet reference with outputs on the same `0.66.0` release)

 - [Pixeltable docs for Pydantic AI Harness](https://docs.pixeltable.com/integrations/pydantic-ai)

 - [Pydantic AI Harness guide to the Pixeltable capability](https://pydantic.dev/docs/ai/harness/pixeltable/)
## Frequently asked questions

### Does the Pixeltable capability let a Pydantic AI agent write to my tables or read raw media file paths?

No. The Pixeltable capability gives the agent four read-only tools: list_tables, describe_table, query_table, and similarity_search. It omits media, array, and binary columns so raw local file paths stay out of tool outputs, and it refuses version handles. Your application loads or serves media files by row ID. Agent notes go into a separate memory table through PixeltableMemoryStore.

### When can the agent find a newly added image, video, audio file, or document?

Once the insert finishes. Computed columns, iterators, and embedding indexes run as part of inserting the row into Pixeltable. If an insert calls external models or transcodes video, the new file does not appear in search results until that insert work completes.

### Does every agent search call an external embedding API?

Only when your embedding index uses a hosted embedding model such as Gemini or OpenAI. Local models such as sentence-transformers and CLIP run on your machine with no API call. With a hosted model, each similarity_search tool call embeds its query string once, and a single user request can trigger multiple similarity_search calls. Filtering stored columns with query_table makes no model call.

### When should I use Pydantic AI native media inputs or Harness file memory instead of Pixeltable?

If your agent inspects a small, known set of files per request, pass them directly with Pydantic AI native media inputs. If you only need agent memory with no media tables or embedding indexes beside it, use the in-memory, file, SQLite, or Postgres store in Pydantic AI Harness. If you do not want a typed table schema, Pixeltable is the wrong tool.