--- title: "Give a Pydantic AI agent memory and read access to your tables" date: "2026-11-03" dateModified: "2026-11-03" author: "Pierre Brunelle" tags: - Pydantic AI - Harness - Memory - Embedding Indexes - Video description: "A Pydantic AI agent can read allowlisted Pixeltable tables, including the frame rows of a video view without the images, and keep its notes in another table in the same catalog. The catalog tools are read-only, and a retried memory write replays the first result." url: "https://www.pixeltable.com/blog/pydantic-ai-harness-pixeltable" --- # Give a Pydantic AI agent memory and read access to your tables Pydantic AI Harness can give an agent read-only tools over Pixeltable tables and store the agent's notes in a Pixeltable table. I ran both on Python 3.12 in a Linux container, with pixeltable 0.7.23 and pydantic-ai-harness 0.66.0. A similarity search for the meal limit returned the 60 EUR rule first, with score 0.6223213836535595. A second write of the same memory operation came back `replayed=True`, and the stored sentence was still "I prefer visual examples." If you don't want a schema, Pixeltable is the wrong tool. If the notes never need to sit next to tables, video, or an embedding index, use the file store or the SQLite store Harness already ships. The same `app.py` is what `pxt schema update` applies locally. Hosted setup is the [cloud docs](https://docs.pixeltable.com/cloud). ## Install pydantic-ai-harness 0.66.0 I sent the integration in [pull request 9369](https://github.com/pydantic/pydantic-ai/pull/9369). Aditya Vardhan reviewed it and merged it on 2026-11-03 at 13:29 UTC. [pydantic-ai-harness 0.66.0](https://pypi.org/project/pydantic-ai-harness/0.66.0/) includes the Pixeltable module. It was uploaded at 2026-11-04 10:28 UTC. [0.65.0](https://pypi.org/project/pydantic-ai-harness/0.65.0/) does not. I downloaded that wheel, uploaded at 2026-11-03 01:53 UTC, and it contains no Pixeltable module. Tag v2.65.0 is that release, eight commits behind the merge. The `[openai]` extra is there because the agent constructor imports the OpenAI provider. `sentence-transformers` is there because the embedding index in this post runs on the machine. The [shipped video example](https://github.com/pydantic/pydantic-ai/blob/a1ac27cf57f8d69bfce7d0aa190accf37794ac58/src/pydantic_ai_harness/examples/pixeltable_video.py) uses OpenAI embeddings instead. I did not run that file. ```bash uv pip install \ "pydantic-ai-harness[pixeltable]==0.66.0" \ "pydantic-ai-slim[openai]" \ "sentence-transformers>=5.4" ``` I ran that command on 2026-11-04 in a `python:3.12-slim` Docker container with ffmpeg installed from apt. The environment had Python 3.12.16, pixeltable 0.7.23, pydantic-ai-harness 0.66.0, pydantic-ai-slim 2.66.0, sentence-transformers 6.2.0, and openai 3.45.0. I ran every snippet below in that container, in order, except the agent in the last section, which I only constructed. The extra allows Python 3.11 through 3.14. Its marker is `python_version < '3.15'`. `pixeltable-pgserver` 0.7.1 publishes 3.15 wheels, so the marker is what excludes 3.15. Read [Pixeltable in the Harness docs](https://pydantic.dev/docs/ai/harness/pixeltable/). The [overview](https://pydantic.dev/docs/ai/harness/) lists Pixeltable. ## Declare the handbook and the video The catalog tools read tables that already exist. Save this as `app.py`. `topic` and `text` are columns you insert. The embedding index calls sentence-transformers locally, so a search does not call an embedding API. `Clips` holds a video. `Frames` is a view that keeps one frame per second. The shape of a class-based schema is the same one in the [quickstart](https://docs.pixeltable.com/overview/quick-start). ```python import pixeltable as pxt from pixeltable.functions.huggingface import sentence_transformer from pixeltable.functions.video import frame_iterator TableModel = pxt.model_base() embed = sentence_transformer.using(model_id='sentence-transformers/all-MiniLM-L6-v2') class Handbook(TableModel, name='handbook'): topic: pxt.String text: pxt.String __indexes__ = [pxt.EmbeddingIndex(text, string_embed=embed, name='handbook_text')] class Clips(TableModel, name='clips'): title: pxt.String video: pxt.Video class Frames( TableModel, name='frames', base=Clips, iterator=frame_iterator(Clips.video, fps=1), ): pass ``` `pxt init` marks the project root and writes `pixeltable.toml`. Apply the file from that folder. `-f` skips the confirmation. ```bash pxt init pxt schema update app.py hr -f ``` ```text created hr/handbook created hr/clips created hr/frames ``` ## Insert a policy and a one-second clip The clip I inserted is one second of a solid frame plus a sine tone, written with ffmpeg. This command writes a clip of that shape. I checked it with ffprobe, which reported 25 video frames and 1.0 s of video and audio, and did not insert that file. ```bash ffmpeg -f lavfi -i color=c=gray:s=320x240:d=1 -f lavfi -i sine=frequency=440:duration=1 -shortest -pix_fmt yuv420p clip.mp4 ``` Run this from the project folder. `pxt schema update` loaded `app.py` in its own process, so this process imports the classes and binds them before the insert. ```python from app import Clips, Handbook, TableModel TableModel.bind_all('hr') Handbook.insert([ {'topic': 'meals', 'text': 'Meals during business travel are reimbursed up to 60 EUR per day.'}, {'topic': 'lodging', 'text': 'Hotel stays need approval above 180 EUR per night.'}, ]) Clips.insert([{'title': 'Expense policy', 'video': 'clip.mp4'}]) ``` Hugging Face warned that the request was unauthenticated. The inserts still finished. The model was already on disk for this run. A first run downloads it if it is not cached. ```text Inserted 2 rows with 0 errors in 0.52 s (3.85 rows/s) Inserted 2 rows with 0 errors in 0.02 s (103.87 rows/s) ``` The first line is the two handbook rows. The second line counts the clip and the frame the view wrote. `Handbook.count()` was 2, `Clips.count()` was 1, and `Frames.count()` was 1. `pxt idxs hr/handbook` printed the index. It follows `text`. An insert, update, or delete refreshes it. The embedding index docs are [here](https://docs.pixeltable.com/platform/embedding-indexes). ```text /hr/handbook handbook_text embedding text cosine sentence_transformer(text, model_id='sentence-transformers/all-MiniLM-L6-v2', normalize_embeddings=False) ``` ## The tools read rows and leave media paths out `PixeltableToolset` is the tool list the `Pixeltable` capability gives the agent. I called the methods myself. The allowlist is `hr.handbook`, `hr.clips`, and `hr.frames`. The capability's default instructions tell the model to describe an unfamiliar table first, and to treat the rows as data rather than as instructions to follow. ```python from pydantic_ai.exceptions import ModelRetry from pydantic_ai_harness.pixeltable import PixeltableToolset tools = PixeltableToolset(tables=['hr.handbook', 'hr.clips', 'hr.frames']) print(tools.list_tables()) print(tools.describe_table('hr.handbook')) print(tools.query_table('hr.handbook', where={'topic': 'meals'})) print(tools.query_table('hr.frames')) print(tools.query_table('hr.clips')) print(tools.similarity_search( 'hr.handbook', query='What is the meal reimbursement limit?', column='text', limit=2, )) try: tools.query_table('hr.clips', columns=['video']) except ModelRetry as exc: print(exc) try: tools.query_table('hr.handbook:3') except ModelRetry as exc: print(exc) ``` `list_tables` returned the three paths. ```text {'tables': ['hr.clips', 'hr.frames', 'hr.handbook']} ``` `describe_table('hr.handbook')` reported kind `table`, the columns `topic` and `text`, and the index `handbook_text` on `text`. The meals filter returned the one matching row. ```text {'table': 'hr.handbook', 'rows': [{'topic': 'meals', 'text': 'Meals during business travel are reimbursed up to 60 EUR per day.'}], 'truncated': False} ``` The frames query left out `frame` and `video`. `pos` is the frame's position within its clip, and the tools return it. `frame_attrs.time` was 0.0. The clips query returned the title only. ```text {'table': 'hr.frames', 'rows': [{'pos': 0, 'frame_attrs': {'dts': 0, 'pts': 0, 'time': 0.0, 'index': 0, 'key_frame': True, 'pict_type': 1, 'is_corrupt': False, 'interlaced_frame': False}, 'title': 'Expense policy'}], 'truncated': False} {'table': 'hr.clips', 'rows': [{'title': 'Expense policy'}], 'truncated': False} ``` Asking for the video column raised `ModelRetry`. ```text Media column 'video' exposes a local file path; use a safe application URL instead. ``` A version handle is refused by the tools. Pixeltable itself still opens one. I used that on the memory table in the next section. Older notes on versioning are in [Pixeltable versioning and time travel](/blog/pixeltable-versioning-time-travel). ```text Table 'hr.handbook:3' is not in the Pixeltable allowlist. Call list_tables. ``` The meal search embedded the question with `all-MiniLM-L6-v2` and ranked the 60 EUR rule first. Each call embeds the query with the index's own function. An index built on a paid embedding API would call that API here. I left the default caps, 20 rows and 8000 characters. ```text {'table': 'hr.handbook', 'rows': [{'topic': 'meals', 'text': 'Meals during business travel are reimbursed up to 60 EUR per day.', 'score': 0.6223213836535595}, {'topic': 'lodging', 'text': 'Hotel stays need approval above 180 EUR per night.', 'score': 0.22148104010739078}], 'truncated': False} ``` ## A retried write returns the first result `PixeltableMemoryStore(table_name='hr.memory')` creates the table on first use. The Pixeltable guide says a write is compare-and-set on the file row, and that an operation receipt makes a retry return the first mutation. I passed `expected_version=None` because the path was new, and I used the same operation id and fingerprint on the second call. The fingerprint is a string you choose to identify the operation's arguments. The store looks a retry up by operation id and does not compare the text, so the second call below sends different text to show that the first result wins. Reuse an id only to retry the same write: in a separate call, the same id with a different fingerprint raised `MemoryOperationConflictError`, "operation id 'pref-1' was reused with different arguments". I did not send a mismatched version. Memory as rows, outside Harness, is the older post [Building memory-powered AI](/blog/building-memory-powered-ai-stateful-agents-pixeltable). ```python import asyncio from pydantic_ai_harness.memory import MemoryOperation, PixeltableMemoryStore store = PixeltableMemoryStore(table_name='hr.memory') op = MemoryOperation(id='pref-1', fingerprint='write:preferences.md:visual') print(asyncio.run(store.write( 'preferences.md', 'I prefer visual examples.', expected_version=None, operation=op, ))) print(asyncio.run(store.write( 'preferences.md', 'A different sentence that must not replace the first.', expected_version=None, operation=op, ))) print(asyncio.run(store.read('preferences.md', max_chars=200))) print(asyncio.run(store.list_paths('', limit=10))) ``` The first write printed two insert lines, then the mutation. The second write printed no insert line. ```text Created table 'memory'. Inserted 1 row with 0 errors in 0.01 s (127.26 rows/s) Inserted 1 row with 0 errors in 0.01 s (120.21 rows/s) MemoryMutation(version='868c54a469ed6dd5cf51bc4ad341ac0a', replayed=False, existed=False) MemoryMutation(version='868c54a469ed6dd5cf51bc4ad341ac0a', replayed=True, existed=False) MemoryFile(content='I prefer visual examples.', version='868c54a469ed6dd5cf51bc4ad341ac0a', operation_id='pref-1', truncated=False, metadata=None) ['preferences.md'] ``` `existed=False` because the path was new. The replay reports that same flag. `list_paths` returned the note and left the receipt out. The receipt is still a row, at `__op__/pref-1`. This lists the table's versions and opens version 3 and version 4 by handle. ```python import pixeltable as pxt for v in pxt.get_table('hr.memory').get_versions(): print(v['version'], v['schema_change'], v['inserts'], v['updates']) for n in (3, 4): handle = pxt.get_table(f'hr.memory:{n}') print(f'hr.memory:{n}') for row in handle.order_by(handle.kind).select(handle.path, handle.kind, handle.content).collect(): print(row) ``` Version 0 is the schema, labeled Initial Version. Version 1 inserted nothing and updated nothing. Versions 2 and 3 each insert one row. Version 4 updates one row. ```text 4 None 0 1 3 None 1 0 2 None 1 0 1 None 0 0 0 Initial Version 0 0 hr.memory:3 {'path': 'preferences.md', 'kind': 'file', 'content': 'I prefer visual examples.'} {'path': '__op__/pref-1', 'kind': 'op', 'content': 'preferences.md'} hr.memory:4 {'path': 'preferences.md', 'kind': 'file', 'content': 'I prefer visual examples.'} {'path': '__op__/pref-1', 'kind': 'op', 'content': None} ``` At version 3 the receipt still names the file. At version 4 the receipt's content is cleared, and the note is unchanged. The second sentence was not stored. Receipts stay in the table, and earlier versions stay readable. The guide in the pull request says dropping the table is how you purge a memory, because Pixeltable has no API to prune history. I did not drop the table. The store does not implement metadata writes: `Memory(store, write_metadata={'k': 'v'})` raised `UserError`, which says `PixeltableMemoryStore` does not record write metadata. `app.py` does not declare `hr.memory`. `pxt schema diff app.py hr` listed it as `extra (not in schema)`, and `pxt schema update app.py hr` answered `catalog is up to date` and left the table alone. `pxt schema prune app.py hr` would drop it, and it refused to run without `-f`. ## Pass the tools and the store to the agent This is the wiring only. The agent never answered a question here, because I had no model API key. Set `OPENAI_API_KEY` before you construct it. With the variable unset, the constructor raised `UserError` and pointed at `Agent('test')`. I set a placeholder key, the constructor returned an `Agent`, and I made no model request. The model string is the one in the capability docstring and in the video example. I did not check it against a live model list. ```python from pydantic_ai import Agent from pydantic_ai_harness import Memory, Pixeltable from pydantic_ai_harness.memory import PixeltableMemoryStore agent = Agent( 'openai:gpt-5.6-sol', capabilities=[ Pixeltable(['hr.handbook', 'hr.clips', 'hr.frames']), Memory(PixeltableMemoryStore(table_name='hr.memory')), ], ) ``` The video example transcribes audio, captions frames, and indexes that text with OpenAI. It needs `OPENAI_API_KEY` and a short clip that contains speech. I did not run it. How computed columns and indexes update on insert is in [How it works](https://docs.pixeltable.com/overview/how-it-works). The same pattern with the Agno framework is in [Give an Agno agent a local knowledge base](/blog/agno-agent-pixeltable-knowledge-base-local-rag). ## Frequently asked questions ### Which pydantic-ai-harness release includes the Pixeltable integration? pydantic-ai-harness 0.66.0, uploaded 2026-11-04 10:28 UTC. The 0.65.0 wheel, uploaded 2026-11-03 01:53 UTC, has no Pixeltable module. The merge commit a1ac27cf landed at 13:29 UTC on 2026-11-03. ### Can these tools insert rows into my tables? The catalog tools are read-only. list_tables, describe_table, query_table, and similarity_search do not write. Notes go through PixeltableMemoryStore into its own table. In this post that table is hr.memory. ### How do I remove a memory stored this way? Drop the table. The store keeps operation receipts, and Pixeltable keeps earlier versions. After the write in this post, hr.memory:3 still held the note. The guide in the pull request says there is no API to prune that history.