---
title: "AI Agents & MCP: Give Your Agents Persistent Multimodal Memory"
description: "Build AI agents with durable memory and tool calling using Pixeltable. Store conversations, images, and documents as queryable tables, give a Pydantic AI agent read access and a memory table, and call MCP server tools from a computed column."
keywords:
  - AI agents MCP
  - model context protocol
  - agent memory
  - tool calling AI
  - persistent agent memory
  - MCP client python
  - agentic AI pipeline
  - multimodal agent
complexity: "intermediate"
estimated_time: "30 min"
url: "https://www.pixeltable.com/use-cases/ai-agents-mcp"
---

# AI Agents & MCP: Give Your Agents Persistent Multimodal Memory

Build AI agents with durable memory and tool calling using Pixeltable. Store conversations, images, and documents as queryable tables, give a Pydantic AI agent read access and a memory table, and call MCP server tools from a computed column.

## Prerequisites

- Familiarity with AI agents and LLMs
- Basic Python programming

## The Problem

AI agents lose context between sessions, struggle with multimodal data, and have no durable memory. Building tool-calling agents requires wiring together separate systems for memory, retrieval, and data storage.

## The Solution

Pixeltable serves as the persistent memory and data layer for AI agents. Tables store conversations, media, and structured data, and computed columns run embeddings and AI inference on insert. Through pydantic-ai-harness, a Pydantic AI agent gets read-only tools over the tables you allowlist and keeps its notes in a Pixeltable table. Pixeltable is also an MCP client: pxt.mcp_udfs() turns an MCP server's tools into functions an LLM column can call. It has no API that turns your own tables into an MCP server; its own MCP servers are for coding agents.

## Implementation

### Agent Memory Table

Create a persistent memory store for agent conversations and context.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer

pxt.create_dir('app', if_exists='ignore')

# Persistent agent memory
memory = pxt.create_table('app.agent_memory', {
    'session_id': pxt.String,
    'role': pxt.String,        # 'user', 'assistant', 'tool'
    'content': pxt.String,
    'timestamp': pxt.Timestamp,
    'metadata': pxt.Json,
}, if_exists='ignore')

# Semantic search over memory; the index updates on every insert
memory.add_embedding_index(
    'content',
    embedding=sentence_transformer.using(
        model_id='sentence-transformers/all-MiniLM-L6-v2'
    ),
    if_exists='ignore',
)

# Agents can now search their own history
sim = memory.content.similarity(string='what did we discuss about pricing?')
relevant = (
    memory.order_by(sim, asc=False)
    .select(memory.content, memory.role, memory.session_id)
    .limit(10)
    .collect()
)
```

Unlike in-memory chat history, this memory persists across sessions and supports semantic retrieval.


### Knowledge Base

Give agents access to a searchable knowledge base with documents and media.

```python
from pixeltable.functions.document import document_splitter

# Agent knowledge base: documents and metadata
knowledge = pxt.create_table('app.knowledge', {
    'document': pxt.Document,
    'title': pxt.String,
    'category': pxt.String,
}, if_exists='ignore')

# Each inserted document is split into chunks of about 300 tokens
chunks = pxt.create_view(
    'app.knowledge_chunks',
    knowledge,
    iterator=document_splitter(
        knowledge.document, separators='token_limit', limit=300
    ),
    if_exists='ignore',
)

chunks.add_embedding_index(
    'text',
    embedding=sentence_transformer.using(
        model_id='sentence-transformers/all-MiniLM-L6-v2'
    ),
    if_exists='ignore',
)

# Reusable retrieval query; pxt.tools() accepts it as an LLM tool
@pxt.query
def search_knowledge(question: str):
    """Find knowledge-base passages relevant to a question."""
    sim = chunks.text.similarity(string=question)
    return chunks.order_by(sim, asc=False).select(
        chunks.text, chunks.title
    ).limit(5)
```

Agents get grounded answers from your documents. New documents are chunked and indexed on insert.


### Pydantic AI Agent

Give a Pydantic AI agent read access to your tables and a memory it keeps in Pixeltable.

```python
# pip install "pydantic-ai-harness[pixeltable]" "pydantic-ai-slim[openai]"
from pydantic_ai import Agent
from pydantic_ai_harness import Memory, Pixeltable
from pydantic_ai_harness.memory import PixeltableMemoryStore

agent = Agent(
    'openai:gpt-4o-mini',
    capabilities=[
        # Read-only tools: list_tables, describe_table, query_table, similarity_search
        Pixeltable(['app.knowledge_chunks']),
        # The agent's notes, kept in their own Pixeltable table
        Memory(PixeltableMemoryStore(table_name='app.agent_notes')),
    ],
)

result = agent.run_sync('What does the handbook say about refunds?')
print(result.output)
```

The Pixeltable capability ships in pydantic-ai-harness 0.66.0. Its catalog tools read only the tables you list; the agent writes notes through PixeltableMemoryStore, into a table of their own.


### Tools from an MCP Server

Call any MCP server's tools from a computed column, and keep every call and result.

```python
from pixeltable.functions import openai

# Each tool the MCP server advertises becomes a Pixeltable function
mcp_tools = pxt.mcp_udfs('https://docs.pixeltable.com/mcp')
# The MCP tools and the knowledge-base query, offered to the model together
toolset = pxt.tools(*mcp_tools, search_knowledge)

questions = pxt.create_table(
    'app.questions', {'query': pxt.String}, if_exists='ignore'
)

# The model picks a tool
questions.add_computed_column(
    response=openai.chat_completions(
        messages=[{'role': 'user', 'content': questions.query}],
        model='gpt-4o-mini',
        tools=toolset,
    ),
    if_exists='ignore',
)

# Pixeltable runs the chosen tool and stores its result
questions.add_computed_column(
    tool_results=openai.invoke_tools(toolset, questions.response),
    if_exists='ignore',
)

questions.insert([{'query': 'How do I add an embedding index?'}])
```

Pixeltable is an MCP client: pxt.mcp_udfs() connects over streamable HTTP and lists the server's tools. Each tool call and its result is a row you can query later.


### Multimodal Tools

Let agents work with images, video, and audio, not just text.

```python
from pixeltable.functions.huggingface import clip

# Multimodal asset table
assets = pxt.create_table('app.assets', {
    'image': pxt.Image,
    'caption': pxt.String,
    'tags': pxt.Json,
    'uploaded_by': pxt.String,
}, if_exists='ignore')

# Visual search index
assets.add_embedding_index(
    'image',
    embedding=clip.using(model_id='openai/clip-vit-base-patch32'),
    if_exists='ignore',
)

# Find images by description; add it to pxt.tools() like search_knowledge
@pxt.query
def find_images(description: str):
    """Find images that match a description."""
    sim = assets.image.similarity(string=description)
    return assets.order_by(sim, asc=False).select(
        assets.image, assets.caption, assets.tags
    ).limit(5)
```

Agents aren't limited to text; they can search and retrieve images, video, and audio through the same interface.


## Benefits

- Persistent multimodal memory that survives across sessions
- Semantic search over agent history and knowledge bases
- Pydantic AI integration: read-only table tools and a memory table through pydantic-ai-harness
- MCP client: call any MCP server's tools from a computed column and keep every result
- Multimodal tools: agents work with images, video, audio, and documents
- Automatic indexing keeps agent knowledge always current

## Use Cases

- Customer support agents with persistent context
- Research assistants with document retrieval
- Content creation agents with media search
- DevOps agents with log analysis and monitoring
- Sales agents with CRM data and document access

## Requirements

- Python 3.11+
- pydantic-ai-harness 0.66.0 or later, for the Pydantic AI capability
- The mcp package, for pxt.mcp_udfs()
- OPENAI_API_KEY for the OpenAI calls; sentence-transformers and transformers for the local MiniLM and CLIP indexes

## Resources

- [Pydantic AI agent with Pixeltable memory](https://www.pixeltable.com/blog/pydantic-ai-harness-pixeltable) - The pydantic-ai-harness Pixeltable capability, wired up step by step
- [Tool calling and MCP servers](https://www.pixeltable.com/notebooks/llm-tool-calling) - pxt.tools(), invoke_tools() and pxt.mcp_udfs() in a notebook
- [Agent memory pattern](https://www.pixeltable.com/notebooks/pattern-agent-memory) - Persistent, searchable agent memory in a notebook
- [Model Context Protocol](https://modelcontextprotocol.io) - Official MCP specification

## How-to guides

- [How do I give a real-time AI avatar memory of past conversations?](https://www.pixeltable.com/learn/how-to/give-an-ai-avatar-memory)