---
title: "Agent memory backend reference architecture"
description: "An agent memory backend: durable rows, embeddings, recall as a query, then a prompt. Named alternatives including Mem0, Letta, and Zep."
url: "https://www.pixeltable.com/reference-architecture/agent-memory-backend"
updated: "2026-10-28"
doc: "https://docs.pixeltable.com/howto/cookbooks/agents/pattern-agent-memory"
---

# Agent memory backend reference architecture

Agent memory is a store of facts and preferences the model did not see in the current messages. Embed the rows, recall the ones similar to this turn, and put them in the prompt. Mem0, Letta, and Zep are products for that job. Pixeltable is the table you write instead.

Updated: 2026-10-28
Docs: https://docs.pixeltable.com/howto/cookbooks/agents/pattern-agent-memory

Short-term context is the messages in this call. Long-term memory is rows that outlive the call: preferences, facts, episode summaries. The architecture is a bank, an index, a recall query, a prompt, and an optional write-back. Forgetting is a product rule (age, category, a score). This page does not invent a forget algorithm.
The interactive agent diagram on this site is /ai-agent-architecture. The docs cookbook is Build an agent with memory (notebook API). The sketch below is the same shape as a TableModel in app.py. The runnable gallery app is Research Agent Memory on /recipes.

## Agent memory: a turn recalls rows, then may write a new one {#diagram}

```mermaid
flowchart TB
  turn[User message] --> recall["recall_memories(context)"]
  bank[Memories table] --> idx[EmbeddingIndex]
  idx --> recall
  recall --> prompt[Prompt with recalled rows]
  prompt --> llm[LLM reply]
  llm --> write[Optional insert of a new fact]
  write --> bank
  idx -.->|keep if already the contract| qdrant[Qdrant / pgvector]
  bank -.->|productized memory| mem0[Mem0 / Letta / Zep]
```

## Role of each component

- **Memory bank.** Rows with content, an optional category, and a timestamp. A vector with no source string cannot be cited or deleted on purpose.
- **Embedding index.** Nearest-neighbor search over content. The same model at insert and at recall. A hosted memory product hides this index.
- **Recall query.** Rank by similarity, optionally filter by category or time, return top_k. The docs cookbook also uses a similarity threshold (sim > 0.5).
- **Prompt builder.** A UDF that joins recalled rows into the system or user prompt. The model never queries the bank itself unless you expose recall as a tool.
- **Write-back.** Insert a new fact after the turn, or let a tool do it. Extraction quality is a model problem, not an index problem.
- **Agent loop.** LangGraph, a custom loop, or Pixelagent. The loop is not the store. Pixeltable vs LangChain is that split.

## Honest alternatives per slot {#alternatives}

| Slot | Role | Pixeltable | Named alternatives |
| --- | --- | --- | --- |
| Memory product | Opinionated extract, recall, and forget | Tables, an EmbeddingIndex, and @pxt.query; you write extract and forget | Mem0: You want a hosted or embeddable memory layer with their extract/forget policy; Letta (MemGPT): You want their agent runtime and hierarchical memory, not only a table; Zep: You want a temporal knowledge graph as the memory product |
| Vector index | Semantic recall | EmbeddingIndex on the content column | Qdrant: Collections, payload filters, and an ANN SLA you already run; pgvector: Postgres already holds user data and you want SQL next to vectors; Pinecone: A hosted ANN namespace per tenant is the contract |
| Embedding model | Vectors for content and for the current turn | openai.embeddings or sentence_transformer as string_embed | OpenAI text-embedding-3-small: A hosted embedding API; used in the docs cookbook; sentence-transformers: Local embeddings with no API key |
| Agent runtime | The loop that calls recall and tools | Computed columns on a conversations table, or a @pxt.query tool | LangGraph: The loop is already LangGraph; Pixeltable is the store the tools hit; Pixelagent: You want Pixeltable examples of planning and tools |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the memory backend in this architecture: a memories table, an EmbeddingIndex, recall as @pxt.query, and a conversations table whose computed columns retrieve then generate. Keep Mem0, Letta, or Zep when their extract/forget policy is the product. Keep Qdrant or pgvector when that engine is already the index. The interactive diagram at /ai-agent-architecture is the broader agent system; this page is the store. Pixeltable Cloud hosts the same app file.

```python
import pixeltable as pxt
from pixeltable.functions.openai import chat_completions, embeddings

TableModel = pxt.model_base()
text_embed = embeddings.using(model='text-embedding-3-small')

class Memories(TableModel, name='memories'):
    content: pxt.String
    category: pxt.String
    created_at: pxt.Timestamp
    __indexes__ = [
        pxt.EmbeddingIndex(content, string_embed=text_embed),
    ]

@pxt.query
def recall_memories(context: str, top_k: int = 3):
    sim = Memories.content.similarity(string=context)
    return (
        Memories.where(sim > 0.5)
        .order_by(sim, asc=False)
        .limit(top_k)
        .select(content=Memories.content, category=Memories.category)
    )

@pxt.udf
def build_memory_prompt(user_message: str, relevant_memories: list[dict]) -> str:
    memory_text = '\n'.join(f"- {row['content']}" for row in relevant_memories)
    return (
        'You are a helpful assistant with access to the following memories:\n\n'
        f'{memory_text}\n\n'
        'Use these memories to personalize your response when relevant.\n\n'
        f'User: {user_message}'
    )

class Conversations(TableModel, name='conversations'):
    user_message: pxt.String
    relevant_memories = recall_memories(user_message, top_k=3)
    prompt = build_memory_prompt(user_message, relevant_memories)
    response = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    )
    assistant_reply = response.choices[0].message.content

# pxt init
# pxt schema update app.py agent_memory -f
# Insert facts into memories; insert turns into conversations.
```

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. [Get started](https://www.pixeltable.com/get-started) covers the CLI.

## Questions {#questions}

### How is this different from Mem0, Letta, or Zep? {#faq-1}

Those products package extract, recall, and (in some cases) forget. Pixeltable is the table, the index, and the query. Use their product when that policy is what you want to buy. Use this schema when the rows, the model, and HTTP should live in a file you own.

### Does Pixeltable decide what to forget? {#faq-2}

No. Forgetting is a rule you write: delete by category, by age, or by a score you compute. This page does not invent a forget policy.

### Can LangGraph call this store? {#faq-3}

Yes. Expose recall_memories as a tool. LangGraph runs the loop. Pixeltable vs LangChain is the data plane versus the loop.

## Related

- [AI agent architecture (interactive diagram)](https://www.pixeltable.com/ai-agent-architecture)
- [Research Agent Memory example app](https://www.pixeltable.com/recipes/research-agent-memory)
- [Chat agent example app](https://www.pixeltable.com/recipes/chat-agent)
- [AI agents and MCP use case](https://www.pixeltable.com/use-cases/ai-agents-mcp)
- [Examples hub: agent memory](https://www.pixeltable.com/examples)
- [What is agentic RAG?](https://www.pixeltable.com/learn/what-is-agentic-rag)
- [Pixeltable vs LangChain](https://www.pixeltable.com/compare/pixeltable-vs-langchain)
- [Build an agent with memory (docs)](https://docs.pixeltable.com/howto/cookbooks/agents/pattern-agent-memory)

This is the architecture. Step-by-step Python is on [how-to guides](https://www.pixeltable.com/learn/how-to) and [examples](https://www.pixeltable.com/examples).