Agent memory backend reference architecture

Agent memory: a turn recalls rows, then may write a new one

Role of each component

Memory bank

Rows with content, an optional category, and a timestamp. A vector with no source string cannot be cited or deleted on purpose.

Embedding index

Nearest-neighbor search over content. The same model at insert and at recall. A hosted memory product hides this index.

Recall query

Rank by similarity, optionally filter by category or time, return top_k. The docs cookbook also uses a similarity threshold (sim > 0.5).

Prompt builder

A UDF that joins recalled rows into the system or user prompt. The model never queries the bank itself unless you expose recall as a tool.

Write-back

Insert a new fact after the turn, or let a tool do it. Extraction quality is a model problem, not an index problem.

Agent loop

LangGraph, a custom loop, or Pixelagent. The loop is not the store. Pixeltable vs LangChain is that split.

Honest alternatives per slot

Keep the named engine when it is already the contract. Pixeltable fills the slot when you want that job on the same table as the files.

Where Pixeltable fits

Pixeltable is the memory backend in this architecture: a memories table, an EmbeddingIndex, recall as @pxt.query, and a conversations table whose computed columns retrieve then generate. Keep Mem0, Letta, or Zep when their extract/forget policy is the product. Keep Qdrant or pgvector when that engine is already the index. The interactive diagram at /ai-agent-architecture is the broader agent system; this page is the store. Pixeltable Cloud hosts the same app file.

import pixeltable as pxt
from pixeltable.functions.openai import chat_completions, embeddings

TableModel = pxt.model_base()
text_embed = embeddings.using(model='text-embedding-3-small')

class Memories(TableModel, name='memories'):
    content: pxt.String
    category: pxt.String
    created_at: pxt.Timestamp
    __indexes__ = [
        pxt.EmbeddingIndex(content, string_embed=text_embed),
    ]

@pxt.query
def recall_memories(context: str, top_k: int = 3):
    sim = Memories.content.similarity(string=context)
    return (
        Memories.where(sim > 0.5)
        .order_by(sim, asc=False)
        .limit(top_k)
        .select(content=Memories.content, category=Memories.category)
    )

@pxt.udf
def build_memory_prompt(user_message: str, relevant_memories: list[dict]) -> str:
    memory_text = '\n'.join(f"- {row['content']}" for row in relevant_memories)
    return (
        'You are a helpful assistant with access to the following memories:\n\n'
        f'{memory_text}\n\n'
        'Use these memories to personalize your response when relevant.\n\n'
        f'User: {user_message}'
    )

class Conversations(TableModel, name='conversations'):
    user_message: pxt.String
    relevant_memories = recall_memories(user_message, top_k=3)
    prompt = build_memory_prompt(user_message, relevant_memories)
    response = chat_completions(
        messages=[{'role': 'user', 'content': prompt}],
        model='gpt-4o-mini',
    )
    assistant_reply = response.choices[0].message.content

# pxt init
# pxt schema update app.py agent_memory -f
# Insert facts into memories; insert turns into conversations.

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. Get started covers the CLI.

Questions