How to

How do I give a real-time AI avatar memory of past conversations?

Save every finished turn, with the user ID and session ID, in a table that embeds the text as it arrives. At the start of the next session, look up this user's closest turns and add them to the language model's prompt. The avatar API renders the face. Memory lives in your own data layer.

Updated 2026-11-02 · Part of How do I ... in Python?

Before you start

About 20 min.

Python 3.11+
pip install -U 'pixeltable[serve]'
pxt init in the project root
OpenAI API key
a voice agent or avatar session that emits transcripts

1. Decide what counts as a turn

A turn is one finished utterance: what the user said, or what the avatar said back. Record the user ID, the session ID, the speaker, and the text. Most real-time avatar platforms and voice agent frameworks give you the transcript of each utterance through an event or webhook. Post those to your table instead of the raw audio stream.

2. Store and embed each turn

Insert each turn as a row. The embedding index updates on insert, so a turn is searchable as soon as it is stored.

import pixeltable as pxt
import pixeltable.functions as pxtf
from pixeltable.functions import openai
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
embed = openai.embeddings.using(model='text-embedding-3-small')


class Turns(TableModel, name='turns'):
    id = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)
    user_id: pxt.String
    session_id: pxt.String
    speaker: pxt.String  # 'user' or 'avatar'
    text: pxt.String

    __indexes__ = [pxt.EmbeddingIndex(text, embedding=embed), pxt.BtreeIndex(user_id)]

3. Recall by user and meaning

Filter by user first, then rank by similarity to the current message. Return a handful of turns, not the whole history.

@pxt.query
def recall(user_id: str, query: str):
    sim = Turns.text.similarity(string=query)
    return (
        Turns.where(Turns.user_id == user_id)
        .order_by(sim, asc=False)
        .limit(5)
        .select(Turns.session_id, Turns.speaker, Turns.text)
    )

4. Put the memories in the prompt

When a session starts, call the memory route with the user opening message (or a fixed query like "what do I know about this user") and prepend the results to the system prompt your agent sends to its LLM. Platforms that let you bring your own LLM, or that accept extra context at session start, are the easiest to wire up. POST each finished utterance to /turns as JSON with user_id, session_id, speaker, and text. At the start of the next session, POST user_id and query to /memory; the response is this user's closest turns across sessions.

api = FastAPIRouter(name='avatar')
api.add_insert_route(Turns, path='/turns', inputs=[Turns.user_id, Turns.session_id, Turns.speaker, Turns.text], outputs=[Turns.id])
api.add_query_route(path='/memory', query=recall)

# pxt init
# pxt schema update app.py avatar_app -f
# pxt service update app.py avatar_app -f

Memory options for avatar apps

Memory options for avatar apps
Approach	What you get	What to watch
Vendor-hosted knowledge base (for example D-ID agents with your documents)	Answers from documents you upload	Per-user history may not be included; data lives with the vendor
Agent memory product (Mem0, Letta, Zep)	Extracted facts and memory APIs	Another service; transcripts and media live elsewhere
Your own table with an embedding index (this guide)	Raw turns, per-user recall, and the same table can hold recordings and generated clips	You choose what to recall and when to forget

Where Pixeltable fits

The avatar platform renders the face and speaks. Pixeltable stores what was said, keeps embeddings in sync with every insert, update, and delete, and serves recall over HTTP. The same project can hold session recordings as Video columns and transcribe them with a computed column, so turns, recordings, and generated clips sit in one place. Deleting a user's rows removes them from the index too.

import pixeltable as pxt
import pixeltable.functions as pxtf
from pixeltable.functions import openai
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
embed = openai.embeddings.using(model='text-embedding-3-small')


class Turns(TableModel, name='turns'):
    id = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)
    user_id: pxt.String
    session_id: pxt.String
    speaker: pxt.String  # 'user' or 'avatar'
    text: pxt.String

    __indexes__ = [pxt.EmbeddingIndex(text, embedding=embed), pxt.BtreeIndex(user_id)]


@pxt.query
def recall(user_id: str, query: str):
    sim = Turns.text.similarity(string=query)
    return (
        Turns.where(Turns.user_id == user_id)
        .order_by(sim, asc=False)
        .limit(5)
        .select(Turns.session_id, Turns.speaker, Turns.text)
    )


api = FastAPIRouter(name='avatar')
api.add_insert_route(Turns, path='/turns', inputs=[Turns.user_id, Turns.session_id, Turns.speaker, Turns.text], outputs=[Turns.id])
api.add_query_route(path='/memory', query=recall)

# pip install -U 'pixeltable[serve]' openai
# export OPENAI_API_KEY=...
# pxt init
# pxt schema update app.py avatar_app -f
# pxt service update app.py avatar_app -f

Questions

Does this work with Tavus, LiveAvatar, Anam, or Simli?

It works with any setup where you can read the transcript of each turn and add text to the LLM prompt. Check your platform docs for its transcript events and for how it accepts your own LLM or extra context.

Should I store audio and video too?

Store recordings if you need review or compliance. Add a Video or Audio column; transcription and indexing become computed columns on it.

How do I make the avatar forget something?

Delete the rows. The embedding index drops them at the same time, so recall stops returning them.

Is this a digital twin?

A digital twin of a person combines a face, a voice, and knowledge. This guide covers the knowledge and memory part. The face and voice come from the avatar and speech providers you pick.

Next step

Copy the commands into an agent, or run them locally with pip install pixeltable.

Get started Quick Start