--- title: "The context architecture for an AI teammate" date: "2026-11-06" dateModified: "2026-11-06" author: "Pierre Brunelle" tags: - Agents - Agent State - Session Log - AI Infrastructure - Durable State - Computed Columns - Embedding Indexes - Versioning - Multimodal AI - Pixeltable description: "A proactive AI teammate needs conversations, a searchable inbox copy, curated memory, and versioned drafts. Those are four tables in one Pixeltable catalog, not a SQLite database per session." url: "https://www.pixeltable.com/blog/context-architecture-ai-teammate" --- # The context architecture for an AI teammate **Summary:** If you were building a proactive AI teammate, the data layer is four tables in one Pixeltable catalog: conversation turns, a searchable copy of inbox and calendar documents, curated memory, and versioned drafts with image, audio, video, and document columns. That is the difference from a SQLite database per session. The teammate reads all four from one catalog, searches the copy instead of calling Gmail on every turn, and re-runs only the columns that a changed prompt, model, or rule actually touches. I ran the schema below on Pixeltable 0.7.24 from PyPI, with a local MiniLM embedding index and no model API key. The four stores come from [Turso's post on Sauna](https://turso.tech/blog/2-million-databases-and-counting-the-architecture-behind-saunas-ai-teammate), a work AI teammate. This post maps those stores onto Pixeltable tables. [Where Does the Agent Live?](/blog/where-does-the-agent-live) makes the companion argument: the agent lives in durable state, not in the sandbox that runs it. ## Sauna splits context across four stores Sauna is a proactive work AI teammate. It reads Slack, Gmail, calendar, and Granola, and it is built around compounding context. Each repetition adds more of what the team already knows. The numbers and the four-store split below are from the Turso post. Every session and agent a user starts gets its own Turso database. So does every inbox and calendar they connect. Sauna created over 2 million Turso databases, about 90,000 a day. The architecture has four pieces. Two run on Turso. The Sauna team built the other two in-house: - **Conversations.** A Turso database per session, holding that chat history. - **Synced connections.** A Turso database per inbox or calendar, kept in sync, so the teammate searches a local copy instead of calling Gmail on every turn. - **Memory system.** A two-tier system, in-house, that curates relationships and information about you and the company, and pulls earlier conversations back when a topic returns. - **File system.** A custom, versioned, searchable store for the drafts, documents, and proposals the agent works on, also in-house. Isolation is a security boundary. Sharing an inbox with the assistant means sharing that one database, not relying on row filters to keep the rest private. Matthias Pichler, Founding Engineer at Wordware, puts the boundary this way in that post: "a lot easier to reason about and a lot harder to break." Sauna ran SQLite from the start. The path in the Turso post is Cloudflare Durable Objects SQLite, then Containers plus Litestream, then Turso after May 2026, when Turso removed the per-database active limit on paid plans. ## Map those four stores onto one catalog Pixeltable is a multimodal backend, not a vector database. The same four concerns become tables in one catalog: | Sauna store | In Pixeltable | | --- | --- | | Conversation / session history | A turns table keyed by session_id | | Synced inbox or calendar | An inbox table you search, so you are not calling the provider on every turn | | Curated memory | A memories table you write on purpose | | Versioned drafts and media | A drafts table with a revision column and pxt.Image, pxt.Audio, pxt.Video, pxt.Document | That is the same catalog boundary as [Where Does the Agent Live?](/blog/where-does-the-agent-live). Session log, memory, media, and versions sit in one catalog, each table with its own version history, instead of in five systems. The session log as system of record is [Agents as Data](/blog/agents-as-data-session-log-system-of-record). Queryable state versus a dashboard that only replays traces is [data plane, not dashboard](/blog/agent-data-plane-not-dashboard). ## Four tables in one app.py Python 3.11 or later. I used 3.12.3. The embedding index in this post runs MiniLM on the machine, so install `sentence-transformers` as well. You need no API key. ```bash pip install -U pixeltable sentence-transformers pxt init # app.py holds the four TableModel classes pxt schema update app.py teammate -f ``` `pxt init` marks the project root and writes `pixeltable.toml`. `pxt schema update` creates the tables. Pass `-f` in a script. I ran Pixeltable 0.7.24 and sentence-transformers 6.2.0. The shape is the [quickstart](https://docs.pixeltable.com/overview/quick-start). ```python import pixeltable as pxt from pixeltable.functions.huggingface import sentence_transformer TableModel = pxt.model_base() embed = sentence_transformer.using(model_id='sentence-transformers/all-MiniLM-L6-v2') @pxt.udf def first_words(text: str, n: int = 12) -> str: words = text.split() if len(words) <= n: return text return ' '.join(words[:n]) + '...' class Turns(TableModel, name='turns'): session_id: pxt.String tenant_id: pxt.String role: pxt.String content: pxt.String occurred_at: pxt.Timestamp class Inbox(TableModel, name='inbox'): tenant_id: pxt.String source: pxt.String provider_id: pxt.String subject: pxt.String body: pxt.String occurred_at: pxt.Timestamp excerpt = first_words(body) __indexes__ = [pxt.EmbeddingIndex(body, string_embed=embed, name='inbox_body')] class Memories(TableModel, name='memories'): tenant_id: pxt.String topic: pxt.String note: pxt.String excerpt = first_words(note) __indexes__ = [pxt.EmbeddingIndex(note, string_embed=embed, name='memory_note')] class Drafts(TableModel, name='drafts'): session_id: pxt.String tenant_id: pxt.String title: pxt.String revision: pxt.Int body: pxt.String image: pxt.Image | None audio: pxt.Audio | None video: pxt.Video | None document: pxt.Document | None excerpt = first_words(body) ``` A line with a colon is a column you insert. A line with `=` is a [computed column](/learn/what-is-a-computed-column). `first_words` is a stand-in for a summarizer or a model. Pixeltable records the function by its module path and fills the column on insert. I ran this UDF. I did not call an LLM. ```text created teammate/turns created teammate/inbox created teammate/memories created teammate/drafts ``` `pxt schema update` loaded `app.py` in its own process. Inserts from Python need a bind first: ```python from app import Drafts, Inbox, Memories, TableModel, Turns TableModel.bind_all('teammate') ``` ## Conversation history is rows One row is one turn. Isolation is `tenant_id` and `session_id`, not a file per chat. That is an [agent session log](/learn/what-is-an-agent-session-log). ```python from datetime import datetime, timedelta, timezone now = datetime(2026, 11, 6, 15, 4, 0, tzinfo=timezone.utc) Turns.insert([ { 'session_id': 'alice-42', 'tenant_id': 'alice', 'role': 'user', 'content': 'Draft a reply to the Q4 budget review.', 'occurred_at': now, }, { 'session_id': 'alice-42', 'tenant_id': 'alice', 'role': 'assistant', 'content': 'I will search the inbox copy and open a draft.', 'occurred_at': now + timedelta(seconds=2), }, { 'session_id': 'bob-7', 'tenant_id': 'bob', 'role': 'user', 'content': 'What is on my calendar tomorrow?', 'occurred_at': now, }, ]) ``` ```text Inserted 3 rows with 0 errors in 0.01 s (268.25 rows/s) ``` ```python alice = (Turns.tenant_id == 'alice') & (Turns.session_id == 'alice-42') print( Turns.where(alice) .order_by(Turns.occurred_at) .select(Turns.role, Turns.content) .collect() ) print('alice session count', Turns.where(alice).count()) print('bob session count', Turns.where((Turns.tenant_id == 'bob') & (Turns.session_id == 'bob-7')).count()) ``` ```text role content 0 user Draft a reply to the Q4 budget review. 1 assistant I will search the inbox copy and open a draft. alice session count 2 bob session count 1 ``` Alice's filter does not return Bob's calendar question. ## Search the copy, not the provider Sauna's synced connections exist so the teammate can search mail and events without calling the provider for every lookup. In Pixeltable that copy is rows. I inserted three documents, two for Alice and one for Bob. I did not call Gmail. ```python Inbox.insert([ { 'tenant_id': 'alice', 'source': 'gmail', 'provider_id': 'msg-1', 'subject': 'Q4 budget review', 'body': 'Please send the Q4 budget draft before Tuesday 10:00. Include headcount and travel.', 'occurred_at': now - timedelta(days=1), }, { 'tenant_id': 'alice', 'source': 'calendar', 'provider_id': 'evt-1', 'subject': 'Q4 budget review', 'body': 'Tuesday 10:00 in the finance room. Alice presents the draft.', 'occurred_at': now + timedelta(days=3), }, { 'tenant_id': 'bob', 'source': 'gmail', 'provider_id': 'msg-9', 'subject': 'Lunch', 'body': 'Can we move lunch to 12:30?', 'occurred_at': now, }, ]) ``` ```text Inserted 3 rows with 0 errors in 0.21 s (13.99 rows/s) ``` Hugging Face warned that the request was unauthenticated. The insert still finished. The MiniLM weights were already on disk for this run. A first run downloads them if they are not cached. `pxt idxs teammate/inbox` shows the index on `body`: ```text /teammate/inbox inbox_body embedding body cosine sentence_transformer(body, model_id='sentence-transformers/all-MiniLM-L6-v2', normalize_embeddings=False) ``` ```python sim = Inbox.body.similarity(string='budget review Tuesday') print( Inbox.where(Inbox.tenant_id == 'alice') .order_by(sim, asc=False) .limit(2) .select(Inbox.source, Inbox.subject, Inbox.body, sim) .collect() ) ``` ```text source subject body similarity 0 gmail Q4 budget review Please send the Q4 budget draft before Tuesday 10:00. Include headcount and travel. 0.458107 1 calendar Q4 budget review Tuesday 10:00 in the finance room. Alice presents the draft. 0.351094 ``` The Gmail copy ranked above the calendar copy for that query. The filter kept Bob's lunch mail out. An [embedding index](/learn/what-is-an-embedding-index) updates on insert. How computed columns and indexes follow the column is in [How it works](https://docs.pixeltable.com/overview/how-it-works). ## Memory is a row you write on purpose Conversation history is everything that happened. Memory is the curated subset you want back later: how Alice likes replies written, who owns the budget, which client thread is current. Sauna built that tier in-house. In Pixeltable it is another table in the same catalog. ```python Memories.insert([ { 'tenant_id': 'alice', 'topic': 'writing', 'note': 'Alice prefers short replies and a numbered list of next steps.', } ]) ``` ```text Inserted 1 row with 0 errors in 0.02 s (60.93 rows/s) ``` A Pydantic AI agent can keep notes in a Pixeltable table the same way: [Give a Pydantic AI agent memory and read access to your tables](/blog/pydantic-ai-harness-pixeltable). I did not run that integration here. ## Drafts keep revisions and media columns Sauna's file system is versioned and searchable, and it is where drafts live. In Pixeltable a draft is a row in a versioned table. Media is a typed column, not a path you copy into object storage by hand. I generated a 64x48 PNG, a one-second sine wav, a one-second gray clip, and a short text file, then inserted them on revision 1. To follow along, put any image, wav, mp4, and text file at these paths. ```python Drafts.insert([ { 'session_id': 'alice-42', 'tenant_id': 'alice', 'title': 'Q4 budget reply', 'revision': 1, 'body': 'Thanks. I will bring the Q4 budget draft on Tuesday, with headcount and travel.', 'image': '/tmp/pxt-teammate/media/note.png', 'audio': '/tmp/pxt-teammate/media/note.wav', 'video': '/tmp/pxt-teammate/media/clip.mp4', 'document': '/tmp/pxt-teammate/media/brief.txt', } ]) ``` ```text Inserted 1 row with 0 errors in 0.03 s (36.91 rows/s) ``` `pxt describe teammate/drafts` printed the media columns as `Image | None`, `Audio | None`, `Video | None`, and `Document | None`. Collecting the row returned a 64x48 RGB PNG, and file paths for the wav, the mp4, and the text document. A Markdown file failed until I installed `mistune`, so I used `.txt`, which Pixeltable already accepts. Revision 2 is another row, not an overwrite: ```python Drafts.insert([ { 'session_id': 'alice-42', 'tenant_id': 'alice', 'title': 'Q4 budget reply', 'revision': 2, 'body': 'Thanks. Draft attached. I will present headcount and travel on Tuesday at 10:00.', 'image': None, 'audio': None, 'video': None, 'document': None, } ]) print( Drafts.where((Drafts.tenant_id == 'alice') & (Drafts.session_id == 'alice-42')) .order_by(Drafts.revision) .select(Drafts.revision, Drafts.excerpt) .collect() ) ``` ```text Inserted 1 row with 0 errors in 0.01 s (81.61 rows/s) revision excerpt 0 1 Thanks. I will bring the Q4 budget draft on Tuesday, with headcount... 1 2 Thanks. Draft attached. I will present headcount and travel on Tuesday at... ``` The catalog also versions the table. After the second insert, `pxt history teammate/drafts` showed schema version 0, then one insert, then another: ```text version created_at change_type inserts updates deletes errors schema_change 2 2026-11-07T05:15:49.568007Z data 1 0 0 0 1 2026-11-07T05:15:10.008239Z data 1 0 0 0 0 2026-11-07T05:15:00.298394Z schema 0 0 0 0 Initial Version ``` ## Isolation is a filter, not a file per chat Sauna treats a database as the thing you share or withhold. That is a real advantage when the security story is "this inbox is this file." Pixeltable does not give you one SQLite file per conversation. Isolation is tenant and session scoping: `tenant_id`, `session_id`, and the authorization in your server. A runtime key to Pixeltable Cloud does not know Alice from Bob. [Authentication and Multi-Tenancy for Pixeltable Apps](/blog/auth-multitenancy-pixeltable-backend) is the post for that boundary. The `tenant_id` column records ownership. Your query has to filter on the verified id. That is not row-level security inside the engine. If the product requirement is a physical database per tenant, and you want that to be the thing you hand to an agent, Turso is built for it. If the product requirement is one versioned catalog for every store, with indexes and recompute, a file per chat is the wrong unit. ## A changed rule touches only the column that depends on it Change a prompt, a model, or a rule, and only the column that depends on it changes. I changed the `Inbox` excerpt from 12 words to 4, for every tenant: ```python class Inbox(TableModel, name='inbox'): ... excerpt = first_words(body, n=4) ``` ```bash pxt schema diff app.py teammate pxt schema update app.py teammate -f ``` ```text = teammate/turns no change ~ teammate/inbox update ~ the value expression of computed column 'excerpt' will be updated; existing values will not be recomputed safe = teammate/memories no change = teammate/drafts no change Plan: 0 create, 1 update, 3 unchanged, 0 extra | 0 destructive ``` `pxt schema update` then printed that `teammate/inbox.excerpt` changed and that stored values were not recomputed. The Gmail excerpt was still the 12-word cut. New rows would get 4 words. Existing rows wait for `pxt recompute`: ```bash pxt recompute teammate/inbox excerpt -f ``` ```text recomputed inbox.excerpt on teammate/inbox: 3 rows updated ``` After that, Alice's Gmail excerpt was `Please send the Q4...` and the calendar excerpt was `Tuesday 10:00 in the...`. Turns, memories, and drafts were not touched. [How it works](https://docs.pixeltable.com/overview/how-it-works) documents that loop. An insert computes new rows, a changed expression applies to new rows, and `pxt recompute` redoes the existing ones. Swap `first_words` for a model column and the commands stay the same. I did not run a model column. Guarded transitions over work in a table are [How to build an agent state machine in Pixeltable](/blog/agent-state-machine-pixeltable). Scoring spans as they arrive is [Build your own agent trace lake with Pixeltable](/blog/build-agent-trace-lake-pixeltable). ## When one SQLite per session is the right tool Use Turso when you want tiny isolated SQL documents, sub-millisecond embedded replicas, a database per tenant, an edge runtime, and no multimodal pipeline. That choice, without recapping its feature tables, is [Pixeltable vs Turso](/compare/pixeltable-vs-turso). If you don't want a schema, Pixeltable is the wrong tool. If the app needs `DATABASE_URL`, SQL, branching, or row-level security, and insert should not run a model, that host list is [Best serverless Postgres databases for AI apps](/blog/best-serverless-postgres-for-ai-apps). ## Try it Put the four classes in `app.py`, run `pxt schema update app.py teammate -f`, and insert a turn. Hosted setup is the [Cloud docs](https://docs.pixeltable.com/cloud). `pxt new` makes a free hosted database with no account and prints its organization. Put that organization in place of `ORG` to create the same tables there. I did not run the hosted step. ```bash npx skills add pixeltable/pixeltable-skill pxt new pxt schema update app.py pxt://ORG:main -f ``` ## See also - [Where Does the Agent Live?](/blog/where-does-the-agent-live) - [Agents as Data](/blog/agents-as-data-session-log-system-of-record) - [Your agent needs a data plane, not a dashboard](/blog/agent-data-plane-not-dashboard) - [Build your own agent trace lake with Pixeltable](/blog/build-agent-trace-lake-pixeltable) - [How to build an agent state machine in Pixeltable](/blog/agent-state-machine-pixeltable) - [Give a Pydantic AI agent memory and read access to your tables](/blog/pydantic-ai-harness-pixeltable) - [Best serverless Postgres databases for AI apps](/blog/best-serverless-postgres-for-ai-apps) - [Authentication and Multi-Tenancy for Pixeltable Apps](/blog/auth-multitenancy-pixeltable-backend) - [Pixeltable vs Turso](/compare/pixeltable-vs-turso) ## Frequently asked questions ### What is the context architecture for an AI teammate? Four stores sit next to the model: conversation history, a searchable copy of connected inboxes and calendars, curated memory, and versioned drafts. In Pixeltable those stores are tables in one catalog, with image, audio, video, and document columns on the drafts table. Isolation is a tenant_id filter, with session_id for turns and drafts, plus the checks in your server, not a physical database per chat. ### When is one SQLite database per session the wrong tool? When conversation, inbox copy, memory, and drafts should live in one catalog, media should be typed columns, or a changed prompt, model, or rule should recompute only the columns that depend on it. Turso remains the better fit for tiny isolated SQL documents, sub-millisecond embedded replicas, a database per tenant, edge runtimes, and no multimodal pipeline. ### How does Pixeltable isolate conversations if it is not one database per chat? Every row carries a tenant_id; turns and drafts also carry a session_id. Queries filter on the verified identity. Pixeltable does not apply row-level security for you. That is application-enforced authorization, described in Authentication and Multi-Tenancy for Pixeltable Apps. ### How do you search Gmail or calendar without calling the provider on every turn? Sync a copy into an inbox table, then search that copy. The listing in this post uses a local MiniLM embedding index on the body column. New rows update the index. I did not call Gmail, Slack, or a calendar API. ### Does Pixeltable replace Turso? No. Pixeltable is a multimodal backend: tables, computed columns, embedding indexes, and media types in one Python file. Turso is distributed SQLite with embedded replicas. Use Turso when you want a database per tenant at the edge. Use Pixeltable when insert should embed, index, and keep drafts next to the session log. Pixeltable vs Turso is the comparison page.