---
title: "Build your own agent trace lake with Pixeltable"
date: "2026-11-03"
dateModified: "2026-11-03"
author: "Pierre Brunelle"
tags:
  - Agents
  - Observability
  - Computed Columns
  - Embedding Indexes
  - Versioning
  - Evals
  - Open Source
  - Session Log
description: "Keep LLM agent traces in a Pixeltable table you query in Python. Computed columns score each new span, an embedding index finds more like a failure, and a snapshot pins the eval set. Open source, on your machine or on Pixeltable Cloud."
url: "https://www.pixeltable.com/blog/build-agent-trace-lake-pixeltable"
---

# Build your own agent trace lake with Pixeltable

**Summary:** An [agent session log](/learn/what-is-an-agent-session-log) is a table. Put each span in Pixeltable, score it with a [computed column](/learn/what-is-a-computed-column) as it arrives, search neighbors with an [embedding index](/learn/what-is-an-embedding-index), and pin the failures with a snapshot. That is a small trace lake you can run from `pip install pixeltable`. It is not a tracing backend for 100,000 spans per second. I ran the snippets below in order on Pixeltable 0.7.23, the latest release on PyPI, in a clean Python 3.12 Docker container, with 10 synthetic spans and local `sentence-transformers` embeddings. No OpenAI or Anthropic key was present, so the LLM-as-judge block and the OpenAI embedding line did not run.

 
[The Trace Lake](https://www.braintrust.dev/blog/trace-lake) (Ankur Goyal, Braintrust, 3 November 2026) is the piece this post answers. Classic APM samples heavily and discards traces. An agent trace is the record of what the agent did, so you need to keep it, query it, and turn patterns into evals. Braintrust names three jobs: fast ingest and query, cheap storage that can stay in your cloud, and continuous analysis that produces evals. This post shows how far open-source Pixeltable goes on the last job, on your machine.

 
Braintrust gives typical sizes of about 50 KB per span and 10 MB per trace, and tens of MB per span and tens of GB per trace at p90. For ingest at that scale, use Braintrust, Langfuse, or an OpenTelemetry collector. Pixeltable sits beside them as the catalog where you enrich spans and freeze an eval set. Pixeltable can also export its own table operations as OpenTelemetry ([Tracing multimodal AI pipelines](/blog/tracing-multimodal-ai-pipelines-opentelemetry)). That is outbound tracing of Pixeltable, not ingest of agent spans.

 
## Install Pixeltable

 
Use Python 3.11 or later. The embedding index in this post uses a local MiniLM model, so install `sentence-transformers` as well. You need no API key.

 
```bash
pip install -U pixeltable sentence-transformers
```

 
Put the schema in an importable module named `app.py`. Pixeltable records each UDF by its module path, so a UDF defined in `__main__` of a pasted script will not reload. The [LLM-as-judge cookbook](https://docs.pixeltable.com/howto/cookbooks/agents/pattern-llm-judge) has the same rule.

 
## Declare a spans table

 
One row is one span. It holds a root id, a type, a model, JSON payloads, a string copy of the text you search and embed, token counts, a tool-error flag, and a timestamp. Computed columns add the token total, 0/1 flags for the aggregations, a topic, a factuality score, and an embedding index on the output text. The `Evals` table at the end of the file is the one the eval section fills.

 
```python
import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
embed = sentence_transformer.using(
 model_id='sentence-transformers/all-MiniLM-L6-v2'
)

@pxt.udf
def classify_topic(output_text: str, span_type: str, tool_error: bool) -> str:
 if tool_error:
 return 'tool_failure'
 text = (output_text or '').lower()
 if 'unable to verify' in text or "i don't know" in text:
 return 'unsupported_claim'
 if span_type == 'llm':
 return 'llm_response'
 if span_type == 'tool':
 return 'tool_ok'
 return 'other'

@pxt.udf
def heuristic_factuality(output_text: str, tool_error: bool) -> float:
 """Stand-in for an LLM-as-judge score. Swap this for the OpenAI column below."""
 if tool_error:
 return 0.12
 text = (output_text or '').lower()
 if "i don't know" in text or 'unable to verify' in text:
 return 0.28
 if 'according to' in text or 'source:' in text:
 return 0.86
 return 0.61

@pxt.udf
def flag_int(flag: bool) -> int:
 """Bool to 0/1. sum() and max() take Int; Bool.astype(Int) rejects a bool."""
 return 1 if flag else 0

class Spans(TableModel, name='spans'):
 span_id: pxt.String
 root_span_id: pxt.String
 parent_span_id: pxt.String | None
 span_type: pxt.String
 model: pxt.String | None
 input: pxt.Json
 output: pxt.Json
 input_text: pxt.String
 output_text: pxt.String
 prompt_tokens: pxt.Int
 completion_tokens: pxt.Int
 tool_error: pxt.Bool
 started_at: pxt.Timestamp
 total_tokens = prompt_tokens + completion_tokens
 tool_error_n = flag_int(tool_error)
 used_gpt4o_mini = flag_int(model == 'gpt-4o-mini')
 topic = classify_topic(output_text, span_type, tool_error)
 factuality = heuristic_factuality(output_text, tool_error)
 __indexes__ = [
 pxt.EmbeddingIndex(output_text, string_embed=embed, name='span_output_embed')
 ]

class Evals(TableModel, name='evals'):
 root_span_id: pxt.String
 input_text: pxt.String
 output_text: pxt.String
 factuality: pxt.Float
```

 
`flag_int` exists because aggregates over a Bool column are not reliable on 0.7.23. `sum()` fails with `function sum(boolean) does not exist`. `max()` collects, but `create_table(source=...)` in the query section fails to store its result with `cannot cast type boolean to bigint`.

 
`input` and `output` stay JSON, so a tool result can be a dict. `input_text` and `output_text` are the strings the index and `contains()` read. I tried `output.astype(pxt.String).contains(...)` on a dict payload, and Pixeltable 0.7.23 raises `Expected string, got dict`. Keep a string column.

 
Create the directory, then the tables. In an application the same classes go through `pxt schema update app.py trace_lake`. I ran `create_all` from Python and did not run `pxt schema update`.

 
```python
import pixeltable as pxt
from app import TableModel

pxt.create_dir('trace_lake')
TableModel.create_all('trace_lake')
```

 
```text
Created directory 'trace_lake'.
Created 'trace_lake/spans' from model `Spans`.
Created 'trace_lake/evals' from model `Evals`.
```

 
## Insert spans from the agent loop

 
`insert()` takes a list. Buffer in the loop, then flush. This snippet builds five synthetic traces of two spans each, an LLM span and a tool span, and inserts the ten spans in one call.

 
```python
from datetime import datetime, timedelta, timezone

import pixeltable as pxt

# Frozen so the 7-day window below matches the printed output.
NOW = datetime(2026, 11, 3, 21, 0, tzinfo=timezone.utc)

def span_row(*, span_id, root, parent, span_type, model, input_text, output_text,
 prompt_tokens, completion_tokens, tool_error, started_at):
 return {
 'span_id': span_id,
 'root_span_id': root,
 'parent_span_id': parent,
 'span_type': span_type,
 'model': model,
 'input': {'text': input_text},
 'output': {'text': output_text},
 'input_text': input_text,
 'output_text': output_text,
 'prompt_tokens': prompt_tokens,
 'completion_tokens': completion_tokens,
 'tool_error': tool_error,
 'started_at': started_at,
 }

# In the agent loop:
# buffer.append(span_row(...))
# if len(buffer) >= 100:
# spans.insert(buffer); buffer.clear()

# root, model, answer, prompt tokens, completion tokens, tool call failed, days ago
TRACES = [
 ('trace-a', 'gpt-4o-mini', 'I am unable to verify the Q3 revenue figure from the filing.', 48000, 2200, True, 1),
 ('trace-b', 'gpt-4o-mini', "I don't know the Q3 revenue change; the filing lookup failed.", 51000, 2760, True, 2),
 ('trace-c', 'gpt-4o-mini', 'According to the filing, Q3 revenue rose 4 percent.', 51000, 2000, False, 2),
 ('trace-d', 'claude-sonnet-4', 'Unable to verify the Q2 figure from the filing.', 52000, 3000, True, 3),
 ('trace-e', 'gpt-4o-mini', 'Unable to verify the revenue figure; the filing lookup returned an error.', 52000, 2000, True, 10),
]

rows = []
for root, model, answer, prompt_tokens, completion_tokens, tool_failed, days_ago in TRACES:
 started_at = NOW - timedelta(days=days_ago)
 rows.append(span_row(
 span_id=f'{root}-llm', root=root, parent=f'{root}-root', span_type='llm',
 model=model, input_text='Write a cited summary of Q3 revenue.',
 output_text=answer, prompt_tokens=prompt_tokens,
 completion_tokens=completion_tokens, tool_error=False, started_at=started_at,
 ))
 rows.append(span_row(
 span_id=f'{root}-tool', root=root, parent=f'{root}-llm', span_type='tool',
 model=None, input_text='sec_get(accession=000123)',
 output_text='HTTP 500 from filings API' if tool_failed else 'HTTP 200 from filings API',
 prompt_tokens=0, completion_tokens=0, tool_error=tool_failed,
 started_at=started_at - timedelta(minutes=1),
 ))

spans = pxt.get_table('trace_lake/spans')
spans.insert(rows)
```

 
```text
Inserted 10 rows with 0 errors in 0.41 s (24.15 rows/s)
```

 
On insert, Pixeltable stores the row, fills `total_tokens`, `topic`, and `factuality`, and adds the output embedding. You do not call the judge in the loop. That is the case for keeping traces in Pixeltable rather than only in a warehouse. The score is a column, not a later job. This query shows the first four rows.

 
```python
import pixeltable as pxt

spans = pxt.get_table('trace_lake/spans')
print(spans.select(
 spans.span_id, spans.total_tokens, spans.topic, spans.factuality, spans.used_gpt4o_mini
).order_by(spans.span_id).limit(4).collect())
```

 
```text
 span_id total_tokens topic factuality used_gpt4o_mini
0 trace-a-llm 50200 unsupported_claim 0.28 1
1 trace-a-tool 0 tool_failure 0.12 0
2 trace-b-llm 53760 unsupported_claim 0.28 1
3 trace-b-tool 0 tool_failure 0.12 0
```

 
## Reproduce Braintrust's example query

 
Braintrust's example (their post, their SQL dialect) asks for traces from the last seven days that used a given model, had tool errors, ran past 50,000 tokens, and scored Factuality below 0.5. They write that SQL for this is overwhelmingly verbose, and their warehouse version is a CTE with six aggregates (SUM, three COUNTs, AVG, and BOOL_OR) and a join back to the roots. Pixeltable's Python query is a rollup plus a filter. A `Query` has no `HAVING`, so you materialize the `group_by` with `create_table(source=...)` and then call `where()`. I find that shorter than the CTE.

 
The materialized table is a copy as of that call. To refresh it after new spans arrive, run the same `create_table` again. The snippet passes `if_exists='replace'`, so the first run and every refresh are the same code. In a test, an extra recent span moved `trace_summary` from 4 rows to 5 on the next run. `trace_summary` is the one table that is not declared in `app.py`, because `create_table(source=...)` is how you build a table from a query.

 
Their example used the model name `gpt-6-luna`. The sample data uses `gpt-4o-mini`.

 
```python
from datetime import datetime, timedelta, timezone

import pixeltable as pxt
import pixeltable.functions as pxtf

NOW = datetime(2026, 11, 3, 21, 0, tzinfo=timezone.utc) # the frozen clock from the insert
cutoff = NOW - timedelta(days=7)
spans = pxt.get_table('trace_lake/spans')

recent = (
 spans.where(spans.started_at >= cutoff)
 .group_by(spans.root_span_id)
 .select(
 spans.root_span_id,
 total_tokens=pxtf.sum(spans.total_tokens),
 tool_errors=pxtf.sum(spans.tool_error_n),
 used_gpt4o_mini=pxtf.max(spans.used_gpt4o_mini),
 factuality=pxtf.mean(spans.factuality),
 )
)
summary = pxt.create_table('trace_lake/trace_summary', source=recent, if_exists='replace')
failing = summary.where(
 (summary.used_gpt4o_mini >= 1)
 & (summary.tool_errors > 0)
 & (summary.total_tokens > 50000)
 & (summary.factuality < 0.5)
)
print(failing.order_by(summary.root_span_id).collect())
```

 
```text
Created table 'trace_summary'.
 root_span_id total_tokens tool_errors used_gpt4o_mini factuality
0 trace-a 50200 1 1 0.2
1 trace-b 53760 1 1 0.2
```

 
Those are the two traces that used `gpt-4o-mini`, had a tool error, crossed 50k tokens, and sat under 0.5 mean factuality. Of the other three traces, `trace-c` used the same model and 53,000 tokens but had no tool error and scored 0.735. `trace-d` had a tool error and 55,000 tokens but used Claude. `trace-e` matched the failure shape and sat ten days back, so the date filter dropped it before the rollup.

 
`mean(factuality)` averages every span in the trace, including the tool span scored 0.12. `mean()` skips nulls, so a judge that returned `None` for non-LLM spans would leave them out of the average. I checked that `mean()` over 1.0, null, and 3.0 returns 2.0. I did not run that change to the UDF.

 
Substring search over prompts and outputs is `contains()`, which compiles to SQL `LIKE`. Braintrust asked for true full-text search as part of a trace lake. `contains()` is not a full-text engine. The embedding index in the next section adds search by meaning.

 
```python
import pixeltable as pxt

spans = pxt.get_table('trace_lake/spans')

print(
 spans.where(spans.output_text.contains('unable to verify', case=False))
 .select(spans.span_id, spans.output_text)
 .order_by(spans.span_id)
 .collect()
)
```

 
```text
 span_id output_text
0 trace-a-llm I am unable to verify the Q3 revenue figure from the filing.
1 trace-d-llm Unable to verify the Q2 figure from the filing.
2 trace-e-llm Unable to verify the revenue figure; the filing lookup returned an error.
```

 
## Score every new span as it arrives

 
A [computed column](/learn/what-is-a-computed-column) is not a job you schedule. It is part of the table. Insert a span, and Pixeltable runs `classify_topic` and `heuristic_factuality` for that row only. [Unchanged rows stay as they are](/learn/what-is-incremental-computation). If you change the judge expression later, new rows use it, and existing rows wait for `pxt recompute`.

 
The functions above are keyword stand-ins so the catalog can run without an API key. The published Pixeltable form of an LLM judge is a second model call on the same table, as in the [LLM-as-judge cookbook](https://docs.pixeltable.com/howto/cookbooks/agents/pattern-llm-judge). I did not execute this block, because it needs `OPENAI_API_KEY`. `extract_score` is the short UDF from that cookbook, which returns `None` when the judge ignores the format. Build the prompt with `pxtf.string.format`, not Python's `str.format`, or every row gets the column name instead of the value.

 
```python
import pixeltable.functions as pxtf
from pixeltable.functions import openai

JUDGE = """You are scoring factuality from 0 to 1.
Prompt: {prompt}
Response: {response}
Reply on exactly two lines:
Score: <0-1>
Explanation: <one sentence>
"""

# On the same Spans class, instead of heuristic_factuality(...):
# judge_prompt = pxtf.string.format(
# JUDGE, prompt=input_text, response=output_text
# )
# judge_response = openai.chat_completions(
# messages=[
# {'role': 'system', 'content': 'You grade factuality. Be strict.'},
# {'role': 'user', 'content': judge_prompt},
# ],
# model='gpt-4o-mini',
# )
# verdict = judge_response.choices[0].message.content
# factuality = extract_score(verdict) # the UDF from the cookbook
```

 
The same swap works for embeddings. I ran MiniLM locally. The OpenAI index is `pxt.EmbeddingIndex(output_text, string_embed=openai.embeddings.using(model='text-embedding-3-small'))`. I did not run that line.

 
## Find more traces like this one

 
Braintrust asks whether, after you find an agent that changed languages mid-sentence, you can find 20 more. On an embedding index, that is `similarity()`. I probed with the failing Q3 answer. The index is cosine over MiniLM of `output_text`. These are the top five of the ten spans:

 
```python
import pixeltable as pxt

spans = pxt.get_table('trace_lake/spans')

probe = 'Unable to verify the Q3 revenue figure from the filing.'
sim = spans.output_text.similarity(string=probe)
print(
 spans.order_by(sim, asc=False)
 .limit(5)
 .select(spans.span_id, spans.root_span_id, spans.output_text, sim=sim)
 .collect()
)
```

 
```text
 span_id root_span_id output_text sim
0 trace-a-llm trace-a I am unable to verify the Q3 revenue figure from the filing. 0.979689
1 trace-b-llm trace-b I don't know the Q3 revenue change; the filing lookup failed. 0.812841
2 trace-d-llm trace-d Unable to verify the Q2 figure from the filing. 0.766443
3 trace-e-llm trace-e Unable to verify the revenue figure; the filing lookup returned an error. 0.735087
4 trace-c-llm trace-c According to the filing, Q3 revenue rose 4 percent. 0.701820
```

 
Use `.limit(20)` on a real lake. The index stays current on insert, update, and delete, so you do not rebuild it. It is the same incremental index as in [What is an embedding index?](/learn/what-is-an-embedding-index).

 
## Pin a failing set as an eval table

 
[Reproducibility](/learn/what-is-reproducibility) here means the eval rows do not move when production keeps ingesting. Copy the prompts of the failing spans into the `Evals` table, then snapshot that table. It is the set you rerun a new judge or a new model against. The snippet selects the LLM spans from the last seven days with `factuality < 0.5` and `model == 'gpt-4o-mini'` (2 rows), inserts them, snapshots the table, and then inserts one more live row. The live table has 3 rows, and the snapshot keeps 2.

 
```python
from datetime import datetime, timedelta, timezone

import pixeltable as pxt

NOW = datetime(2026, 11, 3, 21, 0, tzinfo=timezone.utc) # the frozen clock from the insert
cutoff = NOW - timedelta(days=7)
spans = pxt.get_table('trace_lake/spans')
evals = pxt.get_table('trace_lake/evals')

sample = (
 spans.where(
 (spans.started_at >= cutoff)
 & (spans.span_type == 'llm')
 & (spans.model == 'gpt-4o-mini')
 & (spans.factuality < 0.5)
 )
 .select(spans.root_span_id, spans.input_text, spans.output_text, spans.factuality)
 .collect()
)
evals.insert(list(sample))
pinned = pxt.create_snapshot('trace_lake/evals_pinned', evals)

evals.insert([{
 'root_span_id': 'trace-f',
 'input_text': 'Write a cited summary of Q3 revenue.',
 'output_text': 'Unable to verify the figure.',
 'factuality': 0.28,
}])
print(evals.count(), pinned.count())
```

 
```text
Inserted 2 rows with 0 errors in 0.01 s (173.62 rows/s)
Created snapshot 'trace_lake/evals_pinned'.
Inserted 1 row with 0 errors in 0.00 s (246.62 rows/s)
3 2
```

 
Every insert also creates a table version. `spans.history()` after the insert shows version 0 (the initial version) and version 1, and `pxt.get_table('trace_lake/spans:1').count()` returns 10, the whole batch. The [version-control docs](https://docs.pixeltable.com/platform/version-control) cover `revert()` and [time travel](/learn/what-is-time-travel). Pixeltable is not an eval framework. It is the data layer an eval loop reads, which is the argument in [You Don't Need an Eval Framework](/blog/pixeltable-not-eval-framework) and in [What is ML evaluation?](/learn/what-is-ml-evaluation).

 
Keep the context that produced the trace on the same row or a parent row. That means the prompt, the tool arguments, the model, and any resource version you already store. A snapshot of that table is the reproduction bundle. [Agents as Data](/blog/agents-as-data-session-log-system-of-record) and [Where Does the Agent Live?](/blog/where-does-the-agent-live) make the same claim, that the log is the agent and not the process.

 
## Where the files live

 
Locally, Pixeltable writes an embedded Postgres catalog and media files under `PIXELTABLE_HOME` (default `~/.pixeltable`). That directory can sit on a disk in your cloud. Nothing in this post sent a prompt to a tracing vendor. See [What is a local catalog?](/learn/what-is-a-local-catalog) and the [self-hosting guide](https://docs.pixeltable.com/howto/deployment/overview).

 
Pixeltable Cloud can host the same `app.py`. The [Cloud docs](https://docs.pixeltable.com/cloud) list the commands in order. They start with `pxt login` and `pxt init` and end with `pxt schema update app.py pxt://org:main`, which creates the tables. This app has no HTTP routes, so it has no `pxt service update` step. I did not run any Cloud step for this app. Community is the free plan. A walkthrough of serving a table as HTTP is [How to Serve a Typed AI Endpoint](/blog/serve-typed-ai-endpoint-pixeltable).

 
## When to use a dedicated tracing product

 
I did not measure an ingest rate for Pixeltable, and I am not going to invent one. If you need Braintrust's 100,000 spans per second, live tail, sampling, or cheap object storage for gigabyte traces, use the product built for that. Braintrust's post is a clear description of that product. Langfuse, Phoenix, and others take the same OpenTelemetry export Pixeltable already has.

 
Use Pixeltable when the next question is "score this, find more like it, freeze an eval." That work wants a schema, computed columns, and versions. The same argument, from the other side of the stack, is [the agent data plane is not a dashboard](/blog/agent-data-plane-not-dashboard).

 
## What to do next

 

 - [Docs quickstart](https://docs.pixeltable.com/overview/quick-start) and [Get started](/get-started) (skill, CLI, Cloud)

 - [LLM as judge](https://docs.pixeltable.com/howto/cookbooks/agents/pattern-llm-judge), [Embedding indexes](https://docs.pixeltable.com/platform/embedding-indexes), [Version control](https://docs.pixeltable.com/platform/version-control)

 - [OpenTelemetry export](/blog/tracing-multimodal-ai-pipelines-opentelemetry) of Pixeltable table operations to Langfuse or Phoenix

 - [The Trace Lake](https://www.braintrust.dev/blog/trace-lake) for the argument this post builds on
## Frequently asked questions

### How do I store agent traces in open-source Pixeltable?

Declare a Spans table with root span id, span type, model, JSON input and output, token counts, a tool-error flag, and a timestamp. pip install pixeltable, then TableModel.create_all() or pxt schema update. Insert a list of span dicts from the agent loop. The catalog lives in PIXELTABLE_HOME on your machine, or on Pixeltable Cloud.

### How do I query LLM traces for tool errors, token count, and a low judge score?

Filter spans by started_at, group_by root_span_id, and aggregate with sum, max, and mean. Materialize that rollup with create_table(source=query), then where() for the model flag, tool_errors > 0, total_tokens above your cutoff, and mean factuality below 0.5. Pixeltable has no HAVING on the Query object, so the extra table is the filter step.

### How do I turn production agent traces into a reproducible eval set?

Filter the traces you care about, insert those prompts and outputs into an evals table, and call pxt.create_snapshot() on it. Later inserts on the live table do not change the snapshot. Table versions are available as pxt.get_table("trace_lake/spans:N").

### Can Pixeltable replace Braintrust or Langfuse for agent observability?

No, not as a high-throughput tracing backend. Braintrust describes production systems on the order of 100,000 spans per second. Pixeltable is the place you analyze traces, run judges and embeddings as computed columns, and pin evals. Keep Braintrust, Langfuse, or OpenTelemetry for ingest, and insert or export into Pixeltable when you want that analysis.

### Where does Pixeltable store traces, and do they leave my cloud?

Locally, the catalog is embedded Postgres plus files under PIXELTABLE_HOME, on the machine that runs Pixeltable. You can point that home directory at a disk you already run. Pixeltable Cloud hosts the same tables. Nothing here requires sending prompts to a tracing vendor.