---
title: "Tracing multimodal AI pipelines with OpenTelemetry: Pixeltable + Langfuse, Phoenix and MLflow"
date: "2026-10-21"
author: "Pierre Brunelle"
tags:
  - OpenTelemetry
  - Observability
  - Langfuse
  - Phoenix
  - MLflow
  - Multimodal AI
  - Computed Columns
description: "Pixeltable exports one OpenTelemetry trace per table operation, with a span tree for rows, files, and UDF calls. Point that same trace at Langfuse, Arize Phoenix, or MLflow."
url: "https://www.pixeltable.com/blog/tracing-multimodal-ai-pipelines-opentelemetry"
---

# Tracing multimodal AI pipelines with OpenTelemetry: Pixeltable + Langfuse, Phoenix and MLflow

**Summary:** OpenTelemetry data pipeline tracing in Pixeltable is one trace per table operation. `insert`, `collect`, `update`, and the rest each root a span named `pixeltable.<operation>`. Under it sits the work the engine actually did: catalog and store steps, and, when you ask, a span per row, per file, and per UDF call. Every attribute the runtime records is prefixed `pxt.`: `pxt.table` and `pxt.num_rows` on an insert or update, `pxt.tables` and `pxt.rows` on a query such as `collect`, and `pxt.column`, `pxt.udf_path`, `pxt.resource_pool`, and `pxt.retries` on the UDF call. To trace multimodal pipelines, the same tree grows a span per downloaded file and, at the finest level, a span per iterator step such as a video frame. Point the exporter at Langfuse, Arize Phoenix, or MLflow. The catalog does not move with the trace.

 
This was checked on 2026-10-21 against Pixeltable 0.7.19 and `opentelemetry-instrumentation-pixeltable` 0.1.0 in [that tag of the Pixeltable repo](https://github.com/pixeltable/pixeltable/tree/v0.7.19/packages/opentelemetry-instrumentation-pixeltable), with the Langfuse SDK 5.0.4, `openinference-instrumentation-openai` 0.1.65, `openai` 3.36.0, and the Anthropic Python SDK 2.0.0. The attribute names and span levels below are from that source, not from a vendor diagram.

 
## One operation, one trace

 
A [declarative pipeline](/learn/what-is-a-declarative-pipeline) is a schema. The trace is the engine running that schema, not a DAG you assembled by hand. An insert of a row whose computed column calls a model looks like this:

 
```text
pixeltable.insert root of the trace
└─ pixeltable.store.insert_rows
 ├─ pixeltable.row one per row, at span_level='debug'
 ├─ pixeltable.udf.chat_completions one per LLM cell, beside the row span
 │ └─ LLM span OpenInference or the provider SDK: model, tokens, cost
 └─ pixeltable.sa.insert_rows the SQL insert
```

 
Pixeltable's own spans say which operation, row, column, and UDF ran. They do not carry row ids or cell values. A provider UDF such as `chat_completions` records its token counts as a `pixeltable.udf.tokens` event (`pxt.input_tokens`, `pxt.output_tokens`) on the UDF span. An LLM instrumentor adds the model call as a child span, with the prompt, the completion, and its own token counts. The bridge sets the active OpenTelemetry context for every UDF call, including calls on its thread pools and its async scheduler, so the generation nests under `pixeltable.udf.<name>`.

 
The operation span is filled in when the operation finishes. From `OpAttrs` in `pixeltable/telemetry_schemas.py`, that includes `pxt.table` (the table name), `pxt.table_id`, `pxt.path`, `pxt.version`, `pxt.num_rows`, `pxt.num_computed_values`, and `pxt.num_excs`. The UDF span, from `UdfCallAttrs`, carries:

 

 - `pxt.udf_path`, the UDF's fully qualified name, so two functions with the same short name stay distinct

 - `pxt.column`, the named column this call materializes, when it has one

 - `pxt.batch_size`, on a batched call

 - `pxt.resource_pool` and `pxt.retries`, on a call that went through a resource-pool scheduler. `pxt.retries` is how many retries this attempt has already used, so the first try is 0

 

 
## Why a computed column is hard to trace

 
A computed column is not a function you call on the request thread. On insert, an expression-evaluation node hands cells to asyncio tasks. A call that names a resource pool waits in a scheduler: the pool that tracks provider usage retries up to 10 times, and the fixed request-rate scheduler retries up to 3. Media downloads and saves run on a thread pool. A video view's `frame_iterator` yields one row per frame, and each of those rows can call a model of its own.

 
A patch on `openai` that only sees the main thread misses the worker. A trace id you set around `insert()` does not follow a task unless something restores the context on that task. Pixeltable's telemetry does. Worker threads take a `capture_context` snapshot and `restore_context` it before the fetch or the save. Async tasks copy a context var at creation, so concurrent UDF calls keep their own ambient span. That is the mechanism that puts the generation under the UDF span instead of beside it.

 
## Turn it on

 
The bridge, `opentelemetry-instrumentation-pixeltable`, is opt-in. Pixeltable itself never imports OpenTelemetry. The extra declared in Pixeltable 0.7.19 is:

 
```bash
pip install 'pixeltable[otel]' openai openinference-instrumentation-openai
```

 
`pixeltable[otel]` pulls `opentelemetry-instrumentation-pixeltable==0.1.0`, the OpenTelemetry SDK, and the HTTP and gRPC exporters. `openai` is the client `chat_completions` calls; neither Pixeltable nor the instrumentor installs it. On 2026-10-21 the instrumentation package was not on PyPI yet, so the extra did not resolve. Until it is published, install the bridge from the same tag:

 
```bash
pip install pixeltable==0.7.19 openai openinference-instrumentation-openai \
 opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-exporter-otlp-proto-grpc
pip install "opentelemetry-instrumentation-pixeltable @ git+https://github.com/pixeltable/pixeltable@v0.7.19#subdirectory=packages/opentelemetry-instrumentation-pixeltable"
```

 
Call `pxt_otel.init()` once per process, before the first table operation. It reads the endpoint from the argument, then from `db_exporter_otlp_endpoint` on the `[[pixeltable.database]]` entry in `pixeltable.toml`, then from `OTEL_EXPORTER_OTLP_ENDPOINT`, then from the `[otel]` section of `~/.pixeltable/config.toml`. With no endpoint and no provider your application already configured, instrumentation stays inert and exports nothing. A second `init()` raises, so a kernel that already called it with no endpoint has to be restarted. The headers come from `OTEL_EXPORTER_OTLP_HEADERS` the same way. The default protocol is `http/protobuf`. For HTTP, `init()` appends `/v1/traces` unless the endpoint already ends with that path.

 
```python
import opentelemetry.instrumentation.pixeltable as pxt_otel
from openinference.instrumentation.openai import OpenAIInstrumentor

# endpoint and headers come from OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS
pxt_otel.init(span_level='debug', metrics=False)
OpenAIInstrumentor().instrument()

import pixeltable as pxt
from pixeltable.functions import openai

TableModel = pxt.model_base()

class Articles(TableModel, name='articles'):
 body: pxt.String
 summary = openai.chat_completions(
 messages=[{'role': 'user', 'content': 'Summarize in one sentence: ' + body}],
 model='gpt-4o-mini',
 ).choices[0].message.content

TableModel.create_all()
Articles.insert([{'body': 'Pixeltable is declarative data infrastructure for multimodal AI.'}])
```

 
`metrics=False` because the three tools below ingest traces. Leave logs alone: log export is off unless you pass `logs=True`. An application file applies the same class with `pxt schema update` instead of `TableModel.create_all()`. Creating the table and inserting rows are separate traces. The generation is on the insert, and on an update, and when a new computed column backfills rows that already exist. A later `collect()` is a `pixeltable.collect` span and no new model call, because the computed value is stored.

 
A script that exits should flush. `init()` uses a `BatchSpanProcessor`, which ships on a timer while the process stays up:

 
```python
from opentelemetry import trace
trace.get_tracer_provider().force_flush()
```

 
## The same trace in Langfuse, Phoenix, and MLflow

 
Nothing in the bridge is specific to a vendor. The endpoint and one header are the whole difference. Values below are the paths in each product's docs as of 2026-10-21. Pass the base URL to `init()`; it appends `/v1/traces` for HTTP.

 
| Backend | Endpoint for init() | Auth header |
| --- | --- | --- |
| Langfuse | https://cloud.langfuse.com/api/public/otel (EU) or https://us.cloud.langfuse.com/api/public/otel (US). Traces are posted to /api/public/otel/v1/traces. HTTP only. | Basic auth, Authorization=Basic plus the base64 of public_key:secret_key. Keys are per project and per region. |
| Arize Phoenix | Self-hosted HTTP: http://localhost:6006. The collector is /v1/traces on port 6006, the UI port, not the generic OTLP port 4318. gRPC: http://localhost:4317 with protocol='grpc' and no path. The http:// scheme selects plaintext; without a scheme the gRPC exporter uses TLS. | Authorization=Bearer <PHOENIX_API_KEY>. Optional on a local server, required when that server has auth on. |
| MLflow | http://localhost:5000. The server exposes /v1/traces. MLflow 3.6.0 or newer, a SQL backend (SQLite, PostgreSQL, MySQL, or MSSQL), HTTP only. | x-mlflow-experiment-id=<experiment id>. When workspaces are on, add X-MLFLOW-WORKSPACE=<name>. |

 
Sources: [Langfuse OpenTelemetry](https://langfuse.com/docs/opentelemetry), [Phoenix exporter](https://arize.com/docs/phoenix/tracing/concepts-tracing/otel-openinference/exporter) and the [Phoenix endpoint FAQ](https://arize.com/docs/phoenix/resources/frequently-asked-questions/what-is-my-phoenix-endpoint), and [MLflow OTLP ingest](https://mlflow.org/docs/latest/genai/tracing/opentelemetry/ingest/). A remote Phoenix host is the hostname on its Settings page, with the same `/v1/traces` path. Pixeltable's own walkthrough of the bridge uses Grafana the same way: [Observability](https://docs.pixeltable.com/platform/observability).

 
For Langfuse, build the header in the process and run this before `pxt_otel.init()`. Do not write it into `config.toml`; it contains the secret key. A US project key on the EU host returns 401. `LANGFUSE_HOST` still works and is deprecated; use `LANGFUSE_BASE_URL`.

 
```python
import base64, os

base_url = os.environ.get('LANGFUSE_BASE_URL', 'https://cloud.langfuse.com').rstrip('/')
token = base64.b64encode(
 f"{os.environ['LANGFUSE_PUBLIC_KEY']}:{os.environ['LANGFUSE_SECRET_KEY']}".encode()
).decode()
os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = f'{base_url}/api/public/otel'
os.environ['OTEL_EXPORTER_OTLP_HEADERS'] = f'Authorization=Basic {token},x-langfuse-ingestion-version=4'
```

 
`x-langfuse-ingestion-version=4` makes a direct OTLP export show up in real time. Without it, Langfuse says ingestion can lag by about 10 minutes. The screenshot below was recorded through the Langfuse SDK path in Pitfalls, with the insert wrapped in a Langfuse span named `ingest-docs`.

 
 [Image: Langfuse trace graph for a Pixeltable insert wrapped in an ingest-docs span: pixeltable.insert, its catalog, store, row, and UDF spans, and a ChatCompletion generation. The attributes panel for pixeltable.plan.create shows version 0.7.19, scope.name pixeltable, and pxt.nodes.]
 *One Pixeltable operation in Langfuse: the span tree, and `pxt.*` attributes on the selected span. Phoenix and MLflow take the same OTLP payload; this post has no screenshots of them.*
 

 
## gen_ai and OpenInference, in plain terms

 
Two vocabularies describe a model call, and they are not the same attributes.

 
**gen_ai** is the OpenTelemetry GenAI semantic convention. A model-call span carries `gen_ai.request.model`, `gen_ai.usage.*`, and the prompt and completion under `gen_ai.input.messages` and `gen_ai.output.messages`. Older instrumentations write the deprecated `gen_ai.prompt` and `gen_ai.completion`. The Anthropic Python SDK 2.0.0 we tested emits these spans itself, as `anthropic.messages.create`, with no extra instrumentor. It records message content only when its traces `content_mode` is `content`; the default is `metadata_only`. Langfuse showed those spans as generations, with cost.

 
**OpenInference** is Arize's vocabulary for LLM apps. A model call's span kind (`openinference.span.kind`) is `LLM`. Tokens are `llm.token_count.*`. The prompt and completion are `input.value` and `output.value`. `openinference-instrumentation-openai` writes that, and it is what we tested under `pixeltable.udf.chat_completions`.

 
Who reads which:

 

 - **Langfuse** maps both into one generation. Its OTEL attribute table accepts `gen_ai.*` attributes, including `gen_ai.usage.*`, OpenInference `input.value` and `llm.token_count.*`, and `mlflow.spanInputs`. Pixeltable's `pxt.*` spans still appear in the tree; they are ordinary OTLP spans. The Langfuse SDK's default filter would have dropped them. See the pitfalls below.

 - **Phoenix** is built around OpenInference, so the generation view is that child span. The Pixeltable tree arrives with it, because the collector accepts any OTLP trace.

 - **MLflow** stores the OTLP spans you send. Its own SDK speaks `mlflow.*` and can emit `gen_ai.*` on the way out when `MLFLOW_ENABLE_OTEL_GENAI_SEMCONV` is set. For a Pixeltable trace, read the span tree in MLflow. The GenAI columns fill in when the child span speaks `gen_ai.*`, which the Anthropic SDK did in our test. OpenInference attributes stay on the span as attributes.

 

 
Do not attach two instrumentors to the same provider. In our Anthropic test, the SDK's native spans plus `openinference-instrumentation-anthropic` produced a second generation for every call and double-counted cost. OpenAI needs the OpenInference instrumentor for a generation: Pixeltable records token counts as an event on the UDF span, and the backends do not show that event as a generation. Other OpenInference or OpenLLMetry instrumentors that attach to the global provider should nest the same way. We tested OpenAI and Anthropic.

 
## Files and frames

 
`span_level` is a threshold: `info`, `debug`, or `trace`. The docstring on `init()` matches the call sites.

 
| Level | What you get |
| --- | --- |
| info (default) | The operation span, plus catalog, store, and SQL work: pixeltable.catalog.begin_xact, pixeltable.sa.insert_rows, pixeltable.store.insert_rows, pixeltable.store.write_column, pixeltable.view_load, metadata writes. |
| debug | Adds pixeltable.row, pixeltable.udf.<name>, per-file spans, and the finer catalog and store spans such as pixeltable.plan.create and pixeltable.store.build_rows. |
| trace | Adds one span per iterator step, named pixeltable.iter.<iterator>. A frame_iterator step is one decoded frame. |

 
Per-file spans, all at debug:

 

 - `pixeltable.media.fetch`, one per downloaded file, and only on a cache miss. `pxt.url` is set at the start with userinfo, query, and fragment stripped, because a presigned URL carries credentials. `pxt.bytes` is attached when the download succeeds.

 - `pixeltable.media.encode`, one per generated file written to a temp file.

 - `pixeltable.media.save`, one per persisted file, with `pxt.bytes`, and `pxt.destination` when the column uses a configured store. It is left out for the default store.

 

 
There is a cap, and it is narrow. `ExprEvalNode.MAX_ROW_SPANS` is 100. Each expression-evaluation node opens at most 100 `pixeltable.row` spans per operation, and only for rows whose `parent_row` is unset. Rows past that still run. A UDF span that nests only under the row span is skipped with it. A call that went through a resource-pool scheduler still emits its `pixeltable.udf.*` span at debug, beside the row span, with `pxt.resource_pool` and `pxt.retries`. Iterator spans have no cap in the source. `span_level='trace'` on a long video is one span per frame. Use `debug` unless you are inspecting the iterator itself. `info` drops the row and UDF spans, so on an insert the generation's parent is `pixeltable.store.insert_rows`.

 
```python
from pixeltable.functions.video import frame_iterator

class Videos(TableModel, name='videos'):
 video: pxt.Video

class Frames(
 TableModel,
 name='frames',
 base=Videos,
 iterator=frame_iterator(video=Videos.video, fps=1),
):
 thumb = frame.resize((64, 64))
```

 
At debug, each sampled row has a `pixeltable.row` span and a `pixeltable.udf.resize` span, and each file that had to be fetched or saved has a media span. At trace, each decoded frame adds `pixeltable.iter.frame_iterator` around that step. The first 100 rows of the evaluation node get the row span; the iterator spans keep going for every frame.

 
## Lineage stays. The trace is optional.

 
[Your Agent Needs a Data Plane, Not a Dashboard](/blog/agent-data-plane-not-dashboard) makes a specific objection: almost every route to a pretty timeline asks you to ship traces, prompts, and user payloads to a vendor. That objection is about the trace, and it is right. It does not describe the catalog. Pixeltable does not turn telemetry on by itself. The bridge is a separate install, and `pxt_otel.init()` exports nothing until you configure an endpoint or attach it to a provider you set up. Rows, computed values, and column lineage stay under `PIXELTABLE_HOME` whether or not a span leaves the process. You choose the receiver: Langfuse, a Phoenix you run, MLflow on your machine, or a collector that redacts before anything crosses the network. The session log and the dependency graph remain the system of record. The span tree answers the question lineage does not: which attempt, which file, which retry, how long.

 
That split is also the observability layer in [The Invisible 80%](/blog/invisible-infrastructure-production-ai-agents). Lineage, version history, and reproducibility come from the computed columns. Trace ids, input and output capture, and a latency breakdown are the export you turn on here, with a retention policy you set on the collector, not a second copy of the database.

 
## Pitfalls

 

 - **The Langfuse SDK drops Pixeltable spans by default.** This applies when the Langfuse client is your tracer provider. Since SDK v4, `should_export_span` keeps Langfuse spans, `gen_ai.*` spans, and known LLM-instrumentor spans. Pixeltable's scope name is `pixeltable`, so its spans are filtered out. Create the client first, widen the filter, and attach with `PixeltableInstrumentor`. `pxt_otel.init()` is not needed here: it would reuse the client's provider and add no exporter. Flush with `langfuse.flush()`.

 

 
```python
from langfuse import Langfuse
from langfuse.span_filter import is_default_export_span

def export_span(span):
 scope = span.instrumentation_scope
 return is_default_export_span(span) or (scope is not None and scope.name == 'pixeltable')

langfuse = Langfuse(should_export_span=export_span)

from opentelemetry.instrumentation.pixeltable import PixeltableInstrumentor
PixeltableInstrumentor().instrument(span_level='debug')
```

 

 - **Noise.** With the bridge on, `create_dir`, `create_table`, `drop_dir`, and `collect` are traces too. `span_level='info'` keeps the operation and the INFO work spans and drops the per-row detail.

 

 
### What leaves the process

 
Prompts and completions travel on the LLM child span, to whatever endpoint you set. Pixeltable's log export is off by default. Leave it off. Log records can carry API keys in error messages, signed URLs, and connection strings. If you do enable `logs=True`, or if you want prompts redacted before they reach a vendor, do that in an OpenTelemetry Collector in front of the backend. The fetch span already strips credentials from `pxt.url`. The header that authenticates the exporter contains a secret; keep it in the environment, or on a hosted database run `pxt secret set pxt://<org>:<db> OTEL_EXPORTER_OTLP_HEADERS=...` and then `pxt service restart` (see [Configuration](https://docs.pixeltable.com/platform/configuration)), not in a file you commit.

 
## See also

 

 - [Observability](https://docs.pixeltable.com/platform/observability) in the Pixeltable docs, including the span tree and the metric names

 - [What is a declarative pipeline?](/learn/what-is-a-declarative-pipeline)

 - [The Invisible 80%](/blog/invisible-infrastructure-production-ai-agents), where lineage is the observability you get from the schema

 - [Your Agent Needs a Data Plane, Not a Dashboard](/blog/agent-data-plane-not-dashboard)

 - [Langfuse OpenTelemetry](https://langfuse.com/docs/opentelemetry), [Phoenix OTLP](https://arize.com/docs/phoenix/tracing/concepts-tracing/otel-openinference/exporter), [MLflow OTLP ingest](https://mlflow.org/docs/latest/genai/tracing/opentelemetry/ingest/)
## Frequently asked questions

### How do you trace an OpenTelemetry data pipeline in Pixeltable?

Install the OpenTelemetry bridge and call pxt_otel.init() once, before the first table operation. Each operation, such as insert or collect, is one trace. An insert or update carries pxt.table and pxt.num_rows; a query such as collect carries pxt.tables and pxt.rows. At span_level debug the trace adds per-row, per-UDF, and per-file spans with pxt.column, pxt.udf_path, and pxt.retries. Configure an OTLP endpoint for Langfuse, Arize Phoenix, MLflow, or any collector. Export is off until you do.

### How do you trace a multimodal pipeline, including video frames?

span_level debug adds one span per downloaded or saved file. span_level trace adds one span per iterator step, named pixeltable.iter.<iterator>, such as one span per decoded video frame. Those iterator spans are not capped. Per-row spans are capped at 100 per expression-evaluation node.

### Does Pixeltable send prompts to Langfuse, Phoenix, or MLflow by default?

No. Pixeltable itself never imports OpenTelemetry. Spans leave only after you install the bridge and either give pxt_otel.init() an endpoint (the endpoint argument, db_exporter_otlp_endpoint on the database entry, OTEL_EXPORTER_OTLP_ENDPOINT, or the [otel] section of config.toml) or attach it to a TracerProvider your application configured. Rows, computed values, and column lineage stay in the catalog. Prompts and completions leave only on the child spans of an LLM instrumentor you enable, and only to the endpoint you configure.