A hosted Pixeltable database can now send its logs, metrics and traces to the observability tool your team already uses: Grafana Cloud, Datadog, Pydantic Logfire, Honeycomb, New Relic, Axiom, or any endpoint that speaks OTLP. You add the backend on the database's Settings page, or with one pxt secret set, and restart. CloudWatch keeps receiving the same data, so pxt db logs and pxt db spans keep working.
This is the command the Settings dialog prints for a Grafana Cloud stack, before you type the credentials:
The rest of this post covers what the pods send, what each provider keeps, how traces are sampled, and when you need none of this. If your app runs on your own machine, the OpenTelemetry tracing post is the one you want: it covers pxt_otel.init(), the span tree, and Langfuse, Phoenix, MLflow and Opik.
Every pod already runs a collector#
Each database and service pod on Pixeltable Cloud runs an OpenTelemetry Collector next to the application. Pixeltable in the pod sends its logs, metrics and traces over OTLP to that collector on localhost, and the collector forwards them to CloudWatch, where pxt db logs and pxt db spans read, and to every backend you configure. What is new is the Settings dialog, which turns a provider's credentials into those backends.
The collector adds the same four resource attributes to every log record, metric and span: pxt.org, pxt.db, k8s.namespace.name and k8s.pod.name. It sets them itself, so code in a pod cannot claim another database's name, and you can filter by database in Grafana or Datadog without adding anything to your app. It also drops the log line of every health probe, which would otherwise bury the rest of the log.
The traces follow the work. Each table operation is one trace, and an insert carries pxt.table and pxt.num_rows. An insert runs the computed columns on its rows inside the database, so the insert's duration includes the model calls those columns make. A request to a service route starts its trace when the gateway receives it, so the trace also covers the time before your route code runs. In a stack where a queue worker calls the model, the same request is spread over several services and several traces.
Add a backend from Settings or the CLI#
On the database's Settings page, the Telemetry backends section lists the backends that apply to this database: the organization's and its own. Click Add:
- Pick the provider. The dialog shows what that provider drops, if anything, and the steps to find its endpoint and credential in the provider's console.
- Paste the endpoint, or pick the region for providers that publish one host per region.
- Paste the credential: Grafana's instance ID and token, Datadog's API key, Honeycomb's ingest key, and so on. The dialog turns them into the header the provider expects.
- Choose the scope: this database only, or every database in the organization.
- Save, then restart.
Each backend is a set of secrets that share one name suffix. For a backend named GRAFANA they are OTEL_EXPORTER_OTLP_ENDPOINT__GRAFANA, OTEL_EXPORTER_OTLP_HEADERS__GRAFANA, OTEL_EXPORTER_OTLP_PROTOCOL__GRAFANA and, optionally, OTEL_TRACES_SAMPLER_ARG__GRAFANA. The dialog prints the pxt secret set command that writes the same values, with each credential as a placeholder. With an instance ID and token filled in, and a sampling ratio of 0.25, it writes these:
The endpoint comes last on purpose. The collector skips a backend with no endpoint, so a save that stops halfway never sends new headers to an old endpoint. Header values are percent-encoded, as the OpenTelemetry specification asks, so a token that contains =, , or % survives.
The pods read their secrets at startup, so a change applies after a restart:
Stopping and starting the database on its Overview page works too.
What each provider keeps#
Every backend gets the same logs, metrics and traces. Two providers drop part of it, and the dialog says so when you pick them.
| Provider | Protocols the dialog offers | What to know |
|---|---|---|
| Grafana Cloud | http/protobuf | The endpoint is your stack's, from its OpenTelemetry tile, and ends in /otlp. |
| Datadog | http/protobuf, http/json | Traces and logs arrive. Its metrics intake accepts delta metrics only and refuses the cumulative ones Pixeltable sends. |
| Pydantic Logfire | http/protobuf, grpc | The write token goes in the Authorization header as it is, with no Bearer prefix. |
| Honeycomb | http/protobuf, grpc, http/json | Traces and logs land in a dataset named after the service. A Classic account, and metrics on some accounts, need the optional dataset field. |
| New Relic | http/protobuf, grpc | Use an Ingest - License key. |
| Axiom | http/protobuf | Traces and logs arrive. Its metrics endpoint wants a dataset header of its own, which one set of headers per backend cannot send, so metrics are not kept. |
| Any OTLP endpoint | http/protobuf, grpc, http/json | Give the base URL and the headers it authenticates with. |
CloudWatch gets only the metrics Pixeltable itself emits. A backend you add gets every metric the pod sends.
Logs go to every backend too. A log record can carry an API key quoted in an error message, or a signed URL, so add only a backend you would trust with the pod's logs.
Errors and slow traces always go#
Logs and metrics are sent in full. Traces are sampled per backend, after each trace is complete, so the decision can look at the whole trace. This is the sampling policy the control plane writes into the collector config for a GRAFANA backend with OTEL_TRACES_SAMPLER_ARG__GRAFANA=0.25:
A trace that contains an error, or that lasts over one second, is always kept. Of the rest, the backend keeps its ratio. Without its own ratio, a backend uses OTEL_TRACES_SAMPLER_ARG, and without that, 10%. CloudWatch is sampled the same way, and OTEL_TRACES_SAMPLER_ARG__CLOUDWATCH sets its share. On the Settings page, the "Set its sampling ratio" link under the backends list sets it for you.
So a backend can keep every trace while CloudWatch keeps 10%, or the reverse. If you debug a slow route in Honeycomb this week, set its ratio to 1 and leave the others alone.
Read the spans from the terminal#
You don't need a backend to look at a trace. pxt db spans and pxt service spans read the spans CloudWatch kept:
They read no further back than your plan keeps logs: one day on Community, seven on Pro. A backend you add keeps data as long as that backend's own retention allows.
Organization or database scope#
A backend set at organization scope applies to every database in the organization. A database can set its own settings for a backend of the same name, and each one replaces the organization's setting of that key, one key at a time: a database that sets only a sampling ratio still sends to the organization's endpoint. The dialog says when you are about to override the organization.
One more source can set the default backend, the one whose secrets have no suffix: an endpoint or protocol in the project's [[pixeltable.database]] entry replaces the secret of the same name. That change applies after pxt db update and pxt service update, not a restart.
The full rules are on the configuration page of the docs.
When you don't need this#
If your team has no observability tool yet, CloudWatch and pxt db spans may be enough, and adding a backend is one more account to pay for. If Pixeltable runs on your laptop or your own server, nothing here applies: call pxt_otel.init() and point it at any OTLP endpoint, as the tracing post shows. That post also covers Langfuse, Phoenix, MLflow and Opik, which show a model call as a generation.
See also#
- Tracing multimodal AI pipelines with OpenTelemetry, for the span tree and the local setup.
- Telemetry from a hosted database, in the docs.
- The pxt CLI reference, for
pxt secret set,pxt db restartandpxt db spans. - Pixeltable Cloud, to set up a hosted database. pxt new makes a free trial database with no account.
