--- title: "Send your hosted database's logs, metrics and traces to Grafana, Datadog or any OTLP backend" date: "2026-11-04" dateModified: "2026-11-04" author: "Pierre Brunelle" tags: - OpenTelemetry - Observability - Pixeltable Cloud - Grafana - Datadog description: "Every Pixeltable Cloud database and service pod runs an OpenTelemetry Collector. Add a backend on the database Settings page, or with pxt secret set, and the pods send their logs, metrics and sampled traces there as well as to CloudWatch." url: "https://www.pixeltable.com/blog/pixeltable-cloud-telemetry-backends" --- # Send your hosted database's logs, metrics and traces to Grafana, Datadog or any OTLP backend A hosted Pixeltable database can now send its logs, metrics and traces to the observability tool your team already uses: Grafana Cloud, Datadog, Pydantic Logfire, Honeycomb, New Relic, Axiom, or any endpoint that speaks OTLP. You add the backend on the database's Settings page, or with one `pxt secret set`, and restart. CloudWatch keeps receiving the same data, so `pxt db logs` and `pxt db spans` keep working. This is the command the Settings dialog prints for a Grafana Cloud stack, before you type the credentials: ```bash pxt secret set pxt://myorg:mydb \ 'OTEL_EXPORTER_OTLP_HEADERS__GRAFANA=Authorization=Basic%20' \ OTEL_EXPORTER_OTLP_PROTOCOL__GRAFANA=http/protobuf \ 'OTEL_EXPORTER_OTLP_ENDPOINT__GRAFANA=' ``` The rest of this post covers what the pods send, what each provider keeps, how traces are sampled, and when you need none of this. If your app runs on your own machine, the [OpenTelemetry tracing post](/blog/tracing-multimodal-ai-pipelines-opentelemetry) is the one you want: it covers `pxt_otel.init()`, the span tree, and Langfuse, Phoenix, MLflow and Opik. ## Every pod already runs a collector Each database and service pod on Pixeltable Cloud runs an OpenTelemetry Collector next to the application. Pixeltable in the pod sends its logs, metrics and traces over OTLP to that collector on `localhost`, and the collector forwards them to CloudWatch, where `pxt db logs` and `pxt db spans` read, and to every backend you configure. What is new is the Settings dialog, which turns a provider's credentials into those backends. The collector adds the same four resource attributes to every log record, metric and span: `pxt.org`, `pxt.db`, `k8s.namespace.name` and `k8s.pod.name`. It sets them itself, so code in a pod cannot claim another database's name, and you can filter by database in Grafana or Datadog without adding anything to your app. It also drops the log line of every health probe, which would otherwise bury the rest of the log. The traces follow the work. Each table operation is one trace, and an insert carries `pxt.table` and `pxt.num_rows`. An insert runs the computed columns on its rows inside the database, so the insert's duration includes the model calls those columns make. A request to a service route starts its trace when the gateway receives it, so the trace also covers the time before your route code runs. In a stack where a queue worker calls the model, the same request is spread over several services and several traces. ## Add a backend from Settings or the CLI On the database's Settings page, the Telemetry backends section lists the backends that apply to this database: the organization's and its own. Click Add: - Pick the provider. The dialog shows what that provider drops, if anything, and the steps to find its endpoint and credential in the provider's console. - Paste the endpoint, or pick the region for providers that publish one host per region. - Paste the credential: Grafana's instance ID and token, Datadog's API key, Honeycomb's ingest key, and so on. The dialog turns them into the header the provider expects. - Choose the scope: this database only, or every database in the organization. - Save, then restart. Each backend is a set of secrets that share one name suffix. For a backend named `GRAFANA` they are `OTEL_EXPORTER_OTLP_ENDPOINT__GRAFANA`, `OTEL_EXPORTER_OTLP_HEADERS__GRAFANA`, `OTEL_EXPORTER_OTLP_PROTOCOL__GRAFANA` and, optionally, `OTEL_TRACES_SAMPLER_ARG__GRAFANA`. The dialog prints the `pxt secret set` command that writes the same values, with each credential as a placeholder. With an instance ID and token filled in, and a sampling ratio of 0.25, it writes these: ```bash pxt secret set pxt://myorg:mydb \ OTEL_EXPORTER_OTLP_HEADERS__GRAFANA=Authorization=Basic%20MTIzNDU2OmdsY19leGFtcGxl \ OTEL_EXPORTER_OTLP_PROTOCOL__GRAFANA=http/protobuf \ OTEL_TRACES_SAMPLER_ARG__GRAFANA=0.25 \ OTEL_EXPORTER_OTLP_ENDPOINT__GRAFANA=https://otlp-gateway-prod-us-east-0.grafana.net/otlp ``` The endpoint comes last on purpose. The collector skips a backend with no endpoint, so a save that stops halfway never sends new headers to an old endpoint. Header values are percent-encoded, as the OpenTelemetry specification asks, so a token that contains `=`, `,` or `%` survives. The pods read their secrets at startup, so a change applies after a restart: ```bash pxt db restart pxt://myorg:mydb pxt service restart my-service ``` Stopping and starting the database on its Overview page works too. ## What each provider keeps Every backend gets the same logs, metrics and traces. Two providers drop part of it, and the dialog says so when you pick them. | Provider | Protocols the dialog offers | What to know | | --- | --- | --- | | Grafana Cloud | http/protobuf | The endpoint is your stack's, from its OpenTelemetry tile, and ends in /otlp. | | Datadog | http/protobuf, http/json | Traces and logs arrive. Its metrics intake accepts delta metrics only and refuses the cumulative ones Pixeltable sends. | | Pydantic Logfire | http/protobuf, grpc | The write token goes in the Authorization header as it is, with no Bearer prefix. | | Honeycomb | http/protobuf, grpc, http/json | Traces and logs land in a dataset named after the service. A Classic account, and metrics on some accounts, need the optional dataset field. | | New Relic | http/protobuf, grpc | Use an Ingest - License key. | | Axiom | http/protobuf | Traces and logs arrive. Its metrics endpoint wants a dataset header of its own, which one set of headers per backend cannot send, so metrics are not kept. | | Any OTLP endpoint | http/protobuf, grpc, http/json | Give the base URL and the headers it authenticates with. | CloudWatch gets only the metrics Pixeltable itself emits. A backend you add gets every metric the pod sends. Logs go to every backend too. A log record can carry an API key quoted in an error message, or a signed URL, so add only a backend you would trust with the pod's logs. ## Errors and slow traces always go Logs and metrics are sent in full. Traces are sampled per backend, after each trace is complete, so the decision can look at the whole trace. This is the sampling policy the control plane writes into the collector config for a `GRAFANA` backend with `OTEL_TRACES_SAMPLER_ARG__GRAFANA=0.25`: ```json { "policies": [ { "name": "errors", "type": "status_code", "status_code": { "status_codes": [ "ERROR" ] } }, { "name": "slow", "type": "latency", "latency": { "threshold_ms": 1000 } }, { "name": "sampled", "type": "probabilistic", "probabilistic": { "sampling_percentage": 25.0 } } ] } ``` A trace that contains an error, or that lasts over one second, is always kept. Of the rest, the backend keeps its ratio. Without its own ratio, a backend uses `OTEL_TRACES_SAMPLER_ARG`, and without that, 10%. CloudWatch is sampled the same way, and `OTEL_TRACES_SAMPLER_ARG__CLOUDWATCH` sets its share. On the Settings page, the "Set its sampling ratio" link under the backends list sets it for you. So a backend can keep every trace while CloudWatch keeps 10%, or the reverse. If you debug a slow route in Honeycomb this week, set its ratio to 1 and leave the others alone. ## Read the spans from the terminal You don't need a backend to look at a trace. `pxt db spans` and `pxt service spans` read the spans CloudWatch kept: ```bash pxt db spans --help ``` ```text usage: pxt db spans [-h] [--since SINCE] [--tail LIMIT] [--trace TRACE_ID] [--json] [db_uri] positional arguments: db_uri Database URI: pxt://org:db (default: db_uri from the config) options: -h, --help show this help message and exit --since SINCE how far back to read: 30s, 10m, 1h, 2d (default: 1h) --tail LIMIT the newest N spans in the window, at most 10000 (default: 200) --trace TRACE_ID only the spans of this trace (32 lowercase hex digits) --json Emit JSON output ``` They read no further back than your plan keeps logs: one day on Community, seven on Pro. A backend you add keeps data as long as that backend's own retention allows. ## Organization or database scope A backend set at organization scope applies to every database in the organization. A database can set its own settings for a backend of the same name, and each one replaces the organization's setting of that key, one key at a time: a database that sets only a sampling ratio still sends to the organization's endpoint. The dialog says when you are about to override the organization. One more source can set the default backend, the one whose secrets have no suffix: an endpoint or protocol in the project's `[[pixeltable.database]]` entry replaces the secret of the same name. That change applies after `pxt db update` and `pxt service update`, not a restart. The full rules are on the [configuration page](https://docs.pixeltable.com/platform/configuration#telemetry-from-a-hosted-database) of the docs. ## When you don't need this If your team has no observability tool yet, CloudWatch and `pxt db spans` may be enough, and adding a backend is one more account to pay for. If Pixeltable runs on your laptop or your own server, nothing here applies: call `pxt_otel.init()` and point it at any OTLP endpoint, as the [tracing post](/blog/tracing-multimodal-ai-pipelines-opentelemetry) shows. That post also covers Langfuse, Phoenix, MLflow and Opik, which show a model call as a generation. ## See also - [Tracing multimodal AI pipelines with OpenTelemetry](/blog/tracing-multimodal-ai-pipelines-opentelemetry), for the span tree and the local setup. - [Telemetry from a hosted database](https://docs.pixeltable.com/platform/configuration#telemetry-from-a-hosted-database), in the docs. - [The pxt CLI reference](https://docs.pixeltable.com/platform/cli), for `pxt secret set`, `pxt db restart` and `pxt db spans`. - [Pixeltable Cloud](https://docs.pixeltable.com/cloud), to set up a hosted database. [pxt new](/new) makes a free trial database with no account. ## Frequently asked questions ### How do I send a Pixeltable Cloud database to Grafana Cloud or Datadog? Open the database's Settings page, find Telemetry backends, click Add, pick the provider and paste its endpoint and credential. Or set the same secrets with pxt secret set: OTEL_EXPORTER_OTLP_ENDPOINT__, OTEL_EXPORTER_OTLP_HEADERS__ and OTEL_EXPORTER_OTLP_PROTOCOL__. Then run pxt db restart and pxt service restart, since the pods read their secrets at startup. ### Which traces does the hosted collector send? Every trace with an error, every trace that lasts over one second, and 10% of the rest. OTEL_TRACES_SAMPLER_ARG__ sets the share of the rest for one backend, OTEL_TRACES_SAMPLER_ARG for every backend without its own, and OTEL_TRACES_SAMPLER_ARG__CLOUDWATCH for CloudWatch. Logs and metrics are not sampled. ### Do I still get pxt db logs and pxt db spans after adding a backend? Yes. CloudWatch stays on as the built-in backend, and pxt db logs, pxt db spans and pxt service spans read from it. A backend you add receives the same data in addition.