---
title: "Pixeltable for robotics"
description: "Store robot episodes as video plus telemetry, sample stills, detect, and retrieve similar successes. A worked warehouse-pick pipeline with TableModel code."
url: "https://www.pixeltable.com/industries/robotics"
updated: "2026-10-28"
doc: "https://docs.pixeltable.com/howto/cookbooks/images/yolox-object-detection"
---

# Pixeltable for robotics

Robotics teams curate teleop and autonomy runs: a clip, joint logs, a success flag, stills, and an index so a new attempt retrieves similar successes and failures. Pixeltable is the episode table. Rerun, Foxglove, FiftyOne, and LeRobot stay in their slots.

Updated: 2026-10-28
Docs: https://docs.pixeltable.com/howto/cookbooks/images/yolox-object-detection

A warehouse pick episode is not one tensor. It is a wrist camera MP4, a scene camera, a JSON stream of joint targets and gripper width, a language instruction, and a boolean for whether the SKU landed in the tote. Training code wants a layout such as LeRobot. Reviewers want a timeline such as Rerun. Someone still has to answer: show me successful picks of SKU-441 that look like this miss.
The usual split is bags on S3, a spreadsheet of paths, ffmpeg dumping JPEGs, a notebook that runs a detector, and a vector database that does not know the success flag. A new bag means rerunning the notebook. Pixeltable is the table that keeps the clip, the log, the stills, and the index on one schema. It is not a robot runtime and not a visualizer.

## Warehouse pick episode pipeline {#pipeline}

1. Insert one row per episode: robot_id, sku, success, the wrist video, JSON telemetry.
2. Sample stills with frame_iterator at 1 fps. Each still keeps frame_attrs.
3. Run yolox on each still for a first-pass person/object screen. A SKU detector is a UDF you own.
4. Index stills with CLIP so a language instruction or a miss still retrieves similar successes.
5. Query successful episodes nearest to a failure. Export stills to FiftyOne when a person must label.

## Worked example: retrieve successful picks similar to a miss {#worked-example}

This sketch is the robotics episode-curation architecture applied to a pick cell. COCO class ids are a screen, not a SKU ontology. Wrap a custom detector in @pxt.udf when you have one. The architecture page names Rerun, Foxglove, FiftyOne, LeRobot, S3, and Qdrant as slots.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Picks(TableModel, name='picks'):
    robot_id: pxt.String
    sku: pxt.String
    success: pxt.Bool
    video: pxt.Video
    telemetry: pxt.Json

class Frames(
    TableModel,
    name='frames',
    base=Picks,
    iterator=frame_iterator(video=Picks.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def similar_successes(question: str, sku: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.where(Picks.success == True)
        .where(Picks.sku == sku)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(
            robot_id=Picks.robot_id,
            sku=Picks.sku,
            frame_attrs=Frames.frame_attrs,
            score=sim,
        )
    )

# frame_attrs["time"] is the presentation timestamp in seconds.
# pxt init
# pxt schema update app.py picks -f
```

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. [Get started](https://www.pixeltable.com/get-started) covers the CLI.

## Honest alternatives per slot {#alternatives}

| Slot | Role | Pixeltable | Named alternatives |
| --- | --- | --- | --- |
| Visualizer | Watch the episode | Stores the clip; not a live timeline | Rerun: You want images, transforms, and logs on one timeline; Foxglove: The fleet already speaks ROS / MCAP; FiftyOne: Interactive visual QA of stills and embeddings |
| Training layout | What the policy trainer reads | The episode table; export when the trainer needs a layout | LeRobot dataset: Hugging Face policy training expects that format; MCAP / rosbag: The cell already writes bags; ingest them into rows |
| Object store | Long MP4s and bags | Video column, or insert S3 URLs | Amazon S3: The cell already dumps there; MinIO: Air-gapped lab with an S3 API |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the episode table: Video, Json telemetry, frame_iterator, yolox, CLIP, and a query that returns similar successful picks. Keep Rerun or Foxglove for the live timeline. Keep FiftyOne for labeling QA. Keep LeRobot for training. The reference architecture is /reference-architecture/robotics-episode-curation. Pixeltable vs Voxel51 is the App versus the production table.

## Questions {#questions}

### Does this replace ROS? {#faq-1}

No. ROS, a real-time controller, and a policy trainer stay. Pixeltable stores completed episodes and runs models on stills so you can curate and retrieve them.

### Can I keep bags on S3? {#faq-2}

Yes. Insert the object URL into a Video column (or a string you astype). The docs cookbook is Load media from S3. Pixeltable caches bytes when a column needs to read the file.

## Related

- [Robotics episode curation architecture](https://www.pixeltable.com/reference-architecture/robotics-episode-curation)
- [Video processing pipeline how-to](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline)
- [Object detection in videos notebook](https://www.pixeltable.com/notebooks/object-detection-in-videos)
- [Working with FiftyOne notebook](https://www.pixeltable.com/notebooks/working-with-fiftyone)
- [Pixeltable vs Voxel51](https://www.pixeltable.com/compare/pixeltable-vs-voxel51)
- [Examples hub](https://www.pixeltable.com/examples)
- [YOLOX cookbook](https://docs.pixeltable.com/howto/cookbooks/images/yolox-object-detection)

Architectures: [reference architectures](https://www.pixeltable.com/reference-architecture). Task-grouped examples: [examples](https://www.pixeltable.com/examples).