---
title: "Robotics episode curation reference architecture"
description: "A robotics episode-curation architecture: video plus telemetry, sampled frames, YOLOX, CLIP retrieval. Named alternatives including FiftyOne and Rerun."
url: "https://www.pixeltable.com/reference-architecture/robotics-episode-curation"
updated: "2026-10-28"
doc: "https://docs.pixeltable.com/howto/cookbooks/images/yolox-object-detection"
---

# Robotics episode curation reference architecture

Episode curation keeps robot teleop or autonomy runs as rows: a clip, JSON telemetry, a success flag, sampled stills, detections, and an index so a new run retrieves similar successes and failures. FiftyOne and Rerun are the visualizers. Pixeltable is the production table those exports come from.

Updated: 2026-10-28
Docs: https://docs.pixeltable.com/howto/cookbooks/images/yolox-object-detection

A robotics dataset is multimodal whether or not the training code says so: video, joint trajectories, gripper states, language instructions, and a success label. Curation is finding the runs you will keep, the stills that show a failure, and the nearest successful episode to a new attempt.
This page is the production table. FiftyOne is the dataset App for visual review (Pixeltable vs Voxel51). Rerun and Foxglove are streaming visualizers. LeRobot is a dataset layout for policy training. Keep those tools; export into them. Do not pretend Pixeltable is a robot runtime.

## Episode curation: a run becomes stills, detections, and similar episodes {#diagram}

```mermaid
flowchart TB
  cam[Camera] --> video[Video column]
  logs[Joint / action log] --> json[Json telemetry]
  video --> episode[Episodes table]
  json --> episode
  episode --> frames[frame_iterator]
  frames --> yolox[yolox on frame]
  frames --> clipIdx[CLIP EmbeddingIndex]
  yolox --> query[Filter by class or success]
  clipIdx --> similar[Similar episodes]
  query --> export[Optional FiftyOne / LeRobot]
  similar --> export
  frames -.->|visualizer| rerun[Rerun or Foxglove]
```

## Role of each component

- **Episode row.** robot_id, task, success, a Video column, and Json telemetry. The clip and the log stay on one row so a still can join back to joints.
- **Frame view.** frame_iterator at a rate you can afford. fps=1 is a starting point for long teleop. Each still keeps frame_attrs.
- **Detector.** yolox on the still. COCO ids (person=0, car=2) are a starting point. A task-specific detector is a UDF you own; Pixeltable does not ship a robot-part model.
- **Similarity index.** CLIP on the frame so a language instruction or a still retrieves episodes. Optional text index on the task string.
- **Curation queries.** Success versus failure, frames whose labels contain a class, nearest neighbors to a new attempt.
- **Export.** Optional. FiftyOne for visual QA, LeRobot or Parquet for training. The table remains the source of truth.

## Honest alternatives per slot {#alternatives}

| Slot | Role | Pixeltable | Named alternatives |
| --- | --- | --- | --- |
| Visualizer | Watch a live or recorded episode | Not a streaming visualizer. Store the clip; open it in a tool built for that | Rerun: You want a timeline of images, transforms, and logs while the robot runs; Foxglove: You already speak ROS / MCAP and want their layout; FiftyOne: You want a dataset App for visual QA and embedding brains |
| Dataset format | What training code reads | The episode table; export when a trainer needs a layout | LeRobot dataset: Hugging Face policy training expects that layout; MCAP / rosbag: The fleet already writes those files; ingest them into rows |
| Object store | Bytes for long videos | A Video column, or insert S3 URLs | Amazon S3: The fleet already dumps bags and MP4s there; MinIO: Air-gapped lab storage with an S3 API |
| Detector / embedder | Boxes and similar stills | yolox on frames; CLIP EmbeddingIndex on the frame column | A custom detector: COCO classes are the wrong vocabulary; wrap it in a @pxt.udf; Qdrant: You already embed frames in a training job and only need ANN |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the episode table in this architecture: Video plus Json telemetry, frame_iterator, yolox, CLIP, and queries for success and similarity. Keep Rerun or Foxglove for the live timeline. Keep FiftyOne for interactive visual QA (pxt.io.export_images_as_fo_dataset is the documented export). Keep LeRobot as the training layout. Pixeltable vs Voxel51 is the App versus the production table. Pixeltable Cloud hosts the same app file. The industry page /industries/robotics is a worked warehouse-pick example on this shape.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Episodes(TableModel, name='episodes'):
    robot_id: pxt.String
    task: pxt.String
    success: pxt.Bool
    video: pxt.Video
    telemetry: pxt.Json

class Frames(
    TableModel,
    name='frames',
    base=Episodes,
    iterator=frame_iterator(video=Episodes.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def similar_successes(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.where(Episodes.success == True)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(
            robot_id=Episodes.robot_id,
            task=Episodes.task,
            frame_attrs=Frames.frame_attrs,
            score=sim,
        )
    )

# detections["labels"] is COCO class ids. 0 is person.
# frame_attrs["time"] is the presentation timestamp in seconds.
# pxt init
# pxt schema update app.py episodes -f
```

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. [Get started](https://www.pixeltable.com/get-started) covers the CLI.

## Questions {#questions}

### Is Pixeltable a robot runtime? {#faq-1}

No. It does not replace ROS, a real-time controller, or a policy trainer. It stores episodes, runs models on stills, and keeps an index so you can curate and retrieve runs.

### COCO classes are wrong for my gripper. What then? {#faq-2}

Wrap your detector in a @pxt.udf and assign it on the frame view the same way yolox is assigned. yolox is the documented starting detector, not a claim that COCO is a robotics ontology.

### How do I review episodes visually? {#faq-3}

FiftyOne is the dataset App. Rerun and Foxglove are streaming visualizers. Export stills or the clip; do not expect Pixeltable to be that UI. Pixeltable vs Voxel51 is that split.

## Related

- [Pixeltable for robotics (worked example)](https://www.pixeltable.com/industries/robotics)
- [How do I build a video processing pipeline?](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline)
- [How do I extract keyframes from video?](https://www.pixeltable.com/learn/how-to/extract-keyframes-from-video)
- [Object detection in videos notebook](https://www.pixeltable.com/notebooks/object-detection-in-videos)
- [Working with FiftyOne notebook](https://www.pixeltable.com/notebooks/working-with-fiftyone)
- [Pixeltable vs Voxel51 (FiftyOne)](https://www.pixeltable.com/compare/pixeltable-vs-voxel51)
- [Examples hub](https://www.pixeltable.com/examples)
- [Detect objects with YOLOX (docs)](https://docs.pixeltable.com/howto/cookbooks/images/yolox-object-detection)
- [Extract frames from videos (docs)](https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames)

This is the architecture. Step-by-step Python is on [how-to guides](https://www.pixeltable.com/learn/how-to) and [examples](https://www.pixeltable.com/examples).