Reference architecture

Robotics episode curation reference architecture

Episode curation keeps robot teleop or autonomy runs as rows: a clip, JSON telemetry, a success flag, sampled stills, detections, and an index so a new run retrieves similar successes and failures. FiftyOne and Rerun are the visualizers. Pixeltable is the production table those exports come from.

Updated 2026-10-28 · Matching docs cookbook

A robotics dataset is multimodal whether or not the training code says so: video, joint trajectories, gripper states, language instructions, and a success label. Curation is finding the runs you will keep, the stills that show a failure, and the nearest successful episode to a new attempt.

This page is the production table. FiftyOne is the dataset App for visual review (Pixeltable vs Voxel51). Rerun and Foxglove are streaming visualizers. LeRobot is a dataset layout for policy training. Keep those tools; export into them. Do not pretend Pixeltable is a robot runtime.

Episode curation: a run becomes stills, detections, and similar episodes

Role of each component

Episode row: robot_id, task, success, a Video column, and Json telemetry. The clip and the log stay on one row so a still can join back to joints.
Frame view: frame_iterator at a rate you can afford. fps=1 is a starting point for long teleop. Each still keeps frame_attrs.
Detector: yolox on the still. COCO ids (person=0, car=2) are a starting point. A task-specific detector is a UDF you own; Pixeltable does not ship a robot-part model.
Similarity index: CLIP on the frame so a language instruction or a still retrieves episodes. Optional text index on the task string.
Curation queries: Success versus failure, frames whose labels contain a class, nearest neighbors to a new attempt.
Export: Optional. FiftyOne for visual QA, LeRobot or Parquet for training. The table remains the source of truth.

Honest alternatives per slot

Keep the named engine when it is already the contract. Pixeltable fills the slot when you want that job on the same table as the files.

Named alternatives for Robotics episode curation reference architecture
Slot	Role	Pixeltable	Named alternatives
Visualizer	Watch a live or recorded episode	Not a streaming visualizer. Store the clip; open it in a tool built for that	Rerun. You want a timeline of images, transforms, and logs while the robot runs Foxglove. You already speak ROS / MCAP and want their layout FiftyOne. You want a dataset App for visual QA and embedding brains
Dataset format	What training code reads	The episode table; export when a trainer needs a layout	LeRobot dataset. Hugging Face policy training expects that layout MCAP / rosbag. The fleet already writes those files; ingest them into rows
Object store	Bytes for long videos	A Video column, or insert S3 URLs	Amazon S3. The fleet already dumps bags and MP4s there MinIO. Air-gapped lab storage with an S3 API
Detector / embedder	Boxes and similar stills	yolox on frames; CLIP EmbeddingIndex on the frame column	A custom detector. COCO classes are the wrong vocabulary; wrap it in a @pxt.udf Qdrant. You already embed frames in a training job and only need ANN

Where Pixeltable fits

Pixeltable is the episode table in this architecture: Video plus Json telemetry, frame_iterator, yolox, CLIP, and queries for success and similarity. Keep Rerun or Foxglove for the live timeline. Keep FiftyOne for interactive visual QA (pxt.io.export_images_as_fo_dataset is the documented export). Keep LeRobot as the training layout. Pixeltable vs Voxel51 is the App versus the production table. Pixeltable Cloud hosts the same app file. The industry page /industries/robotics is a worked warehouse-pick example on this shape.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Episodes(TableModel, name='episodes'):
    robot_id: pxt.String
    task: pxt.String
    success: pxt.Bool
    video: pxt.Video
    telemetry: pxt.Json

class Frames(
    TableModel,
    name='frames',
    base=Episodes,
    iterator=frame_iterator(video=Episodes.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def similar_successes(question: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.where(Episodes.success == True)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(
            robot_id=Episodes.robot_id,
            task=Episodes.task,
            frame_attrs=Frames.frame_attrs,
            score=sim,
        )
    )

# detections["labels"] is COCO class ids. 0 is person.
# frame_attrs["time"] is the presentation timestamp in seconds.
# pxt init
# pxt schema update app.py episodes -f

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. Get started covers the CLI.

Questions

Is Pixeltable a robot runtime?

No. It does not replace ROS, a real-time controller, or a policy trainer. It stores episodes, runs models on stills, and keeps an index so you can curate and retrieve runs.

COCO classes are wrong for my gripper. What then?

Wrap your detector in a @pxt.udf and assign it on the frame view the same way yolox is assigned. yolox is the documented starting detector, not a claim that COCO is a robotics ontology.

How do I review episodes visually?

FiftyOne is the dataset App. Rerun and Foxglove are streaming visualizers. Export stills or the clip; do not expect Pixeltable to be that UI. Pixeltable vs Voxel51 is that split.