Industry

Pixeltable for robotics

Robotics teams curate teleop and autonomy runs: a clip, joint logs, a success flag, stills, and an index so a new attempt retrieves similar successes and failures. Pixeltable is the episode table. Rerun, Foxglove, FiftyOne, and LeRobot stay in their slots.

Updated 2026-10-28 · Matching docs cookbook

A warehouse pick episode is not one tensor. It is a wrist camera MP4, a scene camera, a JSON stream of joint targets and gripper width, a language instruction, and a boolean for whether the SKU landed in the tote. Training code wants a layout such as LeRobot. Reviewers want a timeline such as Rerun. Someone still has to answer: show me successful picks of SKU-441 that look like this miss.

The usual split is bags on S3, a spreadsheet of paths, ffmpeg dumping JPEGs, a notebook that runs a detector, and a vector database that does not know the success flag. A new bag means rerunning the notebook. Pixeltable is the table that keeps the clip, the log, the stills, and the index on one schema. It is not a robot runtime and not a visualizer.

Warehouse pick episode pipeline

Insert one row per episode: robot_id, sku, success, the wrist video, JSON telemetry.
Sample stills with frame_iterator at 1 fps. Each still keeps frame_attrs.
Run yolox on each still for a first-pass person/object screen. A SKU detector is a UDF you own.
Index stills with CLIP so a language instruction or a miss still retrieves similar successes.
Query successful episodes nearest to a failure. Export stills to FiftyOne when a person must label.

Worked example: retrieve successful picks similar to a miss

This sketch is the robotics episode-curation architecture applied to a pick cell. COCO class ids are a screen, not a SKU ontology. Wrap a custom detector in @pxt.udf when you have one. The architecture page names Rerun, Foxglove, FiftyOne, LeRobot, S3, and Qdrant as slots.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Picks(TableModel, name='picks'):
    robot_id: pxt.String
    sku: pxt.String
    success: pxt.Bool
    video: pxt.Video
    telemetry: pxt.Json

class Frames(
    TableModel,
    name='frames',
    base=Picks,
    iterator=frame_iterator(video=Picks.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def similar_successes(question: str, sku: str, limit: int = 5):
    sim = Frames.frame.similarity(string=question)
    return (
        Frames.where(Picks.success == True)
        .where(Picks.sku == sku)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(
            robot_id=Picks.robot_id,
            sku=Picks.sku,
            frame_attrs=Frames.frame_attrs,
            score=sim,
        )
    )

# frame_attrs["time"] is the presentation timestamp in seconds.
# pxt init
# pxt schema update app.py picks -f

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. Get started covers the CLI.

Honest alternatives per slot

Named alternatives for Pixeltable for robotics
Slot	Role	Pixeltable	Named alternatives
Visualizer	Watch the episode	Stores the clip; not a live timeline	Rerun. You want images, transforms, and logs on one timeline Foxglove. The fleet already speaks ROS / MCAP FiftyOne. Interactive visual QA of stills and embeddings
Training layout	What the policy trainer reads	The episode table; export when the trainer needs a layout	LeRobot dataset. Hugging Face policy training expects that format MCAP / rosbag. The cell already writes bags; ingest them into rows
Object store	Long MP4s and bags	Video column, or insert S3 URLs	Amazon S3. The cell already dumps there MinIO. Air-gapped lab with an S3 API

Where Pixeltable fits

Pixeltable is the episode table: Video, Json telemetry, frame_iterator, yolox, CLIP, and a query that returns similar successful picks. Keep Rerun or Foxglove for the live timeline. Keep FiftyOne for labeling QA. Keep LeRobot for training. The reference architecture is /reference-architecture/robotics-episode-curation. Pixeltable vs Voxel51 is the App versus the production table.

Questions

Does this replace ROS?

No. ROS, a real-time controller, and a policy trainer stay. Pixeltable stores completed episodes and runs models on stills so you can curate and retrieve them.

Can I keep bags on S3?

Yes. Insert the object URL into a Video column (or a string you astype). The docs cookbook is Load media from S3. Pixeltable caches bytes when a column needs to read the file.