How do I build a video processing pipeline in Python?

Before you start

About 25 min.

• Python 3.11+
• pip install -U 'pixeltable[serve]'
• pxt init in the project root
• ffmpeg
• openai-whisper
• pixeltable-yolox

1. Ingest clips as video, not as paths in a spreadsheet

A typed Video column is what later frame extraction and audio extraction read. Titles and categories stay on the same row so you can filter after analysis.

import pixeltable as pxt
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    category: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)

2. Extract frames and run object detection on each still

A view emits one image row per sampled frame. YOLOX writes boxes, scores, and labels on that still. frame_attrs keeps the time so a detection is a moment in the clip.

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)

# frame_attrs["index"] is the stream index
# frame_attrs["time"] is the presentation timestamp in seconds (pts * time_base)

3. Split into clips when the unit is a scene, not a still

video_splitter emits time ranges (segment_start, segment_end, video_segment) when you want clip-level analysis instead of one row per frame. Use it beside the frame view, not instead of timestamps on stills.

from pixeltable.functions.video import video_splitter

class Segments(
    TableModel,
    name='segments',
    base=Videos,
    iterator=video_splitter(video=Videos.video, duration=10.0, min_segment_duration=1.0),
):
    pass

Glue stack versus one video processing pipeline

Where Pixeltable fits

Pixeltable is the unified multimodal backend for a video processing pipeline: Video, frame_iterator, yolox, extract_audio, and Whisper as columns. Video analysis AI is those expressions, cached per row. Compare Pixeltable vs Twelve Labs when a hosted video model is the product, and Pixeltable vs Voxel51 when a labeling UI is the product.

import pixeltable as pxt
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)

Questions

Is this an Airflow DAG with ffmpeg tasks?

No. The transforms are columns. Insert a clip and that row runs. Unchanged clips stay cached. Airflow can still sit outside when you already run a fleet of unrelated jobs; see Pixeltable vs Airflow.

Where does video analysis AI run?

On the stills and the soundtrack: YOLOX on each frame, Whisper on extracted audio. The output stays on those rows with frame_attrs so a detection is a timestamp.

Can I use keyframes instead of 1 FPS?

Yes. frame_iterator(..., keyframes_only=True) emits I-frames. Set only one of fps, num_frames, or keyframes_only. Detection then runs on fewer stills.