---
title: "How do I build a video processing pipeline in Python?"
description: "Build a video processing pipeline in Python: ingest clips, extract frames, run detection and transcription, and keep timestamps on every still."
url: "https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline"
updated: "2026-10-14"
doc: "https://docs.pixeltable.com/howto/use-cases/object-detection-in-videos"
---

# How do I build a video processing pipeline in Python?

A video processing pipeline turns clips into frames, detections, transcripts, and searchable moments. Video analysis AI is those models attached to the stills and the soundtrack, not a folder of JPEGs with lost timestamps. The pipeline is incremental: a new file recomputes that row; unchanged clips stay cached.

Updated: 2026-10-14
Part of [How do I ... in Python?](https://www.pixeltable.com/learn/how-to).

## Before you start

About 25 min.

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- pxt init in the project root
- ffmpeg
- openai-whisper
- pixeltable-yolox

## On this page

- [Before you start](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline#before-you-start)
- [Ingest clips as video, not as paths in a spreadsheet](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline#step-1)
- [Extract frames and run object detection on each still](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline#step-2)
- [Split into clips when the unit is a scene, not a still](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline#step-3)
- [Comparison](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/how-to/build-a-video-processing-pipeline#questions)

## 1. Ingest clips as video, not as paths in a spreadsheet {#step-1}

A typed Video column is what later frame extraction and audio extraction read. Titles and categories stay on the same row so you can filter after analysis.

```python
import pixeltable as pxt
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    category: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)
```

## 2. Extract frames and run object detection on each still {#step-2}

A view emits one image row per sampled frame. YOLOX writes boxes, scores, and labels on that still. frame_attrs keeps the time so a detection is a moment in the clip.

```python
class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)

# frame_attrs["index"] is the stream index
# frame_attrs["time"] is the presentation timestamp in seconds (pts * time_base)
```

## 3. Split into clips when the unit is a scene, not a still {#step-3}

video_splitter emits time ranges (segment_start, segment_end, video_segment) when you want clip-level analysis instead of one row per frame. Use it beside the frame view, not instead of timestamps on stills.

```python
from pixeltable.functions.video import video_splitter

class Segments(
    TableModel,
    name='segments',
    base=Videos,
    iterator=video_splitter(video=Videos.video, duration=10.0, min_segment_duration=1.0),
):
    pass
```

## Glue stack versus one video processing pipeline {#comparison}

| Job | Typical glue stack | One schema |
| --- | --- | --- |
| Store the clip | Object storage plus a metadata database | A Video column on the table you own |
| Sample frames | A scheduled ffmpeg job between stores | A frame_iterator view on that clip |
| Detect objects | A vision service you poll and copy out | yolox as a column on each still |
| Transcribe speech | A second job on an extracted wav | extract_audio, then Whisper on the clip |
| After a new file | Rerun the DAG | Recompute the rows that changed |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the unified multimodal backend for a video processing pipeline: Video, frame_iterator, yolox, extract_audio, and Whisper as columns. Video analysis AI is those expressions, cached per row. Compare Pixeltable vs Twelve Labs when a hosted video model is the product, and Pixeltable vs Voxel51 when a labeling UI is the product.

```python
import pixeltable as pxt
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.functions.yolox import yolox

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    detections = yolox(frame, model_id='yolox_s', threshold=0.5)
```

## Concepts

- [What is video intelligence?](https://www.pixeltable.com/learn/what-is-video-intelligence)
- [What is a frame iterator?](https://www.pixeltable.com/learn/what-is-a-frame-iterator)
- [What is object detection?](https://www.pixeltable.com/learn/what-is-object-detection)

## Use case

- [Video Intelligence Pipeline: Extract, Enrich, and Search Video at Scale](https://www.pixeltable.com/use-cases/video-content-analysis)

## Questions {#questions}

### Is this an Airflow DAG with ffmpeg tasks? {#faq-1}

No. The transforms are columns. Insert a clip and that row runs. Unchanged clips stay cached. Airflow can still sit outside when you already run a fleet of unrelated jobs; see Pixeltable vs Airflow.

### Where does video analysis AI run? {#faq-2}

On the stills and the soundtrack: YOLOX on each frame, Whisper on extracted audio. The output stays on those rows with frame_attrs so a detection is a timestamp.

### Can I use keyframes instead of 1 FPS? {#faq-3}

Yes. frame_iterator(..., keyframes_only=True) emits I-frames. Set only one of fps, num_frames, or keyframes_only. Detection then runs on fewer stills.

## Related

- [What is video intelligence?](https://www.pixeltable.com/learn/what-is-video-intelligence)
- [What is a frame iterator?](https://www.pixeltable.com/learn/what-is-a-frame-iterator)
- [What is object detection?](https://www.pixeltable.com/learn/what-is-object-detection)
- [Video intelligence pipeline](https://www.pixeltable.com/use-cases/video-content-analysis)
- [How do I extract keyframes from video in Python?](https://www.pixeltable.com/learn/how-to/extract-keyframes-from-video)
- [How do I build video RAG in Python?](https://www.pixeltable.com/learn/how-to/build-video-rag)
- [Pixeltable vs Twelve Labs](https://www.pixeltable.com/compare/pixeltable-vs-twelve-labs)
- [Pixeltable vs Voxel51](https://www.pixeltable.com/compare/pixeltable-vs-voxel51)
- [Pixeltable vs Airflow](https://www.pixeltable.com/compare/pixeltable-vs-airflow)
- [Object detection in videos](https://docs.pixeltable.com/howto/use-cases/object-detection-in-videos)