---
title: "Video search reference architecture"
description: "A video search architecture: ingest clips, sample frames, optional speech, CLIP index, then HTTP. Named alternatives per slot including Twelve Labs."
url: "https://www.pixeltable.com/reference-architecture/video-search"
updated: "2026-10-28"
doc: "https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames"
---

# Video search reference architecture

Video search ranks a moment in a library, not a filename. Sample stills, embed them in a shared text-and-image space, and return the parent clip with a presentation timestamp. A hosted video API is a different product: JSON over HTTP, no table you own.

Updated: 2026-10-28
Docs: https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames

Video search is not "send the file to a chat model." That can describe one clip. It does not rank a library, and it does not get cheaper as the library grows. Search retrieves a timestamped still, and optionally a transcript line, then returns the parent clip.
This page names the slots. The Python how-tos are How do I index a video library and search it by text, How do I extract keyframes from video, and How do I build video RAG. Compare Pixeltable vs Twelve Labs and Pixeltable vs Mixpeek when a managed video API is the product.

## Video search: a clip becomes timestamped stills, then a ranked moment {#diagram}

```mermaid
flowchart TB
  file[Video file] --> col[Video column]
  col --> frames[frame_iterator view]
  col --> audio[extract_audio]
  audio --> whisper[Whisper transcript]
  frames --> clipIdx[CLIP EmbeddingIndex]
  whisper --> textIdx[Transcript EmbeddingIndex]
  clipIdx --> search["similarity(string=query)"]
  textIdx --> search
  search --> http[FastAPIRouter /search]
  col -.->|keep if already the contract| s3[S3 or R2]
  clipIdx -.->|managed video API| hosted[Twelve Labs or Mixpeek]
```

## Role of each component

- **Clip store.** A typed Video column, or an object store plus a path table. Later frame extraction needs a media type, not a spreadsheet of paths.
- **Frame sampler.** One row per still. keyframes_only emits I-frames. fps=1 samples once a second. num_frames emits evenly spaced stills. Set only one of those three. Each row keeps frame_attrs with a presentation timestamp.
- **Optional speech path.** extract_audio plus Whisper (or a hosted STT API on the same column). Useful when the question is about what was said rather than what was shown.
- **Multimodal index.** CLIP on the frame column so a sentence retrieves a still. A separate vector database is optional. Twelve Labs Marengo is a hosted video embedding you can also call as a column.
- **Search HTTP.** Frame search returns title, frame_attrs, and score. Transcript search returns the clip, not a timestamp inside it. FastAPIRouter exposes insert and both searches. A managed API returns JSON and no table.

## Honest alternatives per slot {#alternatives}

| Slot | Role | Pixeltable | Named alternatives |
| --- | --- | --- | --- |
| Object store | Bytes for the clips | A Video column; media under PIXELTABLE_HOME, or a URL you insert | Amazon S3: The bucket is already the archive; Cloudflare R2: S3-compatible storage with no egress fee on that vendor; MinIO: An S3 API inside your network |
| Frame extraction | Stills with timestamps | frame_iterator on a view; frame_attrs["time"] is the presentation timestamp | ffmpeg in a worker: You already dump JPEGs and must keep that job; PySceneDetect: You want shot boundaries rather than a fixed fps |
| Video embedding / search API | Rank a moment from a sentence or an image | CLIP EmbeddingIndex on the frame column; optional Whisper on the clip | Twelve Labs: You want hosted Marengo / Pegasus and no GPU of your own; Mixpeek: A managed multimodal retrieval API is the product; VideoDB: A hosted video database API, not a table you own; Qdrant or pgvector: You already embed frames elsewhere and only need ANN |
| Speech-to-text | Transcript for spoken-word search | extract_audio plus whisper.transcribe, or openai.transcriptions on the same column | Deepgram, AssemblyAI, Gladia: A hosted STT API is the model choice; Pixeltable is not STT; WhisperX: You need speakers and word-level timestamps; that is a separate cookbook |
| Serving | Insert a clip and search | FastAPIRouter beside the tables | Their HTTP API: Twelve Labs or Mixpeek is the serving contract; Hand-written FastAPI: You already join S3, a vector DB, and a worker |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable is the unified multimodal backend for this architecture: a Video column, a frame_iterator view, CLIP EmbeddingIndex, optional Whisper, and FastAPIRouter insert plus search in one file. Keep S3 when the bucket is the archive. Keep Twelve Labs or Mixpeek when a managed video API is the product. Keep Qdrant or pgvector when you already embed frames in another job. Pixeltable Cloud hosts the same app file.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)
    __indexes__ = [
        pxt.EmbeddingIndex(transcript, string_embed=text_embed),
    ]

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def search_video(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )

@pxt.query
def search_speech(query: str, limit: int = 5):
    sim = Videos.transcript.similarity(string=query)
    return (
        Videos.order_by(sim, asc=False)
        .limit(limit)
        .select(Videos.title, Videos.video, score=sim)
    )

api = FastAPIRouter(name='video_lib')
api.add_insert_route(
    Videos,
    path='/videos',
    inputs=[Videos.title],
    uploadfile_inputs=[Videos.video],
)
api.add_query_route(path='/search', query=search_video)
api.add_query_route(path='/search-speech', query=search_speech)

# pxt init
# pxt schema update app.py video_lib -f
# pxt service update app.py video_lib -f
```

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. [Get started](https://www.pixeltable.com/get-started) covers the CLI.

## Questions {#questions}

### When should I use Twelve Labs instead? {#faq-1}

When you want hosted Marengo and Pegasus, no GPU of your own, and you are fine calling their API. Pixeltable can also call those models as columns when you want the table and the timestamps in a file you own. Pixeltable vs Twelve Labs is that split.

### Do I index every encoded frame? {#faq-2}

Almost never. I-frames, a fixed fps, or a fixed count are the documented sampling contracts. Every encoded frame explodes row counts and embedding cost. Only one of fps, num_frames, or keyframes_only can be set.

### Where is the timestamp? {#faq-3}

frame_attrs["time"] is the presentation timestamp in seconds (pts times the stream time base). frame_attrs["index"] is the stream index. Seconds from the start of the clip was the retired pos_msec value.

## Related

- [How do I index a video library and search it by text?](https://www.pixeltable.com/learn/how-to/search-a-video-library-by-text)
- [How do I extract keyframes from video?](https://www.pixeltable.com/learn/how-to/extract-keyframes-from-video)
- [How do I build video RAG in Python?](https://www.pixeltable.com/learn/how-to/build-video-rag)
- [How does video search work?](https://www.pixeltable.com/learn/how-video-search-works)
- [Video search engine use case](https://www.pixeltable.com/use-cases/video-search-engine)
- [Video search example app](https://www.pixeltable.com/recipes/video-search)
- [Object detection in videos notebook](https://www.pixeltable.com/notebooks/object-detection-in-videos)
- [Examples hub: video search](https://www.pixeltable.com/examples)
- [Pixeltable vs Twelve Labs](https://www.pixeltable.com/compare/pixeltable-vs-twelve-labs)
- [Pixeltable vs Mixpeek](https://www.pixeltable.com/compare/pixeltable-vs-mixpeek)
- [Extract frames from videos (docs)](https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames)

This is the architecture. Step-by-step Python is on [how-to guides](https://www.pixeltable.com/learn/how-to) and [examples](https://www.pixeltable.com/examples).