Video search reference architecture

Video search: a clip becomes timestamped stills, then a ranked moment

Role of each component

Clip store

A typed Video column, or an object store plus a path table. Later frame extraction needs a media type, not a spreadsheet of paths.

Frame sampler

One row per still. keyframes_only emits I-frames. fps=1 samples once a second. num_frames emits evenly spaced stills. Set only one of those three. Each row keeps frame_attrs with a presentation timestamp.

Optional speech path

extract_audio plus Whisper (or a hosted STT API on the same column). Useful when the question is about what was said rather than what was shown.

Multimodal index

CLIP on the frame column so a sentence retrieves a still. A separate vector database is optional. Twelve Labs Marengo is a hosted video embedding you can also call as a column.

Search HTTP

Frame search returns title, frame_attrs, and score. Transcript search returns the clip, not a timestamp inside it. FastAPIRouter exposes insert and both searches. A managed API returns JSON and no table.

Honest alternatives per slot

Keep the named engine when it is already the contract. Pixeltable fills the slot when you want that job on the same table as the files.

Where Pixeltable fits

Pixeltable is the unified multimodal backend for this architecture: a Video column, a frame_iterator view, CLIP EmbeddingIndex, optional Whisper, and FastAPIRouter insert plus search in one file. Keep S3 when the bucket is the archive. Keep Twelve Labs or Mixpeek when a managed video API is the product. Keep Qdrant or pgvector when you already embed frames in another job. Pixeltable Cloud hosts the same app file.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)
    __indexes__ = [
        pxt.EmbeddingIndex(transcript, string_embed=text_embed),
    ]

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def search_video(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )

@pxt.query
def search_speech(query: str, limit: int = 5):
    sim = Videos.transcript.similarity(string=query)
    return (
        Videos.order_by(sim, asc=False)
        .limit(limit)
        .select(Videos.title, Videos.video, score=sim)
    )

api = FastAPIRouter(name='video_lib')
api.add_insert_route(
    Videos,
    path='/videos',
    inputs=[Videos.title],
    uploadfile_inputs=[Videos.video],
)
api.add_query_route(path='/search', query=search_video)
api.add_query_route(path='/search-speech', query=search_speech)

# pxt init
# pxt schema update app.py video_lib -f
# pxt service update app.py video_lib -f

To run a sketch, install pixeltable[serve] and the packages its model functions import (such as sentence-transformers, openai, or openai-whisper), export the provider key a model call reads (such as OPENAI_API_KEY), save the file as app.py, and run the commands in its last lines. The tables start empty: insert a row, or POST to an insert route, then select the computed columns. Get started covers the CLI.

Questions