How does video search work?

What it is

A video file is a sequence of pictures plus a timeline. Search that returns “beach.mp4” has matched a title. Search that returns a frame at 00:01:12 has matched the picture. The second kind is what people mean by video search, video retrieval, or finding a moment.

How it works

Three steps. The frame rate you choose is the tradeoff: more frames catch shorter moments and cost more embedding work.

• Sample. A view walks the video at a fixed rate, one row per frame, and keeps the frame index and the timestamp.
• Embed. A model such as CLIP turns each frame into a vector in a space where a text phrase and a picture can be neighbors.
• Query. The question is embedded the same way. The nearest frames come back with the source video and the time offset.

What it is not

Sending the whole file to a chat model on every question is not an index. It can describe one clip. It does not rank a library, and it does not get cheaper as the library grows.

A transcript search finds words that were spoken. It misses a red car that nobody named. Visual search and speech search answer different questions. A library often wants both, as two indexes on the same clip.

What you keep at each step of video search

Where Pixeltable fits

In Pixeltable a frame view uses frame_iterator on a Video column. An embedding index on the frame column is what the similarity query reads. The parent video and frame_attrs (index and time in seconds) stay on the frame row, so the hit is a moment. The step-by-step version is the video search engine use case.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Videos(TableModel, name='videos'):
    video: pxt.Video

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def find_moment(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(Frames.video, Frames.frame, Frames.frame_attrs, sim)
    )

Questions