Do I need a vector database to search video by text?

You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth. Pixeltable vs Pinecone is that split.

How do I get the timestamp of the match?

Each still keeps frame_attrs. time is the presentation timestamp in seconds. Return it from the query and seek the parent clip to that moment. A filename alone is not a timestamp.

Should I embed frames or transcripts?

Frames catch what is on screen. Transcripts catch what was said. Use both when the library has speech and visuals. CLIP on frames is the usual first index; Whisper plus a text embedding is the optional second.

How do I add new videos without reindexing?

Insert a row. Computed columns, the frame view, and the indexes run for that clip. Unchanged clips stay as they are.

Pixeltable vs Mixpeek or Twelve Labs?

Mixpeek and Twelve Labs are managed video APIs: you call HTTP, they run the models, and pricing is on their sites. They are the better choice when you do not want to run models, or you need their hosted video models. Pixeltable is the table those results can live on, plus CLIP, Whisper, and HTTP you own. See Pixeltable vs Mixpeek and Pixeltable vs Twelve Labs.

How to

How do I index a video library and search it by text in Python?

Text search over video means sampling frames and embedding them with a text-image model such as CLIP, and optionally transcribing the soundtrack and embedding that transcript. Frame search returns a moment (the clip plus seconds). Transcript search returns the clip, because the transcript is one string for the file. A new clip indexes only that clip.

Updated 2026-10-23 · Part of How do I ... in Python?

Before you start

About 30 min.

Python 3.11+
pip install -U 'pixeltable[serve]'
pxt init in the project root
ffmpeg
openai-whisper
torch and transformers for CLIP
sentence-transformers

1. Store videos as a typed video column

Put each clip in a table as video, plus any metadata you will filter on. A path string is not enough: frame extraction and audio extraction need a typed column.

import pixeltable as pxt

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video

2. Sample frames and keep the timestamp

A view walks each clip and emits one still per sample. fps=1 is one still a second. Keep frame_attrs so a hit is a moment, not a filename.

from pixeltable.functions.video import frame_iterator

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    pass

# frame_attrs["time"] is the presentation timestamp in seconds

3. Embed frames with a text-image model

CLIP places a sentence and a still in one space. Declare the index on the frame column so a new clip is indexed when it is inserted. You do not copy vectors into a second database.

from pixeltable.functions.huggingface import clip

clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

4. Optionally transcribe speech and index the text

Extract the soundtrack, transcribe it, and index the transcript with a text embedding model. Leave these columns out and the upload only samples frames. Frame search finds what is on screen and returns a moment. Transcript search finds what was said and returns the clip, not a timestamp inside it.

from pixeltable.functions.huggingface import sentence_transformer
from pixeltable.functions.video import extract_audio
from pixeltable.functions.whisper import transcribe

text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)
    __indexes__ = [
        pxt.EmbeddingIndex(transcript, string_embed=text_embed),
    ]

@pxt.query
def search_speech(query: str, limit: int = 5):
    sim = Videos.transcript.similarity(string=query)
    return (
        Videos.order_by(sim, asc=False)
        .limit(limit)
        .select(Videos.title, Videos.video, score=sim)
    )

5. Rank frames by similarity to a sentence

Embed the question with the same CLIP model. Return the parent title, frame_attrs, and score. The time field is the moment to seek to.

@pxt.query
def search_video(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )

6. Serve insert and search over HTTP

Expose an upload route for new clips and a search route for the frame query. Add /search-speech when the transcript column from the previous step is in the schema. Apply the schema, then start the service. A new file indexes only that file.

from pixeltable.serving import FastAPIRouter

api = FastAPIRouter(name='video_lib')
api.add_insert_route(
    Videos,
    path='/videos',
    inputs=[Videos.title],
    uploadfile_inputs=[Videos.video],
)
api.add_query_route(path='/search', query=search_video)
api.add_query_route(path='/search-speech', query=search_speech)

# pxt init
# pxt schema update app.py video_lib -f
# pxt service update app.py video_lib -f

The stack coding agents usually pick vs one schema

The stack coding agents usually pick vs one schema
Job	Glue stack	One schema	Hosted video APIs
Storage	SQLite plus files on disk	A Video column on the table you own	Your bucket, or an asset they host
Vector index	Chroma, Qdrant, or LanceDB as a second store	EmbeddingIndex on the frame view	Their hosted index
Transcription	whisper as a separate job	extract_audio, then Whisper on the row	Included if their pipeline extracts speech
Embeddings	sentence-transformers and CLIP in a worker	CLIP on frames; a text model on the transcript	Their proprietary video models
Job runner	A queue or a cron between stores	Computed columns and views on insert	Their ingest pipeline
Serving	FastAPI you write against those copies	FastAPIRouter beside the tables	Their HTTP API
New video added	Rerun the DAG or upsert by hand	Insert a row; only that clip is indexed	Upload an asset and wait until ready
Model changed	Re-embed the library in a batch job	Change the embedder in the file and re-apply	Depends on their reindex contract

Where Pixeltable fits

Pixeltable is the unified multimodal backend for this job: a Video column, a frame_iterator view, CLIP EmbeddingIndex, optional Whisper, and FastAPIRouter insert plus search in one file. Mixpeek, Twelve Labs, and VideoDB are managed video APIs you call over HTTP; you do not run the models, and pricing is on their sites. Pick them when you have no GPU, want a managed API, or need their proprietary video models. Pick this schema when the clip, the timestamp, and HTTP should live in one app.py you own. Pixeltable Cloud is Limited Beta. Compare Pixeltable vs Mixpeek and Pixeltable vs Twelve Labs.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.functions.whisper import transcribe
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Videos(TableModel, name='videos'):
    title: pxt.String
    video: pxt.Video
    audio = extract_audio(video)
    transcript = transcribe(audio, model='base').text.astype(pxt.String)
    __indexes__ = [
        pxt.EmbeddingIndex(transcript, string_embed=text_embed),
    ]

class Frames(
    TableModel,
    name='frames',
    base=Videos,
    iterator=frame_iterator(video=Videos.video, fps=1),
):
    __indexes__ = [
        pxt.EmbeddingIndex(frame, embedding=clip_embed),
    ]

@pxt.query
def search_video(query: str, limit: int = 5):
    sim = Frames.frame.similarity(string=query)
    return (
        Frames.order_by(sim, asc=False)
        .limit(limit)
        .select(title=Videos.title, frame_attrs=Frames.frame_attrs, score=sim)
    )

@pxt.query
def search_speech(query: str, limit: int = 5):
    sim = Videos.transcript.similarity(string=query)
    return (
        Videos.order_by(sim, asc=False)
        .limit(limit)
        .select(Videos.title, Videos.video, score=sim)
    )

api = FastAPIRouter(name='video_lib')
api.add_insert_route(
    Videos,
    path='/videos',
    inputs=[Videos.title],
    uploadfile_inputs=[Videos.video],
)
api.add_query_route(path='/search', query=search_video)
api.add_query_route(path='/search-speech', query=search_speech)

# pxt init
# pxt schema update app.py video_lib -f
# pxt service update app.py video_lib -f

Questions

Do I need a vector database to search video by text?: You need an embedding index so similar frames can be ranked. That index can live on the frame view. A separate vector database is optional when something else is already the source of truth. Pixeltable vs Pinecone is that split.
How do I get the timestamp of the match?: Each still keeps frame_attrs. time is the presentation timestamp in seconds. Return it from the query and seek the parent clip to that moment. A filename alone is not a timestamp.
Should I embed frames or transcripts?: Frames catch what is on screen. Transcripts catch what was said. Use both when the library has speech and visuals. CLIP on frames is the usual first index; Whisper plus a text embedding is the optional second.
How do I add new videos without reindexing?: Insert a row. Computed columns, the frame view, and the indexes run for that clip. Unchanged clips stay as they are.
Pixeltable vs Mixpeek or Twelve Labs?: Mixpeek and Twelve Labs are managed video APIs: you call HTTP, they run the models, and pricing is on their sites. They are the better choice when you do not want to run models, or you need their hosted video models. Pixeltable is the table those results can live on, plus CLIP, Whisper, and HTTP you own. See Pixeltable vs Mixpeek and Pixeltable vs Twelve Labs.