Video search
Video frames extracted at one per second and indexed with CLIP, searchable by text or by image.
One app.py with 2 tables, 1 view, 5 computed columns, 6 HTTP routes. Needs pixeltable[serve]>=0.7.16.
app.py
"""pxt schema update app.py videointel && pxt service update app.py videointel"""# ruff: noqa: F821import pixeltable as pxtimport pixeltable.functions as pxtffrom pixeltable.functions import image as pxt_imagefrom pixeltable.functions.huggingface import clipfrom pixeltable.serving import FastAPIRouterTableModel = pxt.model_base()clip_embed = clip.using(model_id="openai/clip-vit-base-patch32")class Videos(TableModel, name="videos"):video: pxt.Videotitle: pxt.Stringuuid = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)timestamp: pxt.Timestamp | Noneclass Frames(TableModel,name="frames",base=Videos,iterator=pxtf.video.frame_iterator(Videos.video, fps=1.0),):thumbnail = pxt_image.b64_encode(pxt_image.thumbnail(frame, size=(320, 320)))__indexes__ = [pxt.EmbeddingIndex(frame, embedding=clip_embed, name="frames_clip")]class Images(TableModel, name="images"):image: pxt.Imagelabel: pxt.String | Nonesource_id: pxt.Stringuuid = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)timestamp: pxt.Timestamp | Nonethumbnail = pxt_image.b64_encode(pxt_image.thumbnail(image, size=(128, 128)))width = image.widthheight = image.heightmode = image.mode@pxt.querydef search_visual(query_text: str, limit: int = 20) -> pxt.Query:sim = Frames.frame.similarity(string=query_text)return (Frames.where(sim > 0.2).order_by(sim, asc=False).select(Frames.thumbnail, timestamp=Frames.frame_attrs.time, score=sim).limit(limit))@pxt.querydef list_images() -> pxt.Query:return Images.select(Images.uuid,Images.label,Images.source_id,Images.width,Images.height,Images.thumbnail,).order_by(Images.timestamp, asc=False)api = FastAPIRouter(name="videointel", prefix="/api")api.add_insert_route(Videos,path="/ingest",uploadfile_inputs=[Videos.video],inputs=[Videos.title],outputs=[Videos.uuid],background=True,)api.add_insert_route(Images,path="/ingest/image",uploadfile_inputs=[Images.image],inputs=[Images.label, Images.source_id],outputs=[Images.uuid],)api.add_query_route(path="/search/visual", query=search_visual, method="post")api.add_query_route(path="/images", query=list_images, method="get")api.add_delete_route(Videos, path="/delete/video")api.add_delete_route(Images, path="/delete/image")
What app.py declares
Drawn from app.py at 15824a3; each row links to its line.
- runs on insert
- stored
- index
- route
serves videointel
- POST/api/ingestinsert, videostakes title, video (file); returns uuidL68
- POST/api/ingest/imageinsert, imagestakes label, source_id, image (file); returns uuidL76
- POST/api/search/visualquery, search_visualtakes query_text, limitL83
- GET/api/imagesquery, list_imagesL84
- POST/api/delete/videodelete, videosL85
- POST/api/delete/imagedelete, imagesL86
Run it
Needs uv. The routes then answer on port 8000.
terminal
git clone https://github.com/pixeltable/pixeltable-starter-kit &&cd pixeltable-starter-kit &&git checkout 15824a3f20f3c67bd355c75471bbdecd8de586ec &&cd video-search &&uv sync &&uv run pxt schema update app.py videointel &&uv run pxt service run app.py videointel --port 8000
Related apps
- Climbing Route ToposClimbing-route topo registry API: photo uploads with thumbnails, grayscale copies, grade bands and index-backed crag lookupsShares: Images, video and audio, File uploads
- Mural RegistryPublic-art mural registry API: wall photo uploads with thumbnails, grayscale copies and color tagsShares: Images, video and audio, File uploads
- Travel ScrapbookTravel photo scrapbook API: image uploads, thumbnails and place tags as computed columnsShares: Images, video and audio, File uploads