ml frameworkmoderateOfficial

Hugging Face Integration

Import Hugging Face datasets into Pixeltable and run CLIP, Sentence Transformers, ViT, and DETR as computed columns without an API key.

Get Started Docs

Challenge

Hugging Face gives you datasets and model weights. Production still needs a table that holds the rows, a way to run those models on insert, and an index that stays current. A notebook that loads a Dataset, loops tensors, and writes vectors to disk does not update when a new split arrives. CLIP for image search and a Sentence Transformer for captions usually live in two scripts. Pixeltable already stores Image and String columns. The integration is the UDFs in pixeltable.functions.huggingface and the import helper that maps Dataset feature types onto those columns.

Solution

Declare the table with TableModel. Use clip.using and sentence_transformer.using on EmbeddingIndex, or vit_for_image_classification and detr_for_object_detection as computed fields. Weights download from the Hub and run locally; there is no Hugging Face Inference API key on this path. pip install torch transformers sentence-transformers datasets as the model family requires. To load a Hub dataset, pxt.io.import_huggingface_dataset maps feature types (Image, Audio, string, ClassLabel, and the rest documented on the import page) into a new table. An app that already has a schema inserts rows the same way it inserts any other media.

Integration Steps

Step 1 of 4

Declare an Image column and a CLIP embedding index so text and stills share one space.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Images(TableModel, name='images'):
    image: pxt.Image
    name: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def search_images(query: str, limit: int = 5):
    sim = Images.image.similarity(string=query)  # string= for a sentence; image= for a still
    return (
        Images.order_by(sim, asc=False)
        .limit(limit)
        .select(Images.image, Images.name, sim)
    )

# pxt schema update app.py hf_vision -f

💡 embedding= on an Image column registers CLIP for both stills and query strings. similarity(string=) ranks by a sentence. similarity(image=) ranks by another still. torch and transformers are required.

Use arrow keys to navigate

Key Benefits

Local Hub checkpoints: no Inference API key for CLIP, ViT, DETR, or Sentence Transformers

Dataset import maps Image, Audio, and ClassLabel onto typed columns

EmbeddingIndex stays current on insert

One module for embeddings, classification, detection, ASR, and generation

Same TableModel file locally and on Cloud

Use Cases

•Image search with CLIP over a Hub or local set

•Semantic search over captions or reviews

•Object detection stored on the same row as the file

•Importing a DatasetDict while keeping the split name

•ASR on Audio columns with automatic_speech_recognition

Technical Requirements

•Python 3.11+

•pip install -U 'pixeltable[serve]' torch transformers sentence-transformers datasets

•Disk and RAM for the chosen Hub checkpoint

•pxt init before pxt schema update

Learn More

Working with Hugging Face Hugging Face functions SDK S3, Hugging Face, and Pixeltable storage

S3 Files, Hugging Face Buckets, and the Storage Problem Pixeltable Already Solved Build Multimodal AI Applications Faster with Pixeltable Infrastructure Your First Pixeltable Project: Build a Smart Image Organizer in 10 Minutes

Related Integrations

OpenAI
Modality: Model API
Computed column: OpenAI: Setup OpenAI Integration
Docs
Anthropic
Modality: Model API
Computed column: Anthropic: Install and set the API key
Docs
Gemini
Modality: Model API
Computed column: Gemini: Authenticate AI Studio or Vertex
Docs

Ready to Integrate Hugging Face?

Get started with Pixeltable and Hugging Face in minutes.

pip install pixeltable

View on GitHub Starter Kit Get started Try in Notebooks