---
title: "Hugging Face Integration with Pixeltable: Datasets and Local Models"
description: "Import Hugging Face datasets into Pixeltable and run CLIP, Sentence Transformers, ViT, and DETR as computed columns without an API key."
tool: "Hugging Face"
keywords:
  - Hugging Face Pixeltable integration
  - Hugging Face datasets Python
  - CLIP embedding index
  - sentence-transformers Pixeltable
  - Hugging Face computed column
url: "https://www.pixeltable.com/integrations/hugging-face"
---

# Hugging Face Integration with Pixeltable: Datasets and Local Models

Import Hugging Face datasets into Pixeltable and run CLIP, Sentence Transformers, ViT, and DETR as computed columns without an API key.

## The Problem

Hugging Face gives you datasets and model weights. Production still needs a table that holds the rows, a way to run those models on insert, and an index that stays current. A notebook that loads a Dataset, loops tensors, and writes vectors to disk does not update when a new split arrives. CLIP for image search and a Sentence Transformer for captions usually live in two scripts. Pixeltable already stores Image and String columns. The integration is the UDFs in pixeltable.functions.huggingface and the import helper that maps Dataset feature types onto those columns.

## The Solution

Declare the table with TableModel. Use clip.using and sentence_transformer.using on EmbeddingIndex, or vit_for_image_classification and detr_for_object_detection as computed fields. Weights download from the Hub and run locally; there is no Hugging Face Inference API key on this path. pip install torch transformers sentence-transformers datasets as the model family requires. To load a Hub dataset, pxt.io.import_huggingface_dataset maps feature types (Image, Audio, string, ClassLabel, and the rest documented on the import page) into a new table. An app that already has a schema inserts rows the same way it inserts any other media.

## Implementation

### Index images with CLIP

Declare an Image column and a CLIP embedding index so text and stills share one space.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Images(TableModel, name='images'):
    image: pxt.Image
    name: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def search_images(query: str, limit: int = 5):
    sim = Images.image.similarity(string=query)  # string= for a sentence; image= for a still
    return (
        Images.order_by(sim, asc=False)
        .limit(limit)
        .select(Images.image, Images.name, sim)
    )

# pxt schema update app.py hf_vision -f
```

embedding= on an Image column registers CLIP for both stills and query strings. similarity(string=) ranks by a sentence. similarity(image=) ranks by another still. torch and transformers are required.


### Embed text with Sentence Transformers

Index a String column with all-MiniLM-L6-v2 and rank with similarity(string=).

```python
from pixeltable.functions.huggingface import sentence_transformer

text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Notes(TableModel, name='notes'):
    text: pxt.String
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

@pxt.query
def search_notes(query: str, limit: int = 5):
    sim = Notes.text.similarity(string=query)  # string= embeds the question
    return Notes.order_by(sim, asc=False).limit(limit).select(Notes.text, sim)

# pxt schema update app.py hf_text -f
```

string_embed= is the text path. all-MiniLM-L6-v2 is the documented fast local default. all-mpnet-base-v2 is stronger and slower. Use the same model_id at insert and at query.


### Import a Hub dataset

Map Hugging Face feature types into a Pixeltable table, then insert more rows later.

```python
import datasets
import pixeltable as pxt

# Notebook or script: creates a table from a Dataset.
# Apps that already have a TableModel insert into that table instead.
padoru = datasets.load_dataset('not-lain/padoru', split='train').select_columns(
    ['Image', 'ImageSize', 'Name', 'ImageSource']
)
images = pxt.io.import_huggingface_dataset('hf_demo.images', padoru)

# Add more of the same schema:
images.insert(padoru.select(range(5)))
```

import_huggingface_dataset is the documented helper. Image maps to pxt.Image, Audio to pxt.Audio, ClassLabel to String. For embeddings already in the Dataset, pass schema_overrides with pxt.Array[(dim,), pxt.Float]. extra_args column_name_for_split keeps train and test when the source is a DatasetDict.


### Classify and detect with ViT and DETR

Run Hub classification and detection checkpoints as computed columns on the image.

```python
from pixeltable.functions.huggingface import (
    detr_for_object_detection,
    vit_for_image_classification,
)

class Images(TableModel, name='images'):
    image: pxt.Image
    classifications = vit_for_image_classification(
        image, model_id='google/vit-base-patch16-224'
    )
    detections = detr_for_object_detection(
        image, model_id='facebook/detr-resnet-50'
    )

# pxt schema update app.py hf_detect -f
```

Use a ViT checkpoint trained for classification, such as google/vit-base-patch16-224, not a feature-extraction-only id. DETR returns scores, labels, label_text, and boxes as [x1, y1, x2, y2]. Other Hugging Face UDFs on the same module include automatic_speech_recognition, translation, and text_generation.


## Hub notebook versus a table that owns the models

| Job | Dataset plus a loop | Table plus huggingface UDFs |
| --- | --- | --- |
| Rows | A Dataset you reload | Typed columns, including Image and Audio |
| Embeddings | A .npy or a second index | EmbeddingIndex on the column |
| Detection | A script that writes JSON | detr_for_object_detection on the image |
| New split | Rerun the notebook | Insert, or import_huggingface_dataset again |

## Benefits

- Local Hub checkpoints: no Inference API key for CLIP, ViT, DETR, or Sentence Transformers
- Dataset import maps Image, Audio, and ClassLabel onto typed columns
- EmbeddingIndex stays current on insert
- One module for embeddings, classification, detection, ASR, and generation
- Same TableModel file locally and on Cloud

## Use Cases

- Image search with CLIP over a Hub or local set
- Semantic search over captions or reviews
- Object detection stored on the same row as the file
- Importing a DatasetDict while keeping the split name
- ASR on Audio columns with automatic_speech_recognition

## Performance


| Metric | Value | Description |

| --- | --- | --- |

| API key | None for local models | CLIP, Sentence Transformers, ViT, and DETR run on the machine that holds the table |

| Index updates | On insert | EmbeddingIndex embeds new rows; you do not export vectors to a second store |

| Import | Feature-type map | import_huggingface_dataset maps Hub features onto Pixeltable columns |

## Requirements

- Python 3.11+
- pip install -U 'pixeltable[serve]' torch transformers sentence-transformers datasets
- Disk and RAM for the chosen Hub checkpoint
- pxt init before pxt schema update

## Resources

- [Working with Hugging Face](https://docs.pixeltable.com/howto/providers/working-with-hugging-face) - Datasets, splits, streaming, CLIP, and Sentence Transformers
- [Hugging Face functions SDK](https://docs.pixeltable.com/sdk/latest/huggingface) - clip, sentence_transformer, ViT, DETR, ASR, and the rest
- [S3, Hugging Face, and Pixeltable storage](https://www.pixeltable.com/blog/s3-files-huggingface-buckets-pixeltable-storage) - Where files live when the source is a bucket or a Hub dataset