---
title: "How do I deduplicate a large image dataset and keep embeddings for near-duplicate search?"
description: "Deduplicate a large image dataset in Python: hash exact copies, find near-duplicates with CLIP, and check new files against the indexed set before insert."
url: "https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings"
updated: "2026-10-23"
doc: "https://docs.pixeltable.com/howto/cookbooks/search/search-similar-images"
---

# How do I deduplicate a large image dataset and keep embeddings for near-duplicate search?

Two passes: exact duplicates by hashing a normalized RGBA PNG; near duplicates by embedding each image (CLIP or similar) and treating pairs above a cosine-similarity threshold as one group. Keep the embeddings indexed so every new image is checked against the set before insert instead of re-running the whole job. POST /images does not run that check.

Updated: 2026-10-23
Part of [How do I ... in Python?](https://www.pixeltable.com/learn/how-to).

## Before you start

About 25 min.

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- pxt init in the project root
- torch and transformers for CLIP

## On this page

- [Before you start](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#before-you-start)
- [Store the set you will keep](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#step-1)
- [Hash a normalized PNG for exact duplicates](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#step-2)
- [Index CLIP embeddings for near duplicates](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#step-3)
- [Query neighbors above a threshold](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#step-4)
- [Check new uploads before insert](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#step-5)
- [Comparison](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/how-to/deduplicate-images-with-embeddings#questions)

## 1. Store the set you will keep {#step-1}

Put each file in a table as image. Exact-hash and near-duplicate search both need a typed column, not a folder of paths.

```python
import hashlib
from io import BytesIO

import pixeltable as pxt
from pixeltable.functions.huggingface import clip

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Images(TableModel, name='images'):
    image: pxt.Image
```

## 2. Hash a normalized PNG for exact duplicates {#step-2}

There is no built-in image hash function, so write a small UDF. Hash a normalized RGBA PNG so two identical pictures share one digest even if the original file encoding differed. Alpha stays in the digest. This is not a hash of the original file bytes.

```python
def rgba_png_sha256(img) -> str:
    buf = BytesIO()
    img.convert('RGBA').save(buf, format='PNG')
    return hashlib.sha256(buf.getvalue()).hexdigest()

@pxt.udf
def image_sha256(img: pxt.Image) -> str:
    return rgba_png_sha256(img)

class Images(TableModel, name='images'):
    image: pxt.Image
    file_sha256 = image_sha256(image)
```

## 3. Index CLIP embeddings for near duplicates {#step-3}

Near duplicates survive a resize or a recompress. CLIP places stills in one space so cosine similarity can group them. Declare the index on the image column.

```python
class Images(TableModel, name='images'):
    image: pxt.Image
    file_sha256 = image_sha256(image)
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]
```

## 4. Query neighbors above a threshold {#step-4}

Rank the kept set against a still. Keep rows whose similarity is above the threshold. 0.95 is a starting point; tune on a labelled sample of true pairs and false pairs from your data.

```python
@pxt.query
def near_duplicates(img: pxt.Image, threshold: float = 0.95, limit: int = 10):
    sim = Images.image.similarity(image=img)
    return (
        Images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(Images.image, Images.file_sha256, sim)
    )
```

## 5. Check new uploads before insert {#step-5}

Schema update binds a @pxt.query written over another model when it creates the computed column, so a second table can store this check without a serving route. A column on Images that queried Images would be self-referencing. This guide checks in a function you call before insert: look up the normalized digest, then run CLIP only when that lookup is empty. POST /images does not call it. Two uploads at once can both pass and both land.

```python
from PIL import Image

def keep_if_new(candidate, threshold: float = 0.95) -> None:
    images = pxt.get_table('image_dedup.images')
    digest = rgba_png_sha256(candidate)
    exact = images.where(images.file_sha256 == digest).select(images.file_sha256).collect()
    if len(exact) > 0:
        return
    sim = images.image.similarity(image=candidate)
    hits = (
        images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(10)
        .select(images.file_sha256, sim)
        .collect()
    )
    if len(hits) == 0:
        images.insert([{'image': candidate}])

# pxt schema update app.py image_dedup -f
# Then, from a shell that has imported the app:
# keep_if_new(Image.open('candidate.png'))
```

## One-off corpus cleanup vs keeping the set {#comparison}

| Job | Batch corpus tools | Index on the table |
| --- | --- | --- |
| Exact duplicates | Byte or perceptual hash in a script | A normalized RGBA PNG hash you look up before CLIP |
| Near duplicates | GPU clustering (NeMo Curator), FAISS plus DuckDB, Lightly sampling | CLIP similarity above a threshold you tune |
| Review UI | FiftyOne image deduplication plugin in the App | Query results; not a CV visualizer |
| New file arrives | Rerun the job on the corpus | keep_if_new before insert; POST /images does not check |
| Keep embeddings for search | Export vectors to a second store | The same CLIP index answers search later |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable keeps the deduplicated dataset and its embeddings in one file: a hash UDF, a CLIP EmbeddingIndex, and a near-duplicate query. Check a candidate before insert: look up the normalized digest, then run CLIP only when that lookup is empty. POST /images does not run the check, so two uploads at once can both pass. Schema update binds a @pxt.query over another model when it creates a computed column, so a second table can store the check without a serving route. A computed column on Images that queried Images would be self-referencing. For a one-off dedup of a very large corpus, batch tools built for that job are a better fit: NVIDIA NeMo Curator image dedup (GPU embedding clustering), FiftyOne image deduplication plugin (operators such as find_approximate_duplicate_images, interactive review in the App), Lightly sampling, or FAISS plus DuckDB scripts. This schema is not a measured million-image benchmark; test on your data. Pixeltable vs Voxel51 covers the App versus the production table. Pixeltable Cloud is Limited Beta.

```python
import hashlib
from io import BytesIO

import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

def rgba_png_sha256(img) -> str:
    buf = BytesIO()
    img.convert('RGBA').save(buf, format='PNG')
    return hashlib.sha256(buf.getvalue()).hexdigest()

@pxt.udf
def image_sha256(img: pxt.Image) -> str:
    return rgba_png_sha256(img)

class Images(TableModel, name='images'):
    image: pxt.Image
    file_sha256 = image_sha256(image)
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def near_duplicates(img: pxt.Image, threshold: float = 0.95, limit: int = 10):
    sim = Images.image.similarity(image=img)
    return (
        Images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(Images.image, Images.file_sha256, sim)
    )

def keep_if_new(candidate, threshold: float = 0.95) -> None:
    images = pxt.get_table('image_dedup.images')
    digest = rgba_png_sha256(candidate)
    exact = images.where(images.file_sha256 == digest).select(images.file_sha256).collect()
    if len(exact) > 0:
        return
    sim = images.image.similarity(image=candidate)
    hits = (
        images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(10)
        .select(images.file_sha256, sim)
        .collect()
    )
    if len(hits) == 0:
        images.insert([{'image': candidate}])

api = FastAPIRouter(name='image_dedup')
api.add_insert_route(Images, path='/images', uploadfile_inputs=[Images.image])
api.add_query_route(path='/near-duplicates', query=near_duplicates, uploadfile_inputs=['img'])

# POST /images does not call keep_if_new. Call it yourself after schema update.
# Two uploads at once can both pass and both land.

# pxt init
# pxt schema update app.py image_dedup -f
# pxt service update app.py image_dedup -f
```

## Concepts

- [What is CLIP?](https://www.pixeltable.com/learn/what-is-clip)
- [What is an embedding index?](https://www.pixeltable.com/learn/what-is-an-embedding-index)
- [What are multimodal embeddings?](https://www.pixeltable.com/learn/what-are-multimodal-embeddings)

## Use case

- [Computer Vision Pipeline: Object Detection, Classification, and Search](https://www.pixeltable.com/use-cases/computer-vision-pipeline-optimization)

## Questions {#questions}

### Exact vs near duplicates? {#faq-1}

Exact duplicates match a hash of a normalized RGBA PNG, so the same pixels, including alpha, share one digest when the file encoding differs. Near duplicates are visually similar after a resize, crop, or recompress; CLIP similarity above a threshold groups them. Look up the digest first so identical pictures never need a vector comparison.

### What similarity threshold? {#faq-2}

0.95 is a starting point on CLIP cosine similarity, not a universal cutoff. Tune on a labelled sample of true pairs and false pairs from your data. Lower it if you miss near-duplicates; raise it if unrelated stills cluster.

### Can I keep the embeddings for search afterwards? {#faq-3}

Yes. The CLIP index on Images is the search index. After you decide which files to keep, query with similarity(string=) or similarity(image=) as in the image-search how-to.

### How does this compare to FiftyOne's dedup plugin? {#faq-4}

FiftyOne image deduplication is an App workflow: operators such as find_approximate_duplicate_images and interactive review (Voxel51 plugin docs, accessed 2026-10-23). Use it when a person must look at the clusters. This page is a check you run before insert. POST /images does not run it. See Pixeltable vs Voxel51.

### Does it scale to millions of images? {#faq-5}

This page does not claim a million-image throughput number. For a one-off cleanup of a very large corpus, use a batch tool built for that job, then load the keepers here if you want the index and the insert check going forward. Test on your data.

## Related

- [What is CLIP?](https://www.pixeltable.com/learn/what-is-clip)
- [What is an embedding index?](https://www.pixeltable.com/learn/what-is-an-embedding-index)
- [What are multimodal embeddings?](https://www.pixeltable.com/learn/what-are-multimodal-embeddings)
- [Computer vision pipeline](https://www.pixeltable.com/use-cases/computer-vision-pipeline-optimization)
- [How do I embed images for search in Python?](https://www.pixeltable.com/learn/how-to/embed-images-for-search)
- [How do I serve image similarity search?](https://www.pixeltable.com/learn/how-to/serve-image-similarity-search)
- [Pixeltable vs Voxel51](https://www.pixeltable.com/compare/pixeltable-vs-voxel51)
- [Find similar images with CLIP](https://docs.pixeltable.com/howto/cookbooks/search/search-similar-images)