How do I deduplicate a large image dataset and keep embeddings for near-duplicate search?

Before you start

About 25 min.

• Python 3.11+
• pip install -U 'pixeltable[serve]'
• pxt init in the project root
• torch and transformers for CLIP

1. Store the set you will keep

Put each file in a table as image. Exact-hash and near-duplicate search both need a typed column, not a folder of paths.

import hashlib
from io import BytesIO

import pixeltable as pxt
from pixeltable.functions.huggingface import clip

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Images(TableModel, name='images'):
    image: pxt.Image

2. Hash a normalized PNG for exact duplicates

There is no built-in image hash function, so write a small UDF. Hash a normalized RGBA PNG so two identical pictures share one digest even if the original file encoding differed. Alpha stays in the digest. This is not a hash of the original file bytes.

def rgba_png_sha256(img) -> str:
    buf = BytesIO()
    img.convert('RGBA').save(buf, format='PNG')
    return hashlib.sha256(buf.getvalue()).hexdigest()

@pxt.udf
def image_sha256(img: pxt.Image) -> str:
    return rgba_png_sha256(img)

class Images(TableModel, name='images'):
    image: pxt.Image
    file_sha256 = image_sha256(image)

3. Index CLIP embeddings for near duplicates

Near duplicates survive a resize or a recompress. CLIP places stills in one space so cosine similarity can group them. Declare the index on the image column.

class Images(TableModel, name='images'):
    image: pxt.Image
    file_sha256 = image_sha256(image)
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

4. Query neighbors above a threshold

Rank the kept set against a still. Keep rows whose similarity is above the threshold. 0.95 is a starting point; tune on a labelled sample of true pairs and false pairs from your data.

@pxt.query
def near_duplicates(img: pxt.Image, threshold: float = 0.95, limit: int = 10):
    sim = Images.image.similarity(image=img)
    return (
        Images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(Images.image, Images.file_sha256, sim)
    )

5. Check new uploads before insert

Schema update binds a @pxt.query written over another model when it creates the computed column, so a second table can store this check without a serving route. A column on Images that queried Images would be self-referencing. This guide checks in a function you call before insert: look up the normalized digest, then run CLIP only when that lookup is empty. POST /images does not call it. Two uploads at once can both pass and both land.

from PIL import Image

def keep_if_new(candidate, threshold: float = 0.95) -> None:
    images = pxt.get_table('image_dedup.images')
    digest = rgba_png_sha256(candidate)
    exact = images.where(images.file_sha256 == digest).select(images.file_sha256).collect()
    if len(exact) > 0:
        return
    sim = images.image.similarity(image=candidate)
    hits = (
        images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(10)
        .select(images.file_sha256, sim)
        .collect()
    )
    if len(hits) == 0:
        images.insert([{'image': candidate}])

# pxt schema update app.py image_dedup -f
# Then, from a shell that has imported the app:
# keep_if_new(Image.open('candidate.png'))

One-off corpus cleanup vs keeping the set

Where Pixeltable fits

Pixeltable keeps the deduplicated dataset and its embeddings in one file: a hash UDF, a CLIP EmbeddingIndex, and a near-duplicate query. Check a candidate before insert: look up the normalized digest, then run CLIP only when that lookup is empty. POST /images does not run the check, so two uploads at once can both pass. Schema update binds a @pxt.query over another model when it creates a computed column, so a second table can store the check without a serving route. A computed column on Images that queried Images would be self-referencing. For a one-off dedup of a very large corpus, batch tools built for that job are a better fit: NVIDIA NeMo Curator image dedup (GPU embedding clustering), FiftyOne image deduplication plugin (operators such as find_approximate_duplicate_images, interactive review in the App), Lightly sampling, or FAISS plus DuckDB scripts. This schema is not a measured million-image benchmark; test on your data. Pixeltable vs Voxel51 covers the App versus the production table. Pixeltable Cloud is Limited Beta.

import hashlib
from io import BytesIO

import pixeltable as pxt
from pixeltable.functions.huggingface import clip
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

def rgba_png_sha256(img) -> str:
    buf = BytesIO()
    img.convert('RGBA').save(buf, format='PNG')
    return hashlib.sha256(buf.getvalue()).hexdigest()

@pxt.udf
def image_sha256(img: pxt.Image) -> str:
    return rgba_png_sha256(img)

class Images(TableModel, name='images'):
    image: pxt.Image
    file_sha256 = image_sha256(image)
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def near_duplicates(img: pxt.Image, threshold: float = 0.95, limit: int = 10):
    sim = Images.image.similarity(image=img)
    return (
        Images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(Images.image, Images.file_sha256, sim)
    )

def keep_if_new(candidate, threshold: float = 0.95) -> None:
    images = pxt.get_table('image_dedup.images')
    digest = rgba_png_sha256(candidate)
    exact = images.where(images.file_sha256 == digest).select(images.file_sha256).collect()
    if len(exact) > 0:
        return
    sim = images.image.similarity(image=candidate)
    hits = (
        images.where(sim > threshold)
        .order_by(sim, asc=False)
        .limit(10)
        .select(images.file_sha256, sim)
        .collect()
    )
    if len(hits) == 0:
        images.insert([{'image': candidate}])

api = FastAPIRouter(name='image_dedup')
api.add_insert_route(Images, path='/images', uploadfile_inputs=[Images.image])
api.add_query_route(path='/near-duplicates', query=near_duplicates, uploadfile_inputs=['img'])

# POST /images does not call keep_if_new. Call it yourself after schema update.
# Two uploads at once can both pass and both land.

# pxt init
# pxt schema update app.py image_dedup -f
# pxt service update app.py image_dedup -f

Questions

Exact vs near duplicates?

Exact duplicates match a hash of a normalized RGBA PNG, so the same pixels, including alpha, share one digest when the file encoding differs. Near duplicates are visually similar after a resize, crop, or recompress; CLIP similarity above a threshold groups them. Look up the digest first so identical pictures never need a vector comparison.

What similarity threshold?

0.95 is a starting point on CLIP cosine similarity, not a universal cutoff. Tune on a labelled sample of true pairs and false pairs from your data. Lower it if you miss near-duplicates; raise it if unrelated stills cluster.

Can I keep the embeddings for search afterwards?

Yes. The CLIP index on Images is the search index. After you decide which files to keep, query with similarity(string=) or similarity(image=) as in the image-search how-to.

How does this compare to FiftyOne's dedup plugin?

FiftyOne image deduplication is an App workflow: operators such as find_approximate_duplicate_images and interactive review (Voxel51 plugin docs, accessed 2026-10-23). Use it when a person must look at the clusters. This page is a check you run before insert. POST /images does not run it. See Pixeltable vs Voxel51.

Does it scale to millions of images?

This page does not claim a million-image throughput number. For a one-off cleanup of a very large corpus, use a batch tool built for that job, then load the keepers here if you want the index and the insert check going forward. Test on your data.