Why CLIP embeddings instead of a text-only model?

A text-only sentence transformer cannot embed a picture. CLIP maps images and text into one space, so a sentence retrieves a photo. That is the usual way to index images for RAG.

Can I search with an image as well as a sentence?

Yes. Images.image.similarity(string=query) ranks stills against a phrase. Images.image.similarity(image=still) ranks against a picture. Both read the same CLIP index.

Do I copy vectors into Pinecone?

Only if a hosted ANN service is already the serving contract. For typical image RAG, the index lives on the image column. Pixeltable vs Pinecone states that split.

How to

How do I embed images for search in Python?

Image embeddings map pictures into a vector space so similar stills can be ranked. CLIP embeddings put text and images in one space, which is how a sentence retrieves a photo. Indexing images for RAG means those vectors live on the rows that hold the files, so a new picture is searchable when it is inserted.

Updated 2026-10-14 · Part of How do I ... in Python?

Before you start

About 15 min.

Python 3.11+
pip install -U 'pixeltable[serve]'
pxt init in the project root
torch and transformers for CLIP

1. Store pictures as an image column

An Image column is the row you will index. A URL in a string is not an embedding. Captions are optional extra text on the same row.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None

2. Declare CLIP embeddings on that column

embedding= registers both CLIP overloads, so a sentence and a still share one index. That is how you index images for RAG: the vector is a property of the file row, not a copy in another store.

class Images(TableModel, name='images'):
    image: pxt.Image
    caption: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

3. Search with a sentence or with another still

similarity(string=) ranks pictures against a phrase. similarity(image=) ranks against a picture. Both work because CLIP embeddings share one space.

@pxt.query
def search_images(query: str, limit: int = 5):
    sim = Images.image.similarity(string=query)
    return (
        Images.order_by(sim, asc=False)
        .limit(limit)
        .select(Images.image, Images.caption, sim)
    )

# Image query: Images.image.similarity(image=still)

Where image embeddings live

Where image embeddings live
	Index on the image row	A separate vector database
Vector	A column index on the file	A copy you upsert
Text query	CLIP similarity(string=)	Only if you also stored a text embedding
New file	Indexed on insert	A job that copies the vector
RAG	Retrieve the still, then generate	Retrieve an id, then join back to bytes

Where Pixeltable fits

Pixeltable declares CLIP as EmbeddingIndex on the Image column. Image embeddings stay on the row that holds the file. Query with similarity(string=) or similarity(image=). A separate vector database is optional; see Pixeltable vs Pinecone when hosted ANN is the product.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Images(TableModel, name='images'):
    image: pxt.Image
    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def search_images(query: str, limit: int = 5):
    sim = Images.image.similarity(string=query)
    return Images.order_by(sim, asc=False).limit(limit).select(Images.image, sim)

Questions

Why CLIP embeddings instead of a text-only model?: A text-only sentence transformer cannot embed a picture. CLIP maps images and text into one space, so a sentence retrieves a photo. That is the usual way to index images for RAG.
Can I search with an image as well as a sentence?: Yes. Images.image.similarity(string=query) ranks stills against a phrase. Images.image.similarity(image=still) ranks against a picture. Both read the same CLIP index.
Do I copy vectors into Pinecone?: Only if a hosted ANN service is already the serving contract. For typical image RAG, the index lives on the image column. Pixeltable vs Pinecone states that split.