What are multimodal embeddings used for?

Cross-modal search and multimodal RAG. Because images and text share a space, a sentence can retrieve a picture or a video frame, and that retrieved still can be passed to a generator with a pointer to the source file.

What is a multimodal embedding model?

A model that maps more than one modality into the same vector space. CLIP is the usual image-text example (Radford et al., ICML 2021). SigLIP is a CLIP-style successor with a sigmoid loss (Zhai et al., ICCV 2023). ImageBind binds additional modalities (Girdhar et al., CVPR 2023). Pixeltable's documented image-text path is huggingface.clip with openai/clip-vit-base-patch32.

What is the best multimodal embedding model?

There is no universal winner. Pick a model whose space covers the query and the stored items. For image-text search and RAG, CLIP is the documented default on Pixeltable. Keep a text-only sentence transformer for passages. Hosted CLIP-style APIs (Jina CLIP, Voyage multimodal) are an option when you do not want to run weights locally. Pixeltable is not a standalone vector database; the embedding is an index on the source row.

How do multimodal embeddings enable cross-modal search and multimodal RAG?

The query and the stored items are embedded with the same model. Nearest neighbors are the hits. For search, that is a ranked still. For RAG, those stills, pages, or transcript windows are the evidence the model sees. Pixeltable declares that index on the image or frame column and queries it with similarity(string=) or similarity(image=).

Retrieval

What are multimodal embeddings?

Multimodal embeddings are vectors that place images, text, and other media in one shared space, so a sentence and a picture that mean similar things sit near each other. A CLIP-style model is the usual example. That shared space is how text-to-image search and multimodal RAG retrieve a still or a page.

Updated 2026-10-19 · Part of What is an embedding?

How it works

A multimodal embedding model maps each item into a vector in one space.
CLIP-style image-text models embed a phrase and a picture so neighbors can be compared.
Cross-modal search ranks pictures by a sentence. Multimodal RAG passes those retrieved stills, pages, or clips to a generator, with a pointer back to the source.

Common multimodal embedding model families

There is no single best multimodal embedding model. The right family is the one whose space covers both the query and the stored items, and whose license and hosting you can operate. Figures below are from the papers, not a bake-off on this site.

CLIP (Radford et al., ICML 2021, "Learning Transferable Visual Models From Natural Language Supervision") is the usual image-text family. The paper describes contrastive training on 400 million image-text pairs so a sentence and a picture share a space. OpenCLIP (Ilharco et al.; github.com/mlfoundations/open_clip) is an open CLIP-style training stack.

SigLIP (Zhai et al., ICCV 2023, "Sigmoid Loss for Language Image Pre-Training") keeps the image-text setup and replaces the softmax contrastive loss with a pairwise sigmoid loss. ImageBind (Girdhar et al., CVPR 2023, "ImageBind: One Embedding Space To Bind Them All") learns one space across six modalities (images, text, audio, depth, thermal, and IMU), using image-paired data to bind the rest.

Hosted CLIP-style APIs exist for teams that do not run weights locally, including Jina CLIP models and Voyage multimodal models. A text-only embedding is the wrong tool when the retrieval unit is a picture or a frame.

What it is not

They are not a text-only sentence transformer, and they are not a captioning model that writes a paragraph instead of a vector.

multimodal embeddings: this, and the thing it is confused with

multimodal embeddings: this, and the thing it is confused with
	This	Not this
Space	One shared space for several modalities	A text model and an image model that cannot be compared
Query	A sentence retrieving a picture, or the reverse	Only the same file type
RAG	Retrieve a frame or page as evidence	Retrieve a text chunk while images sit unused

Where Pixeltable fits

Pixeltable is the unified multimodal AI backend for apps and agents: declare a CLIP EmbeddingIndex on an Image column, and a text query ranks those stills with similarity(string=). The index lives on the source row. Pixeltable Cloud is Limited Beta. A separate vector database is optional.

import pixeltable as pxt
from pixeltable.functions.huggingface import clip

TableModel = pxt.model_base()
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Images(TableModel, name='images'):
    image: pxt.Image

    __indexes__ = [
        pxt.EmbeddingIndex(image, embedding=clip_embed),
    ]

@pxt.query
def search_images(query: str, limit: int = 5):
    sim = Images.image.similarity(string=query)
    return Images.order_by(sim, asc=False).limit(limit).select(Images.image, sim)

Questions

What are multimodal embeddings used for?: Cross-modal search and multimodal RAG. Because images and text share a space, a sentence can retrieve a picture or a video frame, and that retrieved still can be passed to a generator with a pointer to the source file.
What is a multimodal embedding model?: A model that maps more than one modality into the same vector space. CLIP is the usual image-text example (Radford et al., ICML 2021). SigLIP is a CLIP-style successor with a sigmoid loss (Zhai et al., ICCV 2023). ImageBind binds additional modalities (Girdhar et al., CVPR 2023). Pixeltable's documented image-text path is huggingface.clip with openai/clip-vit-base-patch32.
What is the best multimodal embedding model?: There is no universal winner. Pick a model whose space covers the query and the stored items. For image-text search and RAG, CLIP is the documented default on Pixeltable. Keep a text-only sentence transformer for passages. Hosted CLIP-style APIs (Jina CLIP, Voyage multimodal) are an option when you do not want to run weights locally. Pixeltable is not a standalone vector database; the embedding is an index on the source row.
How do multimodal embeddings enable cross-modal search and multimodal RAG?: The query and the stored items are embedded with the same model. Nearest neighbors are the hits. For search, that is a ranked still. For RAG, those stills, pages, or transcript windows are the evidence the model sees. Pixeltable declares that index on the image or frame column and queries it with similarity(string=) or similarity(image=).