How to

How do I migrate from FAISS to an embedding index on the table?

Moving off a FAISS index means keeping the passages that produced the vectors, embedding them with the same model you will query with, and ranking neighbors on that table. A pickled IndexFlatL2 file has no source text. Rebuilding search is an insert plus an embedding index, not a converter for the binary file. You still need the original strings.

Updated 2026-10-28 · Part of How do I ... in Python?

Before you start

About 20 min.

Python 3.11+
pip install -U 'pixeltable[serve]'
pxt init in the project root
sentence-transformers for all-MiniLM-L6-v2

1. Keep the strings that built the index

A FAISS IndexFlatL2 stores vectors. Semantic search still needs the passage that produced each vector, plus any metadata you will filter on. Export those strings from the job that added them. The .index file is not enough.

import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Notes(TableModel, name='notes'):
    title: pxt.String
    content: pxt.String
    source: pxt.String | None

2. Declare the embedding index on that column

Use the same embedding function you used when you added vectors to FAISS. An index on the column updates when you insert. You do not pickle a second IndexFlatL2.

class Notes(TableModel, name='notes'):
    title: pxt.String
    content: pxt.String
    source: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(content, string_embed=text_embed),
    ]

# pxt schema update app.py faiss_notes -f

3. Insert the exported passages

Load the rows you exported (CSV, JSON, or a list of dicts). Insert once. Each row is embedded. There is no faiss.read_index step.

notes = pxt.get_table('faiss_notes.notes')
notes.insert([
    {
        'title': 'Election of 1904',
        'content': 'Theodore Roosevelt won the 1904 US presidential election.',
        'source': 'export.json',
    },
])

4. Rank with the same model at query time

Embed the question with the same function. order_by the similarity score descending. Limit is the number of neighbors. Keyword search is a different index.

@pxt.query
def search_notes(query: str, limit: int = 5):
    sim = Notes.content.similarity(string=query)
    return (
        Notes.order_by(sim, asc=False)
        .limit(limit)
        .select(Notes.title, Notes.content, sim)
    )

FAISS index file versus an index on the table

FAISS index file versus an index on the table
Job	FAISS IndexFlatL2	Index on the table
Stores	Vectors in a file	The passage and the vector
Source text	You keep it somewhere else	A String column on the row
Insert	index.add then write_index	Insert a row
Query	index.search on a numpy vector	column.similarity(string=)
New model	Rebuild the file from source	A destructive schema update rebuilds the index

Where Pixeltable fits

Pixeltable does not import a FAISS file. Keep the passages, declare EmbeddingIndex, insert, then similarity(string=). FAISS still fits when another process already owns that binary index and only needs ANN. Coming from another stack is the docs hub for this move.

import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Notes(TableModel, name='notes'):
    content: pxt.String
    __indexes__ = [
        pxt.EmbeddingIndex(content, string_embed=text_embed),
    ]

@pxt.query
def search_notes(query: str, limit: int = 5):
    sim = Notes.content.similarity(string=query)
    return Notes.order_by(sim, asc=False).limit(limit).select(Notes.content, sim)

# pxt schema update app.py faiss_notes -f

Questions

Can I load a pickled FAISS index?

No. The binary file has vectors without the source strings. Export the passages from the job that built the index, insert those rows, and let the embedding index rebuild.

Do I keep the same embedding model?

Yes, if you want the same neighbor ranking. all-MiniLM-L6-v2 is the documented fast local default. Switching models later is a destructive schema update, not a FAISS converter.

When should I keep FAISS?

When another system already serves that IndexFlatL2 (or IVF, HNSW) and that file is the contract. This page is for search that should live next to the passages.

Next step

Copy the commands into an agent, or run them locally with pip install pixeltable.

Get started Quick Start