How to

How do I migrate from Pinecone to an embedding index on the table?

Leaving Pinecone means exporting the source text and metadata you upserted, not the vectors alone, then storing those rows in a table with an embedding index that updates on insert. Pinecone is a hosted ANN service. You keep it when that engine is the serving contract. You drop the upsert worker when the table already holds the passages.

Updated 2026-10-28 · Part of How do I ... in Python?

Before you start

About 20 min.

Python 3.11+
pip install -U 'pixeltable[serve]'
pxt init in the project root
sentence-transformers for all-MiniLM-L6-v2

1. Export the text you upserted, not the vectors

A Pinecone vector without metadata text cannot cite anything. Dump id, the original string, and filters such as namespace or tenant. Leave the embedding values behind; they will be rebuilt.

import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Notes(TableModel, name='notes'):
    pinecone_id: pxt.String
    content: pxt.String
    namespace: pxt.String | None

2. Put the index on the source column

Namespaces become a metadata column you can filter. The embedding index is on the text, not a second hosted collection.

class Notes(TableModel, name='notes'):
    pinecone_id: pxt.String
    content: pxt.String
    namespace: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(content, string_embed=text_embed),
    ]

# pxt schema update app.py pinecone_notes -f

3. Insert the exported rows

Map each exported record onto pinecone_id, content, and namespace. Insert rebuilds the vectors. A later insert of a new passage does not require a Pinecone upsert client.

notes = pxt.get_table('pinecone_notes.notes')
notes.insert([
    {
        'pinecone_id': 'doc-1904',
        'content': 'Theodore Roosevelt won the 1904 US presidential election.',
        'namespace': 'history',
    },
])

4. Filter metadata, then rank by similarity

where() replaces a Pinecone metadata filter. similarity(string=) ranks the rest. That is hybrid search as dense plus structured predicates, not BM25 fusion.

@pxt.query
def search_notes(query: str, namespace: str, limit: int = 5):
    sim = Notes.content.similarity(string=query)
    return (
        Notes.where(Notes.namespace == namespace)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(Notes.pinecone_id, Notes.content, sim)
    )

Pinecone upsert versus an index on the table

Pinecone upsert versus an index on the table
Job	Pinecone	Index on the table
Stores	Vectors plus metadata you copy in	The passage and the vector
Write path	An upsert client and a worker	Insert a row
Namespace	A Pinecone namespace	A column you filter with where()
Query	A hosted query API	column.similarity(string=)
Keep it when	That engine is the serving contract	The table already holds the passages

Where Pixeltable fits

Pixeltable vs Pinecone is the product split: keep Pinecone when hosted ANN is the contract. For a table that already stores the text, export metadata strings, insert, and declare EmbeddingIndex. Do not copy vectors. Coming from another stack is the docs hub.

import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Notes(TableModel, name='notes'):
    pinecone_id: pxt.String
    content: pxt.String
    namespace: pxt.String | None
    __indexes__ = [
        pxt.EmbeddingIndex(content, string_embed=text_embed),
    ]

@pxt.query
def search_notes(query: str, namespace: str, limit: int = 5):
    sim = Notes.content.similarity(string=query)
    return (
        Notes.where(Notes.namespace == namespace)
        .order_by(sim, asc=False)
        .limit(limit)
        .select(Notes.pinecone_id, Notes.content, sim)
    )

# pxt schema update app.py pinecone_notes -f

Questions

Should I copy the Pinecone vectors?

No. Export the source text and filters. The embedding index rebuilds vectors with the model you declare. Copied vectors would still need that model at query time and would drift if you switch models later.

What happens to namespaces?

Store them as a column and filter with where() before ranking. That is metadata plus dense similarity on the same row, not a second Pinecone index.

When do I keep Pinecone?

When that hosted engine is already the serving contract, with its own SLA and clients. Pixeltable vs Pinecone states that split. This page is for dropping the upsert worker.

Next step

Copy the commands into an agent, or run them locally with pip install pixeltable.

Get started Quick Start