What is AI ETL?

Short answer

ETL means extract, transform, load. Classic warehouse ETL extracts from OLTP, transforms with SQL or Spark, and loads a star schema. AI ETL extracts files or media, transforms with model calls (embed, transcribe, caption, detect, split), and loads a vector index, a lake, or a training store. The letters are the same. The transform and the sink are not.

What AI ETL is not

It is not a vector database. The index is a load target. It is not a context window stuffed with a folder. It is not an agent memory product. It is not a nightly full refresh by definition: some AI ETL jobs rescan everything; some recompute only changed rows.

How the job usually runs

Three common shapes. They can sit beside each other.

• A batch engine scans a lake and writes a sink. Ray Data is a streaming dataset library for AI on Ray (https://docs.ray.io/en/latest/data/data.html). Daft is a multimodal dataframe (https://docs.daft.ai/en/stable/). Apache Spark is a unified analytics engine (https://spark.apache.org/docs/latest/).
• A scheduled DAG (Airflow and cousins) calls extractors, model APIs, and upserts. You own retries and the mapping from file to vector id.
• A schema of computed columns: insert or edit a row and the declared model expressions run for that row. Unchanged rows stay cached. That is still extract-transform-load; the scheduler is the catalog.

How to choose

Match the engine to whether the work is a scan or a live insert.

• Pick Spark, Ray Data, or Daft for a one-shot or periodic scan over a lake.
• Pick a DAG when many systems must be orchestrated and the company already runs that scheduler.
• Pick computed columns when new files arrive as rows and you do not want to rebuild the corpus on every model or prompt change.

Last reviewed

Facts on this page were checked against vendor documentation on 2026-10-28. Product surfaces move; treat version names and APIs as directional and follow the official links in Related.

Warehouse ETL versus AI ETL (last reviewed: Oct 2026)

Where Pixeltable fits

Use Pixeltable when AI ETL should be the schema: typed Document, Image, Video, or Audio columns, computed model columns, iterator views, and EmbeddingIndex, so insert is the extract and the load. It is not Spark, Ray Data, or Daft for a petabyte lake scan. Many teams run both: a batch engine for a backfill, tables for the live app. How-to: build an AI data pipeline in Python. Pixeltable vs Airflow covers the DAG shape.

import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import sentence_transformer
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
embed = sentence_transformer.using(
    model_id='sentence-transformers/all-MiniLM-L6-v2'
)

class Docs(TableModel, name='docs'):
    document: pxt.Document
    title: pxt.String

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=512
    ),
):
    __indexes__ = [pxt.EmbeddingIndex(text, string_embed=embed)]

@pxt.query
def search_docs(query: str, limit: int = 5):
    sim = Chunks.text.similarity(string=query)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(Chunks.text, Chunks.title, sim)
    )

api = FastAPIRouter(name='docs')
api.add_insert_route(Docs, path='/docs', inputs=[Docs.document, Docs.title])
api.add_query_route(path='/search', query=search_docs)

Questions