---
title: "What is AI ETL?"
description: "AI ETL is extract-transform-load where the transform is a model call, and the load is often a vector index or a lake. Last reviewed October 2026."
url: "https://www.pixeltable.com/learn/what-is-ai-etl"
updated: "2026-10-28"
vertical: "Orchestration"
doc: "https://docs.pixeltable.com/datastore/computed-columns"
---

# What is AI ETL?

AI ETL is extract, transform, and load where the transform is a model: embed a passage, transcribe audio, caption an image, or split a document. Warehouse ETL moves tables with SQL. AI ETL moves files and embeddings. The load target is often a vector index or a lake, not only a star schema.

Updated: 2026-10-28
Part of [What is a declarative pipeline?](https://www.pixeltable.com/learn/what-is-a-declarative-pipeline).

## On this page

- [Short answer](https://www.pixeltable.com/learn/what-is-ai-etl#direct-answer)
- [What AI ETL is not](https://www.pixeltable.com/learn/what-is-ai-etl#not-the-same)
- [How the job usually runs](https://www.pixeltable.com/learn/what-is-ai-etl#how-it-runs)
- [How to choose](https://www.pixeltable.com/learn/what-is-ai-etl#how-to-choose)
- [Last reviewed](https://www.pixeltable.com/learn/what-is-ai-etl#review)
- [Comparison](https://www.pixeltable.com/learn/what-is-ai-etl#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/what-is-ai-etl#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/what-is-ai-etl#questions)

## Short answer {#direct-answer}

ETL means extract, transform, load. Classic warehouse ETL extracts from OLTP, transforms with SQL or Spark, and loads a star schema. AI ETL extracts files or media, transforms with model calls (embed, transcribe, caption, detect, split), and loads a vector index, a lake, or a training store. The letters are the same. The transform and the sink are not.

## What AI ETL is not {#not-the-same}

It is not a vector database. The index is a load target. It is not a context window stuffed with a folder. It is not an agent memory product. It is not a nightly full refresh by definition: some AI ETL jobs rescan everything; some recompute only changed rows.

## How the job usually runs {#how-it-runs}

Three common shapes. They can sit beside each other.

- A batch engine scans a lake and writes a sink. Ray Data is a streaming dataset library for AI on Ray (https://docs.ray.io/en/latest/data/data.html). Daft is a multimodal dataframe (https://docs.daft.ai/en/stable/). Apache Spark is a unified analytics engine (https://spark.apache.org/docs/latest/).
- A scheduled DAG (Airflow and cousins) calls extractors, model APIs, and upserts. You own retries and the mapping from file to vector id.
- A schema of computed columns: insert or edit a row and the declared model expressions run for that row. Unchanged rows stay cached. That is still extract-transform-load; the scheduler is the catalog.

## How to choose {#how-to-choose}

Match the engine to whether the work is a scan or a live insert.

- Pick Spark, Ray Data, or Daft for a one-shot or periodic scan over a lake.
- Pick a DAG when many systems must be orchestrated and the company already runs that scheduler.
- Pick computed columns when new files arrive as rows and you do not want to rebuild the corpus on every model or prompt change.

## Last reviewed {#review}

Facts on this page were checked against vendor documentation on 2026-10-28. Product surfaces move; treat version names and APIs as directional and follow the official links in Related.

## Warehouse ETL versus AI ETL (last reviewed: Oct 2026) {#comparison}

|  | Warehouse ETL | AI ETL batch job | Insert-triggered schema |
| --- | --- | --- | --- |
| Extract | OLTP tables, files, APIs | Object storage, lakes, media folders | An insert or update of a typed row |
| Transform | SQL, Spark, dbt | Embed, transcribe, caption, split, detect | Computed columns on that row |
| Load | Warehouse / star schema | Vector index, lake, training store | The same table, plus its indexes |
| When it reruns | A scheduled window | The job you launch | When the row or a dependency changes |
| Typical engines | Spark, warehouse ELT tools | Ray Data, Daft, Spark, a DAG | A catalog of declared expressions |

## Where Pixeltable fits {#where-pixeltable-fits}

Use Pixeltable when AI ETL should be the schema: typed Document, Image, Video, or Audio columns, computed model columns, iterator views, and EmbeddingIndex, so insert is the extract and the load. It is not Spark, Ray Data, or Daft for a petabyte lake scan. Many teams run both: a batch engine for a backfill, tables for the live app. How-to: build an AI data pipeline in Python. Pixeltable vs Airflow covers the DAG shape.

```python
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import sentence_transformer
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
embed = sentence_transformer.using(
    model_id='sentence-transformers/all-MiniLM-L6-v2'
)

class Docs(TableModel, name='docs'):
    document: pxt.Document
    title: pxt.String

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=512
    ),
):
    __indexes__ = [pxt.EmbeddingIndex(text, string_embed=embed)]

@pxt.query
def search_docs(query: str, limit: int = 5):
    sim = Chunks.text.similarity(string=query)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(Chunks.text, Chunks.title, sim)
    )

api = FastAPIRouter(name='docs')
api.add_insert_route(Docs, path='/docs', inputs=[Docs.document, Docs.title])
api.add_query_route(path='/search', query=search_docs)
```

## Questions {#questions}

### Is a vector database AI ETL? {#faq-1}

No. A vector database stores embeddings and answers nearest-neighbor queries. AI ETL is the job that produces those embeddings and keeps them aligned with the source file.

### Is Spark AI ETL? {#faq-2}

Spark can run AI ETL if the transform is a model UDF and the sink is an index or a lake. Spark is also warehouse ETL. The letters depend on the transform, not on the brand.

### Is this the same as a declarative pipeline? {#faq-3}

A declarative pipeline is a schema of transforms the engine runs when rows change. That is one way to implement AI ETL. A Ray or Spark job is another. See What is a declarative pipeline?

### Do I still need object storage? {#faq-4}

Usually yes for the bytes. AI ETL reads files and writes derived columns or vectors. The catalog holds types and pointers; the bucket or media store holds the file.

## Related

- [What is a declarative pipeline?](https://www.pixeltable.com/learn/what-is-a-declarative-pipeline)
- [What is a computed column?](https://www.pixeltable.com/learn/what-is-a-computed-column)
- [Ray Data vs Daft vs Spark](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark)
- [How to run LLM calls over millions of rows](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows)
- [How do I build an AI data pipeline in Python?](https://www.pixeltable.com/learn/how-to/build-an-ai-data-pipeline)
- [Pixeltable vs Daft](https://www.pixeltable.com/compare/pixeltable-vs-daft)
- [Pixeltable vs Airflow](https://www.pixeltable.com/compare/pixeltable-vs-airflow)
- [Computed columns](https://docs.pixeltable.com/datastore/computed-columns)