---
title: "Build an incremental AI pipeline that only reprocesses new or changed files"
description: "An incremental pipeline tracks each file as a row, stores model outputs as computed columns, and recomputes only new or changed rows and their dependents. A new file, a changed file, or a changed prompt reruns that work; unchanged rows keep their stored results. Pixeltable is the unified multimodal AI backend for apps and agents that need this cache."
keywords:
  - incremental pipeline
  - reprocess only new or changed files
  - incremental ai data processing
  - incremental computation
  - computed columns
  - reduce ai compute costs
  - efficient ai pipelines
  - incremental embedding updates
complexity: "beginner"
estimated_time: "15 min"
url: "https://www.pixeltable.com/use-cases/incremental-updates-ai-data-processing"
---

# Build an incremental AI pipeline that only reprocesses new or changed files

An incremental pipeline tracks each file as a row, stores model outputs as computed columns, and recomputes only new or changed rows and their dependents. A new file, a changed file, or a changed prompt reruns that work; unchanged rows keep their stored results. Pixeltable is the unified multimodal AI backend for apps and agents that need this cache.

## Prerequisites

- Basic understanding of AI pipelines
- Python programming

## The Problem

The usual answer to "only reprocess new or changed files" is a stitch: cron or Airflow/Prefect to list a folder, hash each file, skip known hashes, call a model, write vectors into pgvector or another store, then hope the skip list matches the table. Ray or Spark will scan a lake; skip logic is yours. Databricks jobs on Delta can MERGE, if you write that job. cocoindex-style frameworks track source files and rebuild derived index entries for what changed; you still own serving. A Python loop that restarts from file zero pays the model again.

## The Solution

Declare a Document (or other media) column, a computed column that calls a model, and an optional embedding index. Insert a new file and only that row runs. Update the file or its source text and only that row and its dependents recompute. After you edit a prompt or a model expression, pxt recompute reruns that column. The catalog, not a hash file you maintain, is the cache. Keep Ray, Spark, Daft, or Databricks for a lake backfill and load the result.

## Implementation

### Track each file as a row

Store the file as a typed Document column. Apply the file with pxt schema update; do not call create_table at import.

```python
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.openai import chat_completions, embeddings

TableModel = pxt.model_base()
embed = embeddings.using(model='text-embedding-3-small')

class Docs(TableModel, name='docs'):
    title: pxt.String
    document: pxt.Document
```

The catalog records the file. A path string in isolation is not enough: splitting and models need a typed Document column. Detection of a "changed file" is an insert or an update of that row, not a filesystem mtime watcher.


### Compute per row and store the output

document_splitter emits one row of text per chunk. An LLM call and an embedding index read that text. Insert runs those expressions for the new file. Unchanged files keep stored results.

```python
class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=512
    ),
):
    summary = chat_completions(
        messages=[{'role': 'user', 'content': text}],
        model='gpt-4o-mini',
    ).choices[0].message.content
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=embed),
    ]

# pxt schema update app.py docs -f
```

text comes from the PDF (or other Document). summary and the embedding index depend on that text, so they rerun when the file changes. You do not copy vectors into a second database and then write a job to keep them aligned.


### Insert new files; only those rows run

After schema update, insert paths. A new file chunks and computes that file. Updating the Document column rebuilds chunks, summaries, and the index for that file only.

```python
# After: pxt schema update app.py docs -f
docs = pxt.get_table('docs.docs')
docs.insert([
    {'title': 'Q4 report', 'document': 'q4.pdf'},
])
# That insert chunks and embeds only this file. Other rows stay cached.

docs.update(
    {'document': 'q4-revised.pdf'},
    where=docs.title == 'Q4 report',
)
# That update rebuilds chunks, summaries, and the index for this file.
```

A second insert of a different file is another row. Re-inserting the same path as a new row is a new computation; changing an existing file is an update of that row's Document column.


### Rerun when the prompt or model changes

Editing the chat_completions expression changes the column. pxt recompute reruns that column and the columns that depend on it. Unrelated columns stay cached.

```python
# After you edit the summary prompt or model in app.py:
# pxt schema update app.py docs -f
# pxt recompute docs/chunks summary -f
#
# Changing string_embed on EmbeddingIndex drops and rebuilds that index
# (pxt schema update app.py docs --allow-destructive -f). That is a different operation
# from inserting a file.
```

The cache key is the declared expression plus the source row. A nightly full refresh is a different operation you would have to ask for.


## What reruns when a file, a prompt, or a model changes

|  | Cron + Airflow/Prefect + hashes + vector DB | Ray / Spark batch | Databricks (Delta + jobs) | cocoindex-style framework | Pixeltable computed columns |
| --- | --- | --- | --- | --- | --- |
| New file | Your skip list must miss it, then you write the vector | The job you launch; skip logic is yours | Auto Loader can ingest new files; you still write the pipeline that embeds | Recomputes derived entries for that source | Insert the row; only that row runs |
| Changed file | Hash must change, then you overwrite the vector | Rerun the scan unless you key the sink | A Delta Sync index re-embeds changed rows; MERGE or a job still owns the table | Rebuilds derived entries for that source | Update the Document column; the splitter view and dependents of that file recompute |
| Changed prompt or model | You invalidate caches and replay the DAG | Relaunch the job | Relaunch the job | Rebuild the derived index | pxt recompute on that column; unrelated columns stay cached |
| Where state lives | Hash file, object store, orchestrator, vector DB | The sink you write | Delta tables, Auto Loader, jobs, and an optional Delta Sync index | The framework store plus the store you serve from | The catalog: source columns, computed columns, indexes |

## Benefits

- A new file computes only itself; existing rows keep stored results
- An updated row recomputes only its dependent columns
- pxt recompute reruns a prompt or model change without a folder scan
- The embedding index updates with the row, so a separate vector-DB sync job is optional
- Airflow or Prefect can still insert rows; the table is the skip list

## Use Cases

- Document collections that gain files every day
- RAG indexes that must not re-embed the corpus on each drop
- LLM summaries over a live table rather than a lake scan
- Pipelines that used to hash files in Airflow before calling a model

## Requirements

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- OpenAI API key for chat_completions and embeddings in this sample

## Resources

- [What is incremental computation?](https://www.pixeltable.com/learn/what-is-incremental-computation) - Recompute only the rows a change affects
- [What is a computed column?](https://www.pixeltable.com/learn/what-is-a-computed-column) - A stored column computed from other columns on insert and update
- [How to run LLM calls over millions of rows](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows) - Cache per row; a lake scan is a batch engine, not a Python loop
- [How do I add computed columns with LLM calls in Python?](https://www.pixeltable.com/learn/how-to/add-computed-columns-with-llm-calls) - Assign chat_completions on the class so each new or changed row runs
- [Production RAG in one Python file](https://www.pixeltable.com/use-cases/production-rag-implementation) - Chunk, embed, retrieve, and generate on the same incremental table
- [Computed columns](https://docs.pixeltable.com/datastore/computed-columns) - Docs: expressions, stored results, and recompute
- [Incremental Embedding Indexes](https://www.pixeltable.com/blog/pixeltable-incremental-embedding-indexes) - Always-fresh indexes without rebuilds
- [Declarative, Multimodal, Incremental](https://www.pixeltable.com/blog/declarative-multimodal-incremental) - Deep dive into incremental computation

## Frequently asked questions

### What happens when I change the model or prompt? {#faq-1}

Edit the computed-column expression and apply it with pxt schema update. pxt recompute reruns that column and the columns that depend on it. Unrelated columns keep their stored results. Changing string_embed on an EmbeddingIndex drops and rebuilds that index (pxt schema update app.py docs --allow-destructive -f); that is not the same as inserting a file.

### How does it detect changed files? {#faq-2}

It does not watch filesystem mtime. A new file is an insert of a row. A changed file is an update of that row's Document column. Those writes rebuild the splitter view and recompute dependents of that file. A path you never insert is not a row.

### Can I keep Airflow or Prefect? {#faq-3}

Yes. Use the orchestrator to land files and insert rows (or call an HTTP insert route). The table is the skip list: unchanged rows are not recomputed. You do not need Airflow to hash files before the model call unless that DAG is already the contract.

### Do I still need Ray, Spark, or Databricks? {#faq-4}

Use a batch engine when the work is a scan over a lake you do not want as a live application catalog. Use computed columns when new files arrive as rows and the cache must survive the next insert. You can backfill with Ray, Spark, Daft, or Databricks and then serve new rows from the table.

### Does a new insert reprocess the whole table? {#faq-5}

No. A computed column recomputes the row that changed and the columns that depend on it. Unchanged rows keep the cached result. A full refresh is a different operation you would have to ask for.

### Do I still need a separate vector database? {#faq-6}

You need an embedding index so similar chunks can be ranked. That index can live on the splitter view that holds the extracted text. A separate vector database is optional when something else is already the source of truth.

## How-to guides

- [How do I add computed columns with LLM calls in Python?](https://www.pixeltable.com/learn/how-to/add-computed-columns-with-llm-calls)

