---
title: "How to run LLM calls over millions of rows"
description: "Do not loop from row zero. Cache each row, recompute only changes, or scan a lake once with Spark, Ray, or Daft. Last reviewed October 2026."
url: "https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows"
updated: "2026-10-28"
vertical: "Orchestration"
doc: "https://docs.pixeltable.com/datastore/computed-columns"
---

# How to run LLM calls over millions of rows

Running language-model calls over millions of rows is a cache and scheduling problem. A Python loop that restarts from row zero will re-pay every token. A batch engine such as Spark, Ray Data, or Daft scans a lake once. A computed column caches each row and recomputes only rows that changed.

Updated: 2026-10-28
Part of [What is incremental computation?](https://www.pixeltable.com/learn/what-is-incremental-computation).

## On this page

- [Short answer](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#direct-answer)
- [Why a restartable loop fails](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#why-loops-fail)
- [A lake scan versus a live table](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#two-engines)
- [How to choose](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#how-to-choose)
- [Last reviewed](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#review)
- [Comparison](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows#questions)

## Short answer {#direct-answer}

Do not wrap the model client in a for-loop you restart. Persist each result next to its source row. On the next run, skip rows whose inputs have not changed. A batch engine (Spark, Ray Data, Daft) is the right tool for a one-shot scan over a lake. A computed column is the right tool for a live table: insert recomputes that row; unchanged rows keep the cache. This page quotes no tokens-per-second and no dollar totals.

## Why a restartable loop fails {#why-loops-fail}

A script that embeds or summarizes every path in a folder looks fine until it dies at row 400,000 and you run it again. Without a stored result keyed by content (or by a declared dependency), you pay the model vendor a second time for work you already bought. Rate limits and partial failure make that worse. The fix is a cache with lineage, not a longer sleep().

## A lake scan versus a live table {#two-engines}

Pick the engine from the data shape.

- Lake scan: Ray Data streams CPU preprocess into GPU or API inference (https://docs.ray.io/en/latest/data/data.html). Daft is a multimodal dataframe (https://docs.daft.ai/en/stable/). Spark is a unified analytics engine (https://spark.apache.org/docs/latest/). Write the sink so a rerun can skip completed keys if you add that logic.
- Live table: declare the LLM call as a column. Insert or edit a row and that expression runs. Unchanged rows stay cached. Editing a prompt recomputes the columns that depend on it. Docs: computed columns in Related.

## How to choose {#how-to-choose}

This is not a bake-off. It is a fit check.

- Historical dump in a lake, run once or on a schedule: Spark, Ray Data, or Daft.
- New files arriving as application rows: a computed column.
- Both: backfill with a batch engine, then serve new rows from a table.

## Last reviewed {#review}

Facts on this page were checked against vendor documentation on 2026-10-28. Product surfaces move; treat version names and APIs as directional and follow the official links in Related.

## Ways to run LLM calls over a large table (last reviewed: Oct 2026) {#comparison}

| Approach | What reruns | Cache | Fit |
| --- | --- | --- | --- |
| Python for-loop | Whatever the script reads after a restart | Only if you write one | Prototypes. Not millions of rows |
| Spark / Ray Data / Daft job | The job you launch; skip logic is yours | The sink you write | A lake scan or a scheduled backfill |
| Hosted batch API | The vendor job | Vendor-dependent | When you do not want to run the cluster |
| Computed column | New or changed rows, and dependents of a prompt edit | Per row, with lineage | A live table that receives inserts |

## Where Pixeltable fits {#where-pixeltable-fits}

Use Pixeltable when the millions of rows are a live table: assign chat_completions on the class so each new or changed row runs, then caches. It is not a Spark-shaped scan over a lake. For that backfill, keep Ray Data, Daft, or Spark and load the result. How-to: add computed columns with LLM calls in Python. What is incremental computation? covers the cache rule. This page does not publish a tokens-per-second number.

```python
import pixeltable as pxt
from pixeltable.functions.openai import chat_completions

TableModel = pxt.model_base()

class Docs(TableModel, name='docs'):
    text: pxt.String
    source: pxt.String
    summary = chat_completions(
        messages=[{'role': 'user', 'content': text}],
        model='gpt-4o-mini',
    ).choices[0].message.content

# pxt schema update app.py docs -f
# Insert new or changed rows. Unchanged rows keep the cached summary.
# Do not wrap chat_completions in a Python for-loop over the table.
```

## Questions {#questions}

### Does every insert re-call the model for old rows? {#faq-1}

No. A computed column recomputes the row that changed and the columns that depend on it. Unchanged rows keep the cached result. A full refresh is a different operation you would have to ask for.

### Should I use Spark instead of a computed column? {#faq-2}

Use Spark (or Ray Data or Daft) when the work is a scan over a lake you do not want as a live application catalog. Use a computed column when new rows arrive and you need the cache to survive the next insert.

### How do I avoid paying twice for the same prompt? {#faq-3}

Store the output with the row and recompute only when the source text or the prompt expression changes. That is the cache. A loop with no store will pay twice after a restart.

### Will this page quote a cost for one million rows? {#faq-4}

No. Model vendors bill the first call; a cache avoids the second. Any dollar figure would be invented without your model, prompt, and token counts.

## Related

- [What is incremental computation?](https://www.pixeltable.com/learn/what-is-incremental-computation)
- [What is a computed column?](https://www.pixeltable.com/learn/what-is-a-computed-column)
- [What is AI ETL?](https://www.pixeltable.com/learn/what-is-ai-etl)
- [Ray Data vs Daft vs Spark](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark)
- [How do I add computed columns with LLM calls in Python?](https://www.pixeltable.com/learn/how-to/add-computed-columns-with-llm-calls)
- [How do I build an AI data pipeline in Python?](https://www.pixeltable.com/learn/how-to/build-an-ai-data-pipeline)
- [Pixeltable vs Daft](https://www.pixeltable.com/compare/pixeltable-vs-daft)
- [Computed columns](https://docs.pixeltable.com/datastore/computed-columns)