How to run LLM calls over millions of rows

Short answer

Do not wrap the model client in a for-loop you restart. Persist each result next to its source row. On the next run, skip rows whose inputs have not changed. A batch engine (Spark, Ray Data, Daft) is the right tool for a one-shot scan over a lake. A computed column is the right tool for a live table: insert recomputes that row; unchanged rows keep the cache. This page quotes no tokens-per-second and no dollar totals.

Why a restartable loop fails

A script that embeds or summarizes every path in a folder looks fine until it dies at row 400,000 and you run it again. Without a stored result keyed by content (or by a declared dependency), you pay the model vendor a second time for work you already bought. Rate limits and partial failure make that worse. The fix is a cache with lineage, not a longer sleep().

A lake scan versus a live table

Pick the engine from the data shape.

• Lake scan: Ray Data streams CPU preprocess into GPU or API inference (https://docs.ray.io/en/latest/data/data.html). Daft is a multimodal dataframe (https://docs.daft.ai/en/stable/). Spark is a unified analytics engine (https://spark.apache.org/docs/latest/). Write the sink so a rerun can skip completed keys if you add that logic.
• Live table: declare the LLM call as a column. Insert or edit a row and that expression runs. Unchanged rows stay cached. Editing a prompt recomputes the columns that depend on it. Docs: computed columns in Related.

How to choose

This is not a bake-off. It is a fit check.

• Historical dump in a lake, run once or on a schedule: Spark, Ray Data, or Daft.
• New files arriving as application rows: a computed column.
• Both: backfill with a batch engine, then serve new rows from a table.

Last reviewed

Facts on this page were checked against vendor documentation on 2026-10-28. Product surfaces move; treat version names and APIs as directional and follow the official links in Related.

Ways to run LLM calls over a large table (last reviewed: Oct 2026)

Where Pixeltable fits

Use Pixeltable when the millions of rows are a live table: assign chat_completions on the class so each new or changed row runs, then caches. It is not a Spark-shaped scan over a lake. For that backfill, keep Ray Data, Daft, or Spark and load the result. How-to: add computed columns with LLM calls in Python. What is incremental computation? covers the cache rule. This page does not publish a tokens-per-second number.

import pixeltable as pxt
from pixeltable.functions.openai import chat_completions

TableModel = pxt.model_base()

class Docs(TableModel, name='docs'):
    text: pxt.String
    source: pxt.String
    summary = chat_completions(
        messages=[{'role': 'user', 'content': text}],
        model='gpt-4o-mini',
    ).choices[0].message.content

# pxt schema update app.py docs -f
# Insert new or changed rows. Unchanged rows keep the cached summary.
# Do not wrap chat_completions in a Python for-loop over the table.

Questions