Ray Data vs Daft vs Spark

Three batch engines

Each product runs a job: read sources, map or SQL-transform, write a sink. None of them is an insert-triggered application schema.

Ray Data

Ray Data is a scalable data processing library for AI workloads on Ray. Official docs emphasize streaming execution between CPU preprocessing and GPU inference or training, map_batches, ActorPoolStrategy, and first-class integration with vLLM, PyTorch, Hugging Face, and TensorFlow. It reads Parquet, Lance, images, JSON, CSV, audio, video, and more. Install: pip install -U 'ray[data]'. Docs: https://docs.ray.io/en/latest/data/data.html

Daft

Daft, from Eventual, is a data engine for AI and multimodal workloads: tables, images, text, audio, video, and embeddings in one Python API, with a Rust engine and no JVM. Official docs describe reading object storage and lakes (Iceberg, Delta Lake), running GPU models, Python UDFs, and LLM APIs, then writing to search indexes, databases, warehouses, or training stores. Scale from a laptop to a cluster without changing the API. Docs: https://docs.daft.ai/en/stable/

Apache Spark

Apache Spark is a unified analytics engine for large-scale data processing. The current docs (Spark 4.2 line) list Spark SQL, DataFrames, pandas API on Spark, MLlib, GraphX, and Structured Streaming, plus Spark Connect for remote clients. APIs in Java, Scala, Python, and R. It runs on standalone clusters, YARN, and Kubernetes. Docs: https://spark.apache.org/docs/latest/

How to choose

Match the engine to the cluster you already run and the data types in the scan.

• Pick Spark when SQL, warehouses, and an existing Spark/YARN/Kubernetes lakehouse are the company standard.
• Pick Ray Data when the job is GPU batch inference or training ingest on a Ray cluster.
• Pick Daft when the dataframe should treat images, video, and embeddings as native types without a JVM.

Last reviewed

Facts on this page were checked against vendor documentation on 2026-10-28. Product surfaces move; treat version names and APIs as directional and follow the official links in Related.

Ray Data vs Daft vs Spark (last reviewed: Oct 2026)

Where Pixeltable fits

Use Pixeltable when the job is a live application schema: insert a row, computed columns and indexes update, HTTP can sit in the same file. It is not a Spark-shaped or Ray-shaped scan over petabytes. Many teams run both: Daft or Ray Data or Spark for a backfill over a lake, Pixeltable for the app that serves new rows.

import pixeltable as pxt
from pixeltable.functions.openai import chat_completions

TableModel = pxt.model_base()

class Docs(TableModel, name='docs'):
    text: pxt.String
    summary = chat_completions(
        messages=[{'role': 'user', 'content': text}],
        model='gpt-4o-mini',
    ).choices[0].message.content

# pxt schema update app.py docs -f
# Insert recomputes that row. Unchanged rows keep the cached summary.

Questions