---
title: "Ray Data vs Daft vs Spark"
description: "Ray Data vs Daft vs Spark: streaming AI datasets, a multimodal dataframe, and a unified analytics engine. Last reviewed October 2026; vendor docs linked."
url: "https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark"
updated: "2026-10-28"
vertical: "Orchestration"
doc: "https://docs.pixeltable.com/datastore/computed-columns"
---

# Ray Data vs Daft vs Spark

Ray Data, Daft, and Apache Spark are engines that load, transform, and write large datasets as a job. They are dataframes or dataset APIs, not application catalogs. Pick among them for batch inference, preprocessing, and lake scans. A live table that recomputes one inserted row is a different job.

Updated: 2026-10-28
Part of [What is a declarative pipeline?](https://www.pixeltable.com/learn/what-is-a-declarative-pipeline).

## On this page

- [Three batch engines](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#what-it-is)
- [Ray Data](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#ray-data)
- [Daft](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#daft)
- [Apache Spark](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#spark)
- [How to choose](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#how-to-choose)
- [Last reviewed](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#review)
- [Comparison](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/ray-data-vs-daft-vs-spark#questions)

## Three batch engines {#what-it-is}

Each product runs a job: read sources, map or SQL-transform, write a sink. None of them is an insert-triggered application schema.

## Ray Data {#ray-data}

Ray Data is a scalable data processing library for AI workloads on Ray. Official docs emphasize streaming execution between CPU preprocessing and GPU inference or training, map_batches, ActorPoolStrategy, and first-class integration with vLLM, PyTorch, Hugging Face, and TensorFlow. It reads Parquet, Lance, images, JSON, CSV, audio, video, and more. Install: pip install -U 'ray[data]'. Docs: https://docs.ray.io/en/latest/data/data.html

## Daft {#daft}

Daft, from Eventual, is a data engine for AI and multimodal workloads: tables, images, text, audio, video, and embeddings in one Python API, with a Rust engine and no JVM. Official docs describe reading object storage and lakes (Iceberg, Delta Lake), running GPU models, Python UDFs, and LLM APIs, then writing to search indexes, databases, warehouses, or training stores. Scale from a laptop to a cluster without changing the API. Docs: https://docs.daft.ai/en/stable/

## Apache Spark {#spark}

Apache Spark is a unified analytics engine for large-scale data processing. The current docs (Spark 4.2 line) list Spark SQL, DataFrames, pandas API on Spark, MLlib, GraphX, and Structured Streaming, plus Spark Connect for remote clients. APIs in Java, Scala, Python, and R. It runs on standalone clusters, YARN, and Kubernetes. Docs: https://spark.apache.org/docs/latest/

## How to choose {#how-to-choose}

Match the engine to the cluster you already run and the data types in the scan.

- Pick Spark when SQL, warehouses, and an existing Spark/YARN/Kubernetes lakehouse are the company standard.
- Pick Ray Data when the job is GPU batch inference or training ingest on a Ray cluster.
- Pick Daft when the dataframe should treat images, video, and embeddings as native types without a JVM.

## Last reviewed {#review}

Facts on this page were checked against vendor documentation on 2026-10-28. Product surfaces move; treat version names and APIs as directional and follow the official links in Related.

## Ray Data vs Daft vs Spark (last reviewed: Oct 2026) {#comparison}

|  | Ray Data | Daft | Apache Spark |
| --- | --- | --- | --- |
| What it is | Streaming dataset library for AI on Ray | Multimodal dataframe (Python, Rust engine) | Unified analytics engine (SQL, DataFrames, streaming) |
| Execution | Streaming blocks, map_batches, actors, GPUs | Load, transform, write; laptop to cluster | DAG jobs, Spark SQL, Structured Streaming |
| Media types | Images, audio, video among other ML formats | Native image, audio, video, embedding types | Rows and files; media is bytes or paths unless you add UDFs |
| Runtime | Ray cluster | No JVM; Rust engine | JVM; Spark Connect for remote clients |
| Durable catalog | A job; persistence is the sink you choose | A job; persistence is the sink you choose | Tables in a warehouse or lake you configure |
| Official docs | https://docs.ray.io/en/latest/data/data.html | https://docs.daft.ai/en/stable/ | https://spark.apache.org/docs/latest/ |

## Where Pixeltable fits {#where-pixeltable-fits}

Use Pixeltable when the job is a live application schema: insert a row, computed columns and indexes update, HTTP can sit in the same file. It is not a Spark-shaped or Ray-shaped scan over petabytes. Many teams run both: Daft or Ray Data or Spark for a backfill over a lake, Pixeltable for the app that serves new rows.

```python
import pixeltable as pxt
from pixeltable.functions.openai import chat_completions

TableModel = pxt.model_base()

class Docs(TableModel, name='docs'):
    text: pxt.String
    summary = chat_completions(
        messages=[{'role': 'user', 'content': text}],
        model='gpt-4o-mini',
    ).choices[0].message.content

# pxt schema update app.py docs -f
# Insert recomputes that row. Unchanged rows keep the cached summary.
```

## Questions {#questions}

### Is Daft a Spark replacement? {#faq-1}

Daft is a dataframe for multimodal AI workloads with a Python API and a Rust engine. Spark is a unified analytics engine with SQL, streaming, and a huge lakehouse ecosystem. Replace Spark with Daft only when the job matches Daft's multimodal batch shape and you do not need Spark SQL as the company standard.

### When is Ray Data the right engine? {#faq-2}

When you already run Ray and the bottleneck is streaming CPU preprocess into GPU batch inference or training ingest. Official overview: https://docs.ray.io/en/latest/data/data.html

### Do any of these replace a vector database? {#faq-3}

No. They can write embeddings to a sink (a vector DB, a lake, a warehouse). The index is still a separate product unless you add one.

### Where does an application schema fit? {#faq-4}

Beside them. Batch engines scan and write. An application schema stores rows and recomputes dependents of a change. Pixeltable is that second job; see Pixeltable vs Daft.

## Related

- [Pixeltable vs Daft](https://www.pixeltable.com/compare/pixeltable-vs-daft)
- [Best multimodal data platforms](https://www.pixeltable.com/learn/best-multimodal-data-platforms)
- [What is a declarative pipeline?](https://www.pixeltable.com/learn/what-is-a-declarative-pipeline)
- [What is AI ETL?](https://www.pixeltable.com/learn/what-is-ai-etl)
- [How to run LLM calls over millions of rows](https://www.pixeltable.com/learn/how-to-run-llm-calls-over-millions-of-rows)
- [Ray Data documentation](https://docs.ray.io/en/latest/data/data.html)
- [Daft documentation](https://docs.daft.ai/en/stable/)
- [Apache Spark documentation](https://spark.apache.org/docs/latest/)