Pixeltable 0.6.5 adds export_iceberg(): stream a table or query result into an Apache Iceberg table. Process multimodal data in declarative pipelines, then hand curated tabular columns to Spark, DuckDB, Snowflake, or any Iceberg-compatible analytics stack.
Why Iceberg export?#
Pixeltable is your AI data infrastructure layer—storage, computed columns, embeddings, versioning, and lineage for multimodal workflows. Analytics teams often need the same curated outputs in a lakehouse format for SQL dashboards, feature stores, or warehouse joins.
Pixeltable already supports several export paths:
export_sql()— push rows to PostgreSQL, SQLite, Snowflake, and other OLTP databasesexport_lancedb()— vector-focused analytics with LanceDBexport_parquet()/export_csv()— file-based handoff
Iceberg fills the gap when you need open table format semantics—ACID commits, schema evolution, time travel, and catalog-backed tables shared across engines. Export after your Pixeltable pipeline produces the columns downstream teams actually want: summaries, labels, scores, structured JSON fields—not raw video blobs.
When to use which export#
| Export | Best for |
|---|---|
| Iceberg | Lakehouse analytics, Spark/DuckDB/Trino, catalog-managed tables |
| SQL | Operational serving DB, app-facing replicas |
| LanceDB | Vector similarity at analytics scale |
| Parquet / CSV | Ad hoc files, one-off shares |
Getting started#
export_iceberg() streams data via PyArrow RecordBatches. Control batch size with batch_size_bytes (default 128 MB) for memory-efficient exports of large tables.
Basic export#
The table_name argument is a fully qualified Iceberg identifier (e.g. 'namespace.table'). If the namespace does not exist, it is created. Use if_exists='append' when the destination table already exists and schemas are compatible.
Export a query, not just a table#
Pass any Pixeltable Query—filters, projections, and joins applied before export:
Schema overrides#
By default, Pixeltable maps column types to Arrow types automatically. Pass schema_overrides when you need explicit PyArrow types for specific columns:
Multimodal pipeline pattern#
A typical flow:
- Ingest documents, images, or video into Pixeltable tables
- Run computed columns—transcription, summarization, classification, embedding
- Export structured columns (text, labels, scores, JSON metadata) to Iceberg
- Query in DuckDB, Spark, or Snowflake for BI and downstream ML features
Media files stay in Pixeltable's storage layer (or your configured S3/R2 destination); Iceberg carries the derived tabular artifacts your warehouse expects. That split matches how teams already use cloud blob storage for raw assets and warehouses for analytics tables.
Documentation note#
A dedicated Iceberg cookbook is in progress. Until then, refer to the I/O SDK reference for parameter details and the changelog for release notes.
