Split data into multiple rows with iterators

Transform a single document, video, image, or audio file into multiple rows for granular processing.

Core Operations · Notebook · Updated

Colab runs it in your browser. If it calls a model provider, it asks for that provider's API key in a cell.

What it covers

  • Split documents into text chunks for RAG
  • Extract frames or segments from videos
  • Tile images for high-resolution analysis
  • Chunk audio files for transcription

What it uses

Install
terminal
pip install pixeltable spacy tiktoken click
Pixeltable calls
document.document_splitter()video.frame_iterator()video.video_splitter()string.string_splitter()image.tile_iterator()audio.audio_splitter()pxt.create_table()pxt.create_view()insert()sample()

A look at the code

data-split-rows.ipynb
from pixeltable.functions.video import frame_iterator
videos = pxt.create_table('split_demo/videos', {'video': pxt.Video})
videos.insert(
[
{
'video': 'https://github.com/pixeltable/pixeltable/raw/main/docs/resources/bangkok.mp4'
}
]
)

One cell of the notebook, copied as it is. Open it in Colab for the rest.

Browse all 103