storagemoderateOfficial

Amazon S3 Integration

Insert s3:// URIs into Pixeltable media columns, cache on access, write computed media to a bucket, and mint presigned HTTP URLs.

Get Started Docs

Challenge

Media already lives in S3. A pipeline that downloads the bucket into a working directory before it can run a model copies bytes you did not need yet. A pipeline that only stores the URI then cannot resize, transcribe, or sign a URL without a second client. Pixeltable media columns accept s3://, https://, and local paths. The integration is that URI as the stored value, on-demand cache under PIXELTABLE_HOME/file_cache, optional destinations so computed images land in a bucket, and pxtf.net.presigned_url when a browser needs HTTP. Hosted Cloud tables already write to pxt://org:db/buckets/home; this page is the bring-your-own S3 path.

Solution

Insert s3://bucket/key into a pxt.Video or pxt.Image column. Pixeltable stores the URI and downloads when a query or computed column reads the file. AWS credentials follow the usual chain: AWS_ACCESS_KEY_ID, ~/.aws/credentials, or an IAM role. Optional s3_profile in config.toml. For computed media, set PIXELTABLE_OUTPUT_MEDIA_DEST=s3://my-bucket/output/ before creating the table, or pxt.Column(..., destination=...) on that field. presigned_url(uri, expiration_seconds) turns a storage URI into a time-limited HTTP URL. HTTP URLs pass through unchanged. Cloud tables do not need this destination; they already use the managed home bucket.

Integration Steps

Step 1 of 4

Store the URI. The object is fetched and cached when a query or computed column needs the bytes.

import pixeltable as pxt

TableModel = pxt.model_base()

class Videos(TableModel, name='videos'):
    video: pxt.Video

# pxt schema update app.py s3_media -f
videos = pxt.get_table('s3_media.videos')
s3_prefix = 's3://multimedia-commons/'
videos.insert([
    {'video': s3_prefix + 'data/videos/mp4/ffe/ffb/ffeffbef41bbc269810b2a1a888de.mp4'},
])
videos.collect()

💡 The multimedia-commons prefix is the public bucket used in the docs. Private buckets need the AWS credential chain. The same insert shape works with https:// and local paths. Cache lives under ~/.pixeltable/file_cache/ by default.

Use arrow keys to navigate

Key Benefits

No bulk download before the first computed column runs

Same Image, Video, and Audio types for s3://, https://, and local paths

Computed thumbnails can land in a prefix you choose

presigned_url is a column, not a signing script

Cloud home bucket stays the default when dest is unset

Use Cases

•Transcribe or embed objects that already live in a bucket

•Write generated images to s3://.../generated/

•Serve private media to a player without exposing keys

•Point local Pixeltable at a Cloud home bucket with pxt://org:db/buckets/home

•GCS, R2, B2, or Tigris with the same destination fields

Technical Requirements

•Python 3.11+

•pip install -U 'pixeltable[serve]'

•AWS credentials with GetObject, and PutObject when dest is S3

•PIXELTABLE_OUTPUT_MEDIA_DEST set before table create when computed media should land in S3

Learn More

Load media from S3 Cloud Storage S3, Hugging Face, and Pixeltable storage

S3 Files, Hugging Face Buckets, and the Storage Problem Pixeltable Already Solved Build Multimodal AI Applications Faster with Pixeltable Infrastructure Your First Pixeltable Project: Build a Smart Image Organizer in 10 Minutes

Related Integrations

Hugging Face
Modality: Model API
Computed column: Hugging Face: Index images with CLIP
Docs
Langfuse
Modality: Monitoring
Computed column: Langfuse: Install extras and set Langfuse env
Docs
FiftyOne
Modality: Model API
Computed column: FiftyOne: Store stills and run ViT plus DETR
Docs

Ready to Integrate Amazon S3?

Get started with Pixeltable and Amazon S3 in minutes.

pip install pixeltable

View on GitHub Starter Kit Get started Try in Notebooks