Summary: Pixeltable now supports Meta’s Segment Anything Model 3 (SAM 3) via Hugging Face. The sam3_for_segmentation() UDF takes an image plus a concept prompt—a short text noun phrase, bounding boxes, or both—and returns a SamForSegmentationResponse with per-instance scores, boxes, and binary masks. Declare segmentation once as a computed column (or run it inline with select); Pixeltable handles execution, persistence, and incremental updates. This post follows our promptable segmentation cookbook.
What Is Promptable Concept Segmentation?#
Classical detectors and segmenters emit a fixed taxonomy (“person”, “car”, “dog”). SAM 3 instead does promptable concept segmentation: you describe what you want—“zebra”, “solar panel”, “aircraft”—or point at a region with a box, and the model returns one instance mask for every match.
| Approach | Output | Prompting |
|---|---|---|
| Fixed-label detection (e.g. YOLOX) | Boxes / classes from a closed set | None at inference (train-time labels) |
| Panoptic / semantic segmentation | Dense class map | Closed vocabulary |
| SAM 3 promptable segmentation | Per-instance score, box, and mask | Text, boxes, or both |
That open vocabulary is why SAM 3 fits Pixeltable’s declarative model so well: the concept you care about is just another input column (or a literal), and the masks become first-class multimodal data you can version, query, and chain into downstream classifiers.
Setup: Pixeltable, Transformers, and Hugging Face Auth#
facebook/sam3 is a gated model (the default model_id for these UDFs). Request access on Hugging Face, then authenticate locally. Pixeltable requires transformers>=5.6.0—earlier releases have a bug that blocks SAM v3 automatic segmentation.
Tests and CI notebooks that call SAM 3 skip automatically when no HF token is configured—same pattern as other gated Hub models.
Load a Table of Images#
Start with a small demo table. Every example below runs segmentation on the fly with select; you can also persist results with add_computed_column when you want incremental storage.
Segment from a Text Prompt#
Pass a short noun phrase as text. SAM 3 returns one binary mask per matching instance. Overlay masks with overlay_segmentation(), which now accepts a (num_instances, H, W) boolean mask stack.
The return type is SamForSegmentationResponse: scores shape (num_instances,), boxes shape (num_instances, 4) in [x1, y1, x2, y2] pixel coordinates, and masks shape (num_instances, H, W). Access fields with attribute syntax (e.g. .masks) or store the whole object in a computed column.
Segment Many Instances Automatically#
When you do not have a concept phrase—just “segment everything interesting”—use sam_automatic_mask_generation(). It returns unlabeled instance masks you can still overlay and persist.
Segment a Region with Bounding Boxes#
Skip the text prompt and pass boxes in [x1, y1, x2, y2] pixel coordinates via input_boxes to segment whatever concept that region contains. Combine text and boxes when both are available; optional input_boxes_labels marks each box as positive (1) or negative (0) to include or exclude a region.
Segment Objects Across a Video#
Segmentation is not limited to stills. Clip a video, then use the sam3_for_video_segmentation iterator with one or more concept labels. Unlike a plain frame_iterator, it carries tracker memory across frames so each object keeps a stable object_ids entry as it moves or is occluded. Overlay masks and rebuild a preview with make_video.
For longer pipelines—keyframes, detections, embeddings—pair this with our guides on YOLOX video object detection and video intelligence pipelines.
Industry Recipes#
Defense and intelligence: count aircraft in a satellite image#
Geospatial frames often contain hundreds of assets. Tile the image, segment each tile with a concept like aircraft, then stitch overlays back to the original resolution.
Medical imaging: segment lungs in a chest X-ray#
SAM 3 is trained primarily on natural RGB images. Grayscale modalities (X-ray, CT, MRI) sit outside that distribution, so free-form text is less reliable than on photos. High-contrast structures like lungs in a PA chest X-ray are often still workable; treat medical results as assistive, not diagnostic, and validate clinically.
Energy infrastructure: inspect a solar array#
Drone inspection of utility-scale solar is often bottlenecked by finding every panel before defect classification. A single concept prompt returns per-panel masks you can persist and chain into quality models.
Why Run SAM 3 Inside Pixeltable?#
- Declarative computed columns: Define segmentation once; inserts and updates recompute only what changed.
- Typed multimodal outputs: Scores, boxes, and masks stay structured—not one-off NumPy dumps in ad-hoc scripts.
- Visualization built in:
overlay_segmentationandbboxes_drawwork on instance mask stacks and video frames. - Composable pipelines: Chain SAM 3 into detectors, embeddings, RAG, or annotation exports the same way you chain YOLOX or Whisper.
- Same catalog for images and video: Tables, views, iterators, and versioning already cover multimodal media.
If you are still wiring FiftyOne, Label Studio, and custom mask scripts by hand, see unifying CV curation and annotation with Pixeltable and Rerun vs Pixeltable for computer vision.
FAQ#
Do I need a Hugging Face token?#
Yes. facebook/sam3 is gated. Accept the model license on the Hub and set HF_TOKEN or run huggingface-cli login. Without a token, SAM 3 calls fail (tests skip).
Which transformers version do I need?#
Use transformers ≥ 5.6.0. Earlier releases have a bug that prevents SAM v3 automatic segmentation.
When should I use text vs bounding boxes?#
Use text for open-vocabulary “find every X” passes. Use boxes when a region is already known (from a detector, click, or crop). Combine both when you want concept filtering inside a region.
What if no instances match?#
You get an empty mask stack. overlay_segmentation handles empty stacks safely—useful for batch jobs where many frames have zero hits.
Should I use select or add_computed_column?#
Use inline select for exploration. Use add_computed_column when masks should persist, version, and recompute incrementally as new images arrive.
See Also#
- Cookbook: Promptable segmentation with SAM 3
- Notebook on GitHub
- PR #1336 — SAM 3 Hugging Face support
- Docs:
sam3_for_segmentation - Docs:
sam_automatic_mask_generation - Docs:
overlay_segmentation - Hugging Face:
facebook/sam3 - Transformers SAM 3 model docs
- YOLOX object detection for video
- Automate CV data with Pixeltable
- Rerun vs Pixeltable for computer vision
- S3 Files, Hugging Face Buckets, and Pixeltable storage
