---
title: "SAM 3 Promptable Segmentation in Pixeltable"
date: "2026-07-17"
author: "Pierre Brunelle"
tags:
  - SAM 3
  - Segment Anything Model 3
  - Promptable Segmentation
  - Hugging Face
  - Computer Vision
  - Instance Masks
  - Pixeltable
  - Multimodal AI
  - Video Segmentation
  - Open Vocabulary
description: "Open-vocab SAM 3 in Pixeltable: text or box prompts to typed scores, boxes, and masks as computed columns. Gated HF setup, video tracking, and industry recipes."
url: "https://www.pixeltable.com/blog/sam3-promptable-segmentation-pixeltable"
---

# SAM 3 Promptable Segmentation in Pixeltable

**Summary:** Pixeltable now supports [Meta’s Segment Anything Model 3 (SAM 3)](https://huggingface.co/facebook/sam3) via Hugging Face. The [`sam3_for_segmentation()`](https://docs.pixeltable.com/sdk/latest/huggingface#udf-sam3_for_segmentation) UDF takes an image plus a concept prompt—a short text noun phrase, bounding boxes, or both—and returns a `SamForSegmentationResponse` with per-instance `scores`, `boxes`, and binary `masks`. Declare segmentation once as a computed column (or run it inline with `select`); Pixeltable handles execution, persistence, and incremental updates. This post follows our [promptable segmentation cookbook](https://docs.pixeltable.com/howto/cookbooks/images/img-promptable-segmentation).

 
## What Is Promptable Concept Segmentation?

 
Classical detectors and segmenters emit a fixed taxonomy (“person”, “car”, “dog”). SAM 3 instead does *promptable concept segmentation*: you describe what you want—“zebra”, “solar panel”, “aircraft”—or point at a region with a box, and the model returns one instance mask for every match.

 
| Approach | Output | Prompting |
| --- | --- | --- |
| Fixed-label detection (e.g. YOLOX) | Boxes / classes from a closed set | None at inference (train-time labels) |
| Panoptic / semantic segmentation | Dense class map | Closed vocabulary |
| SAM 3 promptable segmentation | Per-instance score, box, and mask | Text, boxes, or both |

 
That open vocabulary is why SAM 3 fits Pixeltable’s declarative model so well: the concept you care about is just another input column (or a literal), and the masks become first-class multimodal data you can version, query, and chain into downstream classifiers.

 
## Setup: Pixeltable, Transformers, and Hugging Face Auth

 
`facebook/sam3` is a **gated** model (the default `model_id` for these UDFs). Request access on Hugging Face, then authenticate locally. Pixeltable requires `transformers>=5.6.0`—earlier releases have a bug that blocks SAM v3 automatic segmentation.

 
```bash

pip install -qU 'pixeltable' 'torch' 'transformers>=5.6.0'

# Authenticate for the gated facebook/sam3 weights
huggingface-cli login
# or: export HF_TOKEN=hf_...
 
```

 
> 
 
Tests and CI notebooks that call SAM 3 skip automatically when no HF token is configured—same pattern as other gated Hub models.

 

 
## Load a Table of Images

 
Start with a small demo table. Every example below runs segmentation on the fly with `select`; you can also persist results with `add_computed_column` when you want incremental storage.

 
```python

import pixeltable as pxt
from pixeltable.functions.huggingface import (
 sam3_for_segmentation,
 sam_automatic_mask_generation,
)
from pixeltable.functions.vision import overlay_segmentation

pxt.drop_dir('sam3_demo', force=True)
pxt.create_dir('sam3_demo')

base_url = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images'

images = pxt.create_table(
 'sam3_demo/images', {'name': pxt.String, 'image': pxt.Image}
)
images.insert([
 {'name': 'zebra', 'image': f'{base_url}/000000000034.jpg'},
 {'name': 'bento', 'image': f'{base_url}/000000000009.jpg'},
 {'name': 'boneyard', 'image': f'{base_url}/amarg-boneyard.jpg'},
 {'name': 'chest_xray', 'image': f'{base_url}/chest-xray.jpg'},
 {'name': 'solar_farm', 'image': f'{base_url}/solar-farm.jpg'},
])
 
```

 
## Segment from a Text Prompt

 
Pass a short noun phrase as `text`. SAM 3 returns one binary mask per matching instance. Overlay masks with [`overlay_segmentation()`](https://docs.pixeltable.com/sdk/latest/vision#udf-overlay_segmentation), which now accepts a `(num_instances, H, W)` boolean mask stack.

 
```python

images.select(
 images.image,
 segmented=overlay_segmentation(
 images.image,
 sam3_for_segmentation(images.image, text='zebra').masks,
 draw_contours=True,
 ),
).where(images.name == 'zebra').collect()
 
```

 
The return type is `SamForSegmentationResponse`: `scores` shape `(num_instances,)`, `boxes` shape `(num_instances, 4)` in `[x1, y1, x2, y2]` pixel coordinates, and `masks` shape `(num_instances, H, W)`. Access fields with attribute syntax (e.g. `.masks`) or store the whole object in a computed column.

 
## Segment Many Instances Automatically

 
When you do not have a concept phrase—just “segment everything interesting”—use `sam_automatic_mask_generation()`. It returns unlabeled instance masks you can still overlay and persist.

 
```python

images.select(
 segmented=overlay_segmentation(
 images.image,
 sam_automatic_mask_generation(images.image).masks,
 draw_contours=True,
 )
).where(images.name == 'bento').collect()
 
```

 
## Segment a Region with Bounding Boxes

 
Skip the text prompt and pass boxes in `[x1, y1, x2, y2]` pixel coordinates via `input_boxes` to segment whatever concept that region contains. Combine text and boxes when both are available; optional `input_boxes_labels` marks each box as positive (`1`) or negative (`0`) to include or exclude a region.

 
```python

images.select(
 images.image,
 segmented=overlay_segmentation(
 images.image,
 sam3_for_segmentation(
 images.image, input_boxes=[[10, 30, 470, 300]]
 ).masks,
 draw_contours=True,
 ),
).where(images.name == 'zebra').collect()
 
```

 
## Segment Objects Across a Video

 
Segmentation is not limited to stills. Clip a video, then use the `sam3_for_video_segmentation` iterator with one or more concept labels. Unlike a plain `frame_iterator`, it carries tracker memory across frames so each object keeps a stable `object_ids` entry as it moves or is occluded. Overlay masks and rebuild a preview with `make_video`.

 
```python

from pixeltable.functions.huggingface import sam3_for_video_segmentation
from pixeltable.functions.video import clip, make_video
from pixeltable.functions.vision import bboxes_draw

pxt.drop_dir('sam3_video', force=True)
pxt.create_dir('sam3_video')

bangkok = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/bangkok.mp4'

videos = pxt.create_table('sam3_video/videos', {'video': pxt.Video})
videos.add_computed_column(
 clipped_video=clip(videos.video, start_time=0, end_time=2)
)
videos.insert([{'video': bangkok}])

frames = pxt.create_view(
 'sam3_video/frames',
 videos,
 iterator=sam3_for_video_segmentation(
 videos.clipped_video, text=['car', 'motorcycle'], fps=5
 ),
)

# Rebuild at the same fps; SAM 3 looks ahead a few frames for tracking
frames.group_by(videos).select(
 segmented_video=make_video(
 frames.pos,
 bboxes_draw(
 img=overlay_segmentation(
 frames.frame, frames.masks, draw_contours=True
 ),
 boxes=frames.boxes.to_list(),
 labels=frames.labels.to_list(),
 ),
 fps=5,
 )
).collect()
 
```

 
For longer pipelines—keyframes, detections, embeddings—pair this with our guides on [YOLOX video object detection](/blog/object-detection-videos-yolox) and [video intelligence pipelines](/blog/video-intelligence-pipeline-tutorial).

 
## Industry Recipes

 
### Defense and intelligence: count aircraft in a satellite image

 
Geospatial frames often contain hundreds of assets. Tile the image, segment each tile with a concept like `aircraft`, then stitch overlays back to the original resolution.

 
```python

from pixeltable.functions.image import stitch_tiles, tile_iterator

tiles = pxt.create_view(
 'sam3_demo/tiles',
 images.where(images.name == 'boneyard'),
 iterator=tile_iterator(
 images.image, tile_size=(1006, 1006), overlap=(45, 45)
 ),
)

tiles.add_computed_column(
 segmented_tile=overlay_segmentation(
 tiles.tile,
 sam3_for_segmentation(tiles.tile, text='aircraft').masks,
 draw_contours=True,
 )
)

tiles.group_by(tiles.image).select(
 stitched=stitch_tiles(
 tiles.segmented_tile,
 tiles.tile_box,
 tiles.image.width,
 tiles.image.height,
 )
).collect()
 
```

 
### Medical imaging: segment lungs in a chest X-ray

 
SAM 3 is trained primarily on natural RGB images. Grayscale modalities (X-ray, CT, MRI) sit outside that distribution, so free-form text is less reliable than on photos. High-contrast structures like lungs in a PA chest X-ray are often still workable; treat medical results as assistive, not diagnostic, and validate clinically.

 
```python

images.select(
 images.image,
 segmented=overlay_segmentation(
 images.image,
 sam3_for_segmentation(images.image, text='lung').masks,
 draw_contours=True,
 ),
).where(images.name == 'chest_xray').collect()
 
```

 
### Energy infrastructure: inspect a solar array

 
Drone inspection of utility-scale solar is often bottlenecked by finding every panel before defect classification. A single concept prompt returns per-panel masks you can persist and chain into quality models.

 
```python

images.select(
 images.image,
 segmented=overlay_segmentation(
 images.image,
 sam3_for_segmentation(images.image, text='solar panel').masks,
 draw_contours=True,
 ),
).where(images.name == 'solar_farm').collect()
 
```

 
## Why Run SAM 3 Inside Pixeltable?

 

 - **Declarative computed columns:** Define segmentation once; inserts and updates recompute only what changed.

 - **Typed multimodal outputs:** Scores, boxes, and masks stay structured—not one-off NumPy dumps in ad-hoc scripts.

 - **Visualization built in:** `overlay_segmentation` and `bboxes_draw` work on instance mask stacks and video frames.

 - **Composable pipelines:** Chain SAM 3 into detectors, embeddings, RAG, or annotation exports the same way you chain YOLOX or Whisper.

 - **Same catalog for images and video:** Tables, views, iterators, and versioning already cover multimodal media.

 

 
If you are still wiring FiftyOne, Label Studio, and custom mask scripts by hand, see [unifying CV curation and annotation with Pixeltable](/blog/automate-cv-data-pixeltable) and [Rerun vs Pixeltable for computer vision](/blog/rerun-vs-pixeltable-computer-vision).

 
## FAQ

 
### Do I need a Hugging Face token?

 
Yes. `facebook/sam3` is gated. Accept the model license on the Hub and set `HF_TOKEN` or run `huggingface-cli login`. Without a token, SAM 3 calls fail (tests skip).

 
### Which transformers version do I need?

 
Use **transformers ≥ 5.6.0**. Earlier releases have a bug that prevents SAM v3 automatic segmentation.

 
### When should I use text vs bounding boxes?

 
Use text for open-vocabulary “find every X” passes. Use boxes when a region is already known (from a detector, click, or crop). Combine both when you want concept filtering inside a region.

 
### What if no instances match?

 
You get an empty mask stack. `overlay_segmentation` handles empty stacks safely—useful for batch jobs where many frames have zero hits.

 
### Should I use select or add_computed_column?

 
Use inline `select` for exploration. Use `add_computed_column` when masks should persist, version, and recompute incrementally as new images arrive.

 
## See Also

 

 - [Cookbook: Promptable segmentation with SAM 3](https://docs.pixeltable.com/howto/cookbooks/images/img-promptable-segmentation)

 - [Notebook on GitHub](https://github.com/pixeltable/pixeltable/blob/main/docs/release/howto/cookbooks/images/img-promptable-segmentation.ipynb)

 - [PR #1336 — SAM 3 Hugging Face support](https://github.com/pixeltable/pixeltable/pull/1336)

 - [Docs: `sam3_for_segmentation`](https://docs.pixeltable.com/sdk/latest/huggingface#udf-sam3_for_segmentation)

 - [Docs: `sam_automatic_mask_generation`](https://docs.pixeltable.com/sdk/latest/huggingface#udf-sam_automatic_mask_generation)

 - [Docs: `overlay_segmentation`](https://docs.pixeltable.com/sdk/latest/vision#udf-overlay_segmentation)

 - [Hugging Face: `facebook/sam3`](https://huggingface.co/facebook/sam3)

 - [Transformers SAM 3 model docs](https://huggingface.co/docs/transformers/model_doc/sam3)

 - [YOLOX object detection for video](/blog/object-detection-videos-yolox)

 - [Automate CV data with Pixeltable](/blog/automate-cv-data-pixeltable)

 - [Rerun vs Pixeltable for computer vision](/blog/rerun-vs-pixeltable-computer-vision)

 - [S3 Files, Hugging Face Buckets, and Pixeltable storage](/blog/s3-files-huggingface-buckets-pixeltable-storage)