Autodistill Segment Anything 3 Module

This repository contains the code supporting the Segment Anything 3 base model for use with Autodistill.

Segment Anything 3 (SAM3), developed by Meta Research, is a state-of-the-art zero-shot image segmentation model. You can prompt SAM3 with image regions and open text prompts.

Autodistill currently supports using text prompts to auto-label images for use in fine-tuning smaller vision models, such as an RF-DETR object detection model.

The Autodistill SAM 3 package only works on a GPU.

Read the full Autodistill documentation.

Installation

To use SAM3 with Autodistill, you need to install the following dependency:

pip3 install autodistill-sam3

You will also need to set a Roboflow API key in your environment:

export ROBOFLOW_API_KEY=YOUR_KEY

This key will allow you to download the model weights required to use this package.

Learn how to retrieve your Roboflow API key.

Quickstart

from autodistill_sam3 import SegmentAnything3
from autodistill.detection import CaptionOntology
from autodistill.helpers import load_image

import supervision as sv

# define an ontology to map class names to our SAM 3 prompt
# the ontology dictionary has the format {caption: class}
# where caption is the prompt sent to the base model, and class is the label that will
# be saved for that caption in the generated annotations
# then, load the model
base_model = SegmentAnything3(
    ontology=CaptionOntology(
        {
            "fruit": "fruit",
            "leaf": "leaf"
        }
    )
)

# run inference on a single iamge
detections = base_model.predict("image.jpg")

image = load_image("image.jpg", return_format="cv2")

# visualise results
label_annotator = sv.LabelAnnotator(text_position=sv.Position.CENTER)
mask_annotator = sv.MaskAnnotator()
annotated_frame = mask_annotator.annotate(
    scene=image.copy(),
    detections=detections
)
annotated_frame = label_annotator.annotate(
    scene=annotated_frame,
    detections=detections,
    labels=[base_model.ontology.classes()[class_id] for class_id in detections.class_id]
)
sv.plot_image(annotated_frame)

# label a folder of images
base_model.label("./images_to_label", extension=".jpeg")

Here is an example output:

License

SAM 3 is licensed under a custom SAM license. See the SAM 3 license in the official facebookresearch/sam3 repository.

🏆 Contributing

We love your input! Please see the core Autodistill contributing guide to get started. Thank you 🙏 to all our contributors!