Segmenting with prompts

Segment Anything Model 3.1 (SAM 3.1) segments by concept. Give it a short noun phrase naming one thing — "yellow school bus", "glasses", "red hat" — and it returns a box and a mask for every match. It reads the phrase as a concept to locate, not as an instruction to follow, so you name the object rather than writing a command or a question.

SAM 3.1 on Meta Model API takes text prompts. Name a concept in input_text and the model detects every instance. The open-weight research model also accepts visual prompts — input boxes, points, and clicks that mark where to look. The Model API works from the text concept alone; the boxes it returns are outputs (located regions), not inputs.

One Responses API call on sam-3.1 carries the prompt and the media, an image or a video, and streams a box and a mask per match. The prompt is the part that decides what you get back, so start there.

Write the concept prompt

The prompt goes in an input_text part as a short noun phrase. SAM 3.1 matches the concept you name, so concrete wording controls the result.

  • Name one concept with a noun phrase: "yellow school bus", "red hat", or "glasses". Commands ("segment the bus") and questions ("where is the bus?") are unreliable and often return nothing. Name the object; don't instruct the model.
  • One concept per request: there is no multi-concept syntax. Comma, semicolon, and plus lists ("bus, bicycle") return zero matches; an "and", a newline, or a second input_text part segments only one of the concepts. To segment several concepts, send one request each.
  • Add attributes to narrow: "square" matches every square; "red square" matches only the red ones. Color, type, and other concrete attributes filter predictably, down to zero when nothing matches.
  • Formatting is ignored: plurals ("red squares"), case, trailing punctuation, and extra whitespace all return the same result. Styling carries no meaning.
  • Skip negation, boolean OR, and spatial phrasing: "person not wearing a hat" and "cat or dog" return inconsistent results, and "person on the left" is only partially honored. Segment the plain concept, then filter by position or attribute in your app using the returned box coordinates.
  • Keep it short: a concise noun phrase works as well as a long description. Extra words give no benefit, and padding a prompt with filler degrades matching.

SAM 3.1 locates the objects you name; it doesn't answer questions or reason about a scene. For scene understanding, use the Muse Spark model family, or a combination: use Muse Spark to caption, answer questions, or identify scene activity via image understanding or video understanding, then pass the concept to SAM 3.1 for the segmentation work.

Attributes change the match set predictably. On an image of six squares, "square" returns six matches, "red square" returns three, and "yellow square" returns none. Each match comes back as one box and one mask; see Segment an image for the wire records and read segmentation output to parse them.

A concept that isn't present returns zero matches: the stream completes with response.completed and an empty output_text. Treat it as a normal result, not an error.

Build the request

A request pairs the concept prompt with one media part inside a single user message. Send the phrase in an input_text part and the media in an input_image or input_video part:

  • input_text: the concept prompt, a short noun phrase for one object (see Write the concept prompt).
  • input_image: a still as image_url.
  • input_video: a clip as video_url, or as file_id for a video uploaded through the Files API.
  • image_url / video_url: a public URL or a base64 data URL (data:image/png;base64,...).

The same model serves both images and video. Swap the media part and keep the rest of the request identical. One concept per request: to segment several object types, send several requests.

Segment an image

Send an input_image part and stream the response to response.completed:

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with client.responses.stream(
    model="sam-3.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "glasses",
                },
                {
                    "type": "input_image",
                    "image_url": "https://example.com/photo.png",
                },
            ],
        },
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
print()

An image returns on a single line: one record per match, comma-separated, each record a box then a mask. Here w and h are the source image dimensions, repeated on every record:

text
<0f>0<|box;x1=181;y1=365;x2=556;y2=486;w=1516;h=1600|><|mask;x=0;y=0;data=122,376,~...|>

<0f> is the frame, the leading integer is the object ordinal, then one box then one mask; ~ marks a lossless mask payload. Full grammar in reading segmentation output.

Segment a video

Swap in an input_video part with a video_url. Start with a hosted clip to tune the prompt before uploading anything:

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with client.responses.stream(
    model="sam-3.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "red bicycle",
                },
                {
                    "type": "input_video",
                    "video_url": "https://example.com/clip.mp4",
                },
            ],
        },
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
print()

A video returns one line per frame, one response.output_text.delta per frame, frame markers <0f>, <1f>, and up. Frame 0 of a clip with two matches:

text
<0f>0<|box;x1=211;y1=228;x2=270;y2=254;w=320;h=334|><|mask;x=0;y=0;data=27,60,~...|>,1<|box;x1=155;y1=228;x2=202;y2=254;w=320;h=334|><|mask;x=0;y=0;data=27,48,~...|>

Upload once, reference by ID

For a clip you segment more than once, upload it with POST /v1/files and set purpose to user_data. Upload is synchronous: a 200 means the file is ready to use in the next request.

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with open("/path/to/clip.mp4", "rb") as f:
    file = client.files.create(
        purpose="user_data",
        file=f,
    )
​
print(file.model_dump_json(indent=2))

Example upload response

json
{
  "id": "file-842549258569145",
  "object": "file",
  "bytes": 245832,
  "created_at": 1714502400,
  "filename": "clip.mp4",
  "purpose": "user_data",
  "status": "uploaded"
}

Pass the returned ID in the file_id field of the input_video part:

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with client.responses.stream(
    model="sam-3.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "delivery van",
                },
                {
                    "type": "input_video",
                    "file_id": "file-abc123",
                },
            ],
        },
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
print()

See file handling for expiration, listing, and deletion.

MP4 faststart. An MP4 stores an index (the moov atom) that players read before they can decode or seek; for streaming, that index should sit at the front of the file ("faststart"). When you upload an MP4, the service checks for this, and if the index is at the end it remuxes the file losslessly — a stream copy, with no re-encoding — to move the index to the front. Frames and timestamps are unchanged, so boxes and masks map exactly onto your original video, and the remuxed file isn't returned.

Skip the remux

Export with faststart already enabled so your upload passes through unchanged: ffmpeg -i input.mp4 -c copy -movflags +faststart output.mp4.

Stream the response

Streaming is the recommended path for segmentation, especially for video: set stream to true, as the examples above do, and read the server-sent events to response.completed. Each frame arrives as its own response.output_text.delta, so partial progress renders while the rest of the clip is still segmenting.

Streaming is not required. A non-streaming request (stream omitted or false) runs the same segmentation and returns the complete result in one response; both paths produce identical output. Prefer streaming for long clips so frames render as they arrive instead of waiting for the whole response.

Choose a mask encoding

Masks arrive in the lossless encoding by default. Set the top-level metadata.mask_encoding field to switch:

  • Omit metadata, or send "mask_encoding": "lossless": default lossless masks (the ~ marker on the wire).
  • Send "mask_encoding": "one_bit": compact one_bit masks (the ! marker). The payload is several times smaller than lossless, so masks stream faster and use less bandwidth — worth choosing for video or many-object results.
jsonc
{
  "model": "sam-3.1",
  "input": [
    { "type": "message", "role": "user", "content": [
        { "type": "input_text",  "text": "yellow school bus" },
        { "type": "input_image", "image_url": "https://example.com/street.png" }
    ]}
  ],
  "stream": true,
  "metadata": { "mask_encoding": "one_bit" }
}

decodeMaskToRaster turns a mask into a binary raster, so request one_bit for the smallest payloads when you decode with the client library. See read segmentation output for the encoding markers and how to decode a mask.

Limits

  • Formats: MP4 or MOV containers with H.264, H.265, or MPEG-4 video. VP9 and WebM are not accepted.
  • Clip length: up to 15,000 frames (about 10 minutes) per request.
  • Objects: image requests return every matching instance. Video tracks up to 16 objects per frame.

Handle errors

An unknown or inaccessible model returns 404. A team without model access gets the same envelope, carrying code: model_not_found:

json
{
  "error": {
    "code": "model_not_found",
    "message": "The requested model was not found.",
    "param": null,
    "type": "invalid_request_error"
  }
}

Fix: verify the model ID and confirm your account has access. See error handling.

Next steps

With a working prompt and request, handle what comes back: