Segmenting with prompts

Segment Anything Model 3.1 (SAM 3.1) segments by concept or by location. Give it a short noun phrase naming one thing — "yellow school bus", "glasses", "red hat" — and it returns a box and a mask for every match. Or mark specific objects with boxes, points, or masks, and it segments those objects and follows them through a video.

A request names what to segment in its input_text part in one of two ways, never both:

  • A concept prompt: a short noun phrase. SAM 3.1 detects every instance of the concept. See Write the concept prompt.
  • Object prompts: a box, points, or a mask on each object you want, under an object id you choose. SAM 3.1 segments those objects and reports each one under your id. See Prompt with boxes, points, and masks.

One Responses API call on sam-3.1 carries the prompt and the media, an image or a video, and streams a box and a mask per object. The prompt is the part that decides what you get back, so start there.

Write the concept prompt

The prompt goes in an input_text part as a short noun phrase. SAM 3.1 matches the concept you name, so concrete wording controls the result.

  • Name one concept with a noun phrase: "yellow school bus", "red hat", or "glasses". Commands ("segment the bus") and questions ("where is the bus?") are unreliable and often return nothing. Name the object; don't instruct the model.
  • One concept per request: there is no multi-concept syntax. Comma, semicolon, and plus lists ("bus, bicycle") return zero matches; an "and", a newline, or a second input_text part segments only one of the concepts. To segment several concepts, send one request each.
  • Add attributes to narrow: "square" matches every square; "red square" matches only the red ones. Color, type, and other concrete attributes filter predictably, down to zero when nothing matches.
  • Formatting is ignored: plurals ("red squares"), case, trailing punctuation, and extra whitespace don't change what the prompt matches. Styling carries no meaning.
  • Skip negation, boolean OR, and spatial phrasing: "person not wearing a hat" and "cat or dog" return inconsistent results, and "person on the left" is only partially honored. Segment the plain concept, then filter by position or attribute in your app using the returned box coordinates.
  • Keep it short: a concise noun phrase works as well as a long description. Extra words give no benefit, and padding a prompt with filler degrades matching.

SAM 3.1 locates the objects you name; it doesn't answer questions or reason about a scene. For scene understanding, use the Muse Spark model line, or a combination: use Muse Spark to caption, answer questions, or identify scene activity via image understanding or video understanding, then pass the concept to SAM 3.1 for the segmentation work.

Attributes change the match set predictably. On an image of six squares, "square" returns six matches, "red square" returns three, and "yellow square" returns none. Each match comes back as one box and one mask; see Segment an image for the wire records and read segmentation output to parse them.

A concept that isn't present returns zero matches: the stream completes with response.completed and an empty output_text. Treat it as a normal result, not an error.

Prompt with boxes, points, and masks

Use object prompts when you already know where an object is: a user clicked it, another model found it, or an earlier response returned its box and mask. You give each object an id, and the response reports the object under that id.

The prompt goes in the same input_text part, written in the same token grammar as the segmentation output. This prompt names two objects on an 1800×1200 image:

text
<0f>1<|box;x1=444;y1=626;x2=675;y2=849;w=1800;h=1200|>,2<|point;x=1430;y=700;w=1800;h=1200|>-<|point;x=1300;y=450;w=1800;h=1200|>

Object 1 is marked with a box. Object 2 is marked with a point on the object and a negative point on a region that isn't part of it. The rules:

  • Frame marker: <Nf> opens the prompts for source frame N, up to the next marker. An image uses <0f> only.
  • Objects: after the marker come comma-separated objects, each an object id followed by one or more tokens. An id appears at most once per frame.
  • Box: <|box;x1=..;y1=..;x2=..;y2=..;w=..;h=..|> marks the whole object. x2 and y2 are inclusive, as in output boxes. An object takes at most one box per frame.
  • Positive point: <|point;x=..;y=..;w=..;h=..|> marks a pixel that is part of the object.
  • Negative point: a - before a point marks a region that isn't part of the object. Only points can be negative.
  • Mask: <|mask;x=0;y=0;data=..|> gives the object's exact pixels on that frame. See Send a mask as a prompt.
  • Size: every box and point token carries the same w and h, the media's own pixel size. See Convert coordinates to media pixels. A mask token carries its size in data instead.
  • One kind of prompt: the text holds either a noun phrase or object prompts, never both.

To build box and point prompts in code instead of by hand, use buildObjectPrompt in TypeScript or build_object_prompt in Python. See Build box and point prompts.

Send the prompt with the image as in any other request:

json
{
  "model": "sam-3.1",
  "input": [
    { "type": "message", "role": "user", "content": [
        { "type": "input_text", "text": "<0f>1<|box;x1=444;y1=626;x2=675;y2=849;w=1800;h=1200|>,2<|point;x=1430;y=700;w=1800;h=1200|>-<|point;x=1300;y=450;w=1800;h=1200|>" },
        { "type": "input_image", "image_url": "https://example.com/truck.jpg" }
    ]}
  ],
  "stream": true
}

The response carries one record per prompted object, under the ids from the request. Representative output, with object 2 and the mask payload elided:

text
<0f>1<|box;x1=444;y1=625;x2=679;y2=850;w=1800;h=1200|><|mask;x=0;y=0;data=226,236,~...|>,2<|box;...|><|mask;...|>

Convert coordinates to media pixels

Set w and h to the media's width and height in pixels as displayed, after any rotation the file specifies, such as a photo's EXIF orientation or a video's rotation flag. These are the same w and h that the response reports. A video stored as 960×540 frames with a 90° rotation flag displays as 540×960, so its prompts use w=540;h=960.

When a user draws on a preview, convert each coordinate before you build the prompt:

  1. Subtract the position where the media starts in your view.
  2. Multiply by the size of the media region shown, divided by its size on screen.
  3. Add the position where that region starts in the media. Without a crop, the region is the whole media and starts at (0, 0).
  4. Round down to whole pixels, and keep each coordinate inside the media.

A click at (100, 50) on a 640×360 preview of a 1920×1080 video gives x = 100 × 1920 / 640 = 300 and y = 50 × 1080 / 360 = 150, sent as <|point;x=300;y=150;w=1920;h=1080|>.

Send a mask as a prompt

A mask prompt gives SAM 3.1 the exact pixels of an object on one frame. Write the mask token the way a response writes it. That lets you send a record from an earlier response back unchanged, box included:

text
<120f>1<|box;x1=491;y1=319;x2=640;y2=444;w=960;h=540;c=0.98|><|mask;x=0;y=0;c=0.98;data=126,150,~...|>
  • Payload: a lossless (~) or one_bit (!) payload, in the format of read segmentation output. SAM 3.1 ignores a c field on the box or the mask.
  • Placement: a box before the mask places it, and SAM 3.1 scales the mask to fill the box. That box isn't a separate box prompt.
  • No box: a mask without a box covers the whole frame, scaled to the frame's size.
  • Size: the mask token has no w and h. The box that places it carries the same w and h as every other box and point token in the request.
  • One mask per object: on each frame, an object carries one mask and at most the box that places it. Don't add points, and don't make the mask negative. The mask must select at least one pixel.

You can combine masks, boxes, and points across objects and frames. This prompt marks object 2 with a box on frame 0, and sends object 1 as a mask on frame 120:

text
<0f>2<|box;x1=100;y1=80;x2=219;y2=239;w=960;h=540|><120f>1<|box;x1=491;y1=319;x2=640;y2=444;w=960;h=540|><|mask;x=0;y=0;data=126,150,~...|>

Build the request

A request pairs the prompt with one media part inside a single user message. Send the prompt in an input_text part and the media in an input_image or input_video part:

  • input_text: the prompt, either a short noun phrase for one concept (see Write the concept prompt) or object prompts (see Prompt with boxes, points, and masks).
  • input_image: a still as image_url.
  • input_video: a clip as video_url, or as file_id for a video uploaded through the Files API.
  • image_url / video_url: a public URL or a base64 data URL (data:image/png;base64,...).

The same model serves both images and video. Swap the media part and keep the rest of the request identical. A concept prompt names one concept per request: to segment several object types, send one request each, or mark each object with object prompts in a single request.

Segment an image

Send an input_image part and stream the response to response.completed:

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with client.responses.stream(
    model="sam-3.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "glasses",
                },
                {
                    "type": "input_image",
                    "image_url": "https://example.com/photo.png",
                },
            ],
        },
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
print()

An image returns on a single line: one record per match, comma-separated, each record a box then a mask. Here w and h are the source image dimensions, repeated on every record:

text
<0f>0<|box;x1=181;y1=365;x2=556;y2=486;w=1516;h=1600|><|mask;x=0;y=0;data=122,376,~...|>

<0f> is the frame, the leading integer is the object ordinal, then one box then one mask; ~ marks a lossless mask payload. Full grammar in reading segmentation output.

Segment a video

Swap in an input_video part with a video_url. Start with a hosted clip to tune the prompt before uploading anything:

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with client.responses.stream(
    model="sam-3.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "red bicycle",
                },
                {
                    "type": "input_video",
                    "video_url": "https://example.com/clip.mp4",
                },
            ],
        },
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
print()

A video returns one line per frame, each in its own response.output_text.delta. For a concept prompt, lines arrive in ascending frame order. Object prompts can return frames out of order (see Choose a tracking direction), so always read the frame marker on each line. Frame 0 of a clip with two matches:

text
<0f>0<|box;x1=211;y1=228;x2=270;y2=254;w=320;h=334|><|mask;x=0;y=0;data=27,60,~...|>,1<|box;x1=155;y1=228;x2=202;y2=254;w=320;h=334|><|mask;x=0;y=0;data=27,48,~...|>

Upload once, reference by ID

For a clip you segment more than once, upload it with POST /v1/files and set purpose to user_data. Upload is synchronous: a 200 means the file is ready to use in the next request.

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with open("/path/to/clip.mp4", "rb") as f:
    file = client.files.create(
        purpose="user_data",
        file=f,
    )
​
print(file.model_dump_json(indent=2))

Example upload response

json
{
  "id": "file-842549258569145",
  "object": "file",
  "bytes": 245832,
  "created_at": 1714502400,
  "filename": "clip.mp4",
  "purpose": "user_data",
  "status": "uploaded"
}

Pass the returned ID in the file_id field of the input_video part:

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with client.responses.stream(
    model="sam-3.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "delivery van",
                },
                {
                    "type": "input_video",
                    "file_id": "file-abc123",
                },
            ],
        },
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
print()

See file handling for expiration, listing, and deletion.

MP4 faststart. An MP4 stores an index (the moov atom) that players read before they can decode or seek; for streaming, that index should sit at the front of the file ("faststart"). When you upload an MP4, the service checks for this, and if the index is at the end it remuxes the file losslessly — a stream copy, with no re-encoding — to move the index to the front. Frames and timestamps are unchanged, so boxes and masks map exactly onto your original video, and the remuxed file isn't returned.

Skip the remux

Export with faststart already enabled so your upload passes through unchanged: ffmpeg -i input.mp4 -c copy -movflags +faststart output.mp4.

Prompt several video frames

In a video, object prompts can go on any frames. Each <Nf> marker opens the prompts for that frame. Repeat an object's id on a later frame to prompt the same object again; a new id starts a new object.

This request prompts two objects across four frames of a 200-frame, 960×540 clip. Object 1 gets a point on frame 30 and a box on frame 120. Object 2 gets a box on frame 75, then a positive and a negative point on frame 160:

json
{
  "model": "sam-3.1",
  "input": [
    { "type": "message", "role": "user", "content": [
        { "type": "input_text", "text": "<30f>1<|point;x=585;y=366;w=960;h=540|><75f>2<|box;x1=405;y1=337;x2=516;y2=420;w=960;h=540|><120f>1<|box;x1=493;y1=320;x2=641;y2=444;w=960;h=540|><160f>2<|point;x=533;y=403;w=960;h=540|>-<|point;x=482;y=405;w=960;h=540|>" },
        { "type": "input_video", "file_id": "file-abc123" }
    ]}
  ],
  "stream": true,
  "metadata": { "propagation_direction": "forward" }
}

SAM 3.1 tracks both objects from frame 30, the earliest prompt frame, to the end of the clip: the response carries frames 30 through 199, with objects 1 and 2 on each frame where they're visible.

Choose a tracking direction

For object prompts in video, metadata.propagation_direction sets which frames SAM 3.1 tracks the prompted objects over. both and forward start at the earliest prompt frame, and backward starts at the latest:

propagation_directionFrames trackedPrompts on frames 30 and 120 of a 200-frame clip
both (default)From the earliest prompt frame to the last frame, then from the frame before it back to frame 030 to 199, then 29 to 0
forwardFrom the earliest prompt frame to the last frame30 to 199
backwardFrom the latest prompt frame back to frame 0120 to 0

Omit the key for both. backward needs at least one prompt on a frame after frame 0. Tracking fewer frames costs less: video is billed per frame, with a minimum of 50 frames per request. With prompts on frames 30 and 120 of a 200-frame clip, forward bills 170 frames, backward bills 121, and both bills 200. With both, the frames before the earliest prompt arrive last, in descending order, so key your results by the frame marker rather than by line position (see frame markers).

Stream the response

Streaming is the recommended path for segmentation, especially for video: set stream to true, as the examples above do, and read the server-sent events to response.completed. Each frame arrives as its own response.output_text.delta, so partial progress renders while the rest of the clip is still segmenting.

Streaming is not required. A non-streaming request (stream omitted or false) runs the same segmentation and returns the complete result in one response, in the same format. Results can vary slightly between runs of the same request, streamed or not: box coordinates, mask edges, and occasionally which objects are found on a frame. Prefer streaming for long clips so frames render as they arrive instead of waiting for the whole response.

Request confidence

Set metadata.include_confidence to "true" to get the model's confidence for each record. Metadata values are strings, so send "true" rather than a JSON boolean:

json
{
  "model": "sam-3.1",
  "input": [
    { "type": "message", "role": "user", "content": [
        { "type": "input_text",  "text": "glasses" },
        { "type": "input_image", "image_url": "https://example.com/photo.png" }
    ]}
  ],
  "stream": true,
  "metadata": { "include_confidence": "true" }
}

Every box and every mask then carries a c field, a number from 0 through 1, on every frame. It works with concept prompts and with object prompts, in every tracking direction. The box and the mask of one record carry the same value, and the same object can carry a different value on each frame. One record with confidence, mask payload elided:

text
<0f>1<|box;x1=444;y1=625;x2=679;y2=850;w=1800;h=1200;c=0.9609375|><|mask;x=0;y=0;c=0.9609375;data=226,236,~...|>

Omit the key, or send "false", for output without c. The field's place in each token is in read segmentation output.

Choose a mask encoding

Masks arrive in the lossless encoding by default. Set the top-level metadata.mask_encoding field to switch:

  • Omit metadata, or send "mask_encoding": "lossless": default lossless masks (the ~ marker on the wire).
  • Send "mask_encoding": "one_bit": compact one_bit masks (the ! marker). The payload is several times smaller than lossless, so masks stream faster and use less bandwidth — worth choosing for video or many-object results.
jsonc
{
  "model": "sam-3.1",
  "input": [
    { "type": "message", "role": "user", "content": [
        { "type": "input_text",  "text": "yellow school bus" },
        { "type": "input_image", "image_url": "https://example.com/street.png" }
    ]}
  ],
  "stream": true,
  "metadata": { "mask_encoding": "one_bit" }
}

Every prompt type honors mask_encoding. SAM 3.1 can still return a one_bit mask for an object on a frame where no lossless mask is available, so read the marker on each mask rather than assuming the encoding you requested.

decodeMaskToRaster turns a mask into a binary raster, so request one_bit for the smallest payloads when you decode with the client library. See read segmentation output for the encoding markers and how to decode a mask.

Limits

  • Formats: MP4 or MOV containers with H.264, H.265, or MPEG-4 video. VP9 and WebM are not accepted.
  • Clip length: up to 15,000 frames (about 10 minutes) per request.
  • Objects: image requests return every matching instance. Video tracks up to 16 objects per frame.
  • Prompted objects: object prompts name up to 16 objects, masks included. An object prompted on several frames counts once.

Handle errors

A prompt or metadata value that breaks the rules above returns 400, with a message that names the problem and quotes the values from your request:

RequestMessage
A noun phrase together with object promptsinvalid prompt: object prompts cannot be combined with a phrase
A negative boxinvalid prompt: a box names the object; it cannot be negative
Two boxes for one object on one frameinvalid prompt: object 1 carries more than one box
The same object id twice on one frameinvalid prompt: object id 1 is used more than once on frame 0
An object with only negative pointsinvalid prompt: object 1 on frame 0 needs a box or a positive point
A mask and points for one object on one frameinvalid prompt: object 1 on frame 0 carries a mask and points; send one or the other
Two masks for one object on one frameinvalid prompt: object 1 on frame 0 carries more than one mask
A negative maskinvalid prompt: a mask names the object; it cannot be negative
A mask that selects no pixelinvalid prompt: object 1 on frame 0 carries a mask that selects no pixel
Tokens with different w and hinvalid prompt: every prompt token must declare the same w/h; got 900x600, 1800x1200
A point outside the mediainvalid prompt: point (1900.0, 100.0) lies outside the 1800x1200 frame
An image prompt on a frame other than 0an image prompt must be on frame 0, got <5f>
More than 16 prompted objectsinvalid prompt: 17 objects exceed this model's limit of 16
An include_confidence value other than "true" or "false"metadata["include_confidence"] must be "true" or "false"
An unknown propagation_directionmetadata["propagation_direction"] must be one of ['both', 'forward', 'backward']
backward with every prompt on frame 0backward propagation needs an object prompt on a frame after the first
w and h with a different aspect ratio from the mediaprompt coordinates are given in a 960x600 frame, whose aspect ratio does not match the 960x540 media

Fix: correct the prompt or the metadata value and resend. A 400 is permanent for that request, so don't retry it unchanged.

An unknown or inaccessible model returns 404. A team without model access gets the same envelope, carrying code: model_not_found:

json
{
  "error": {
    "code": "model_not_found",
    "message": "The requested model was not found.",
    "param": null,
    "type": "invalid_request_error"
  }
}

Fix: verify the model ID and confirm your account has access. See error handling.

A long clip with large masks can produce a response too large for Meta Model API to store. With store at its default, true, a non-streaming request then returns 413 with the code payload_too_large. A streaming request delivers every frame, but the response isn't stored. See 413 Payload Too Large.

Fix: set store to false for long clips, and read the result from the response itself. one_bit masks also make the response several times smaller.

Next steps

With a working prompt and request, handle what comes back: