Video and audio understanding

Muse Spark reads both moving pictures and sound. Summarize a clip, ask what happened when, extract structured details you can use downstream, or transcribe speech from a recording. Upload the media once (or pass a URL), add a text prompt, and the model returns text. Video and audio both work on the Responses API and Chat Completions.

Video

Muse Spark reads a video's visual sequence and any embedded audio together, so one upload can both describe the footage and transcribe its speech.

How it works

Video understanding takes two steps:

Upload the video through the Files API with purpose set to "user_data".
Reference the uploaded file by ID in a Responses API request using an input_file content block.

The model processes the video alongside your prompt and returns a text response.

Basic usage

Upload a file, then ask Muse Spark to describe what it sees. Reference the uploaded video by file_id with an input_file content block.

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.responses.create(
    model="muse-spark-1.3",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Describe what happens in this video.",
                },
                {
                    "type": "input_file",
                    "file_id": "file-abc123",
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.responses.create({
  model: 'muse-spark-1.3',
  input: [
    {
      type: 'message',
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'Describe what happens in this video.',
        },
        {
          type: 'input_file',
          file_id: 'file-abc123',
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.3",
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "Describe what happens in this video.",
                    },
                    {
                        "type": "input_file",
                        "file_id": "file-abc123",
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

shell
curl -X POST "https://api.meta.ai/v1/responses" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "muse-spark-1.3",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "Describe what happens in this video."
        },
        {
          "type": "input_file",
          "file_id": "file-abc123"
        }
      ]
    }
  ]
}'

Reference a video by URL

If the video is already hosted where the API can reach it, skip the upload and pass it directly. An input_video block accepts video_url—a public or base64 data URL—or a file_id for a video uploaded through the Files API.

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.responses.create(
    model="muse-spark-1.3",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Describe what happens in this video.",
                },
                {
                    "type": "input_video",
                    "video_url": "https://example.com/clip.mp4",
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.responses.create({
  model: 'muse-spark-1.3',
  input: [
    {
      type: 'message',
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'Describe what happens in this video.',
        },
        {
          type: 'input_video',
          video_url: 'https://example.com/clip.mp4',
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.3",
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "Describe what happens in this video.",
                    },
                    {
                        "type": "input_video",
                        "video_url": "https://example.com/clip.mp4",
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

shell
curl -X POST "https://api.meta.ai/v1/responses" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "muse-spark-1.3",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "Describe what happens in this video."
        },
        {
          "type": "input_video",
          "video_url": "https://example.com/clip.mp4"
        }
      ]
    }
  ]
}'

Use input_file or input_video with file_id for videos you upload through the Files API; use input_video with video_url for a video at a public URL.

Audio

Audio understanding in Muse Spark 1.3 is currently not fully supported, and response quality for requests including audio content may be degraded. These examples use Muse Spark 1.2 instead.

Muse Spark transcribes spoken audio to text, whether it arrives as a standalone file or as the soundtrack of a video.

Transcribe a standalone audio file

Send the audio as an input_audio content part on Chat Completions or the Responses API (audio/mpeg or audio/wav, uploaded or inline base64). An example for each endpoint follows. First, read the file and base64-encode it; both request examples below reference this audio_b64 variable:

python
import base64
with open("speech.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

typescript
import fs from 'fs';
const audio_b64 = fs.readFileSync('speech.wav').toString('base64');

On Chat Completions, input_audio goes in the message content:

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.chat.completions.create(
    model="muse-spark-1.2",
    max_tokens=4000,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Transcribe this audio. Return only the transcript.",
                },
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": audio_b64,
                        "format": "wav",
                    },
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.2",
        "max_tokens": 4000,
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Transcribe this audio. Return only the transcript.",
                    },
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_b64,
                            "format": "wav",
                        },
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.chat.completions.create({
  model: 'muse-spark-1.2',
  max_tokens: 4000,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'Transcribe this audio. Return only the transcript.',
        },
        {
          type: 'input_audio',
          input_audio: {
            data: audio_b64,
            format: 'wav',
          },
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

The Responses API accepts the same input_audio content part — send inline input_audio ({data, format}), an audio_url data URI, or a file_id for an uploaded audio file:

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.responses.create(
    model="muse-spark-1.2",
    max_output_tokens=4000,
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Transcribe this audio. Return only the transcript.",
                },
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": audio_b64,
                        "format": "wav",
                    },
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.2",
        "max_output_tokens": 4000,
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "Transcribe this audio. Return only the transcript.",
                    },
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_b64,
                            "format": "wav",
                        },
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.responses.create({
  model: 'muse-spark-1.2',
  max_output_tokens: 4000,
  input: [
    {
      type: 'message',
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'Transcribe this audio. Return only the transcript.',
        },
        {
          type: 'input_audio',
          input_audio: {
            data: audio_b64,
            format: 'wav',
          },
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

Speech from a video

A video part on either endpoint also carries its embedded audio, so a single video upload can transcribe its speech alongside visual understanding. See Video above.

Transcription tips

Two things to get right for transcription:

Give the response room. Muse Spark is a reasoning model and reasoning shares the output budget — set max_tokens (or max_output_tokens) to at least 4000, or a transcript can come back empty with finish_reason: length.
Stream long audio. High reasoning effort on a long clip can take a while before the first token; set stream: true and/or lower reasoning_effort to avoid an idle-timeout disconnect.

Supported formats

Video understanding supports mp4 files; audio understanding supports MP3 and WAV.

MIME type	Extension
`video/mp4`	`.mp4`
`audio/mpeg`	`.mp3`
`audio/wav`	`.wav`

Next steps

Add still-image analysis with image understanding.
Manage uploads and reuse files with the Files API.
Check the full request schema in the Responses API reference.

Video and audio understanding

Video

Muse Spark reads a video's visual sequence and any embedded audio together, so one upload can both describe the footage and transcribe its speech.

How it works

Video understanding takes two steps:

Upload the video through the Files API with purpose set to "user_data".
Reference the uploaded file by ID in a Responses API request using an input_file content block.

The model processes the video alongside your prompt and returns a text response.

Basic usage

Upload a file, then ask Muse Spark to describe what it sees. Reference the uploaded video by file_id with an input_file content block.

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.responses.create(
    model="muse-spark-1.3",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Describe what happens in this video.",
                },
                {
                    "type": "input_file",
                    "file_id": "file-abc123",
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.responses.create({
  model: 'muse-spark-1.3',
  input: [
    {
      type: 'message',
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'Describe what happens in this video.',
        },
        {
          type: 'input_file',
          file_id: 'file-abc123',
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.3",
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "Describe what happens in this video.",
                    },
                    {
                        "type": "input_file",
                        "file_id": "file-abc123",
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

shell
curl -X POST "https://api.meta.ai/v1/responses" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "muse-spark-1.3",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "Describe what happens in this video."
        },
        {
          "type": "input_file",
          "file_id": "file-abc123"
        }
      ]
    }
  ]
}'

Reference a video by URL

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.responses.create(
    model="muse-spark-1.3",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Describe what happens in this video.",
                },
                {
                    "type": "input_video",
                    "video_url": "https://example.com/clip.mp4",
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.responses.create({
  model: 'muse-spark-1.3',
  input: [
    {
      type: 'message',
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'Describe what happens in this video.',
        },
        {
          type: 'input_video',
          video_url: 'https://example.com/clip.mp4',
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.3",
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "Describe what happens in this video.",
                    },
                    {
                        "type": "input_video",
                        "video_url": "https://example.com/clip.mp4",
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

shell
curl -X POST "https://api.meta.ai/v1/responses" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "muse-spark-1.3",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "Describe what happens in this video."
        },
        {
          "type": "input_video",
          "video_url": "https://example.com/clip.mp4"
        }
      ]
    }
  ]
}'

Use input_file or input_video with file_id for videos you upload through the Files API; use input_video with video_url for a video at a public URL.

Audio

Audio understanding in Muse Spark 1.3 is currently not fully supported, and response quality for requests including audio content may be degraded. These examples use Muse Spark 1.2 instead.

Muse Spark transcribes spoken audio to text, whether it arrives as a standalone file or as the soundtrack of a video.

Transcribe a standalone audio file

python
import base64
with open("speech.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

typescript
import fs from 'fs';
const audio_b64 = fs.readFileSync('speech.wav').toString('base64');

On Chat Completions, input_audio goes in the message content:

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.chat.completions.create(
    model="muse-spark-1.2",
    max_tokens=4000,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Transcribe this audio. Return only the transcript.",
                },
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": audio_b64,
                        "format": "wav",
                    },
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.2",
        "max_tokens": 4000,
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Transcribe this audio. Return only the transcript.",
                    },
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_b64,
                            "format": "wav",
                        },
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.chat.completions.create({
  model: 'muse-spark-1.2',
  max_tokens: 4000,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'Transcribe this audio. Return only the transcript.',
        },
        {
          type: 'input_audio',
          input_audio: {
            data: audio_b64,
            format: 'wav',
          },
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

The Responses API accepts the same input_audio content part — send inline input_audio ({data, format}), an audio_url data URI, or a file_id for an uploaded audio file:

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.responses.create(
    model="muse-spark-1.2",
    max_output_tokens=4000,
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Transcribe this audio. Return only the transcript.",
                },
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": audio_b64,
                        "format": "wav",
                    },
                },
            ],
        },
    ],
)
print(response.model_dump_json(indent=2))

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.2",
        "max_output_tokens": 4000,
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "Transcribe this audio. Return only the transcript.",
                    },
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_b64,
                            "format": "wav",
                        },
                    },
                ],
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.responses.create({
  model: 'muse-spark-1.2',
  max_output_tokens: 4000,
  input: [
    {
      type: 'message',
      role: 'user',
      content: [
        {
          type: 'input_text',
          text: 'Transcribe this audio. Return only the transcript.',
        },
        {
          type: 'input_audio',
          input_audio: {
            data: audio_b64,
            format: 'wav',
          },
        },
      ],
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

Speech from a video

A video part on either endpoint also carries its embedded audio, so a single video upload can transcribe its speech alongside visual understanding. See Video above.

Transcription tips

Two things to get right for transcription:

Give the response room. Muse Spark is a reasoning model and reasoning shares the output budget — set max_tokens (or max_output_tokens) to at least 4000, or a transcript can come back empty with finish_reason: length.
Stream long audio. High reasoning effort on a long clip can take a while before the first token; set stream: true and/or lower reasoning_effort to avoid an idle-timeout disconnect.

Supported formats

Video understanding supports mp4 files; audio understanding supports MP3 and WAV.

MIME type	Extension
`video/mp4`	`.mp4`
`audio/mpeg`	`.mp3`
`audio/wav`	`.wav`

Next steps

Add still-image analysis with image understanding.
Manage uploads and reuse files with the Files API.
Check the full request schema in the Responses API reference.

Video and audio understanding

Video

How it works

Basic usage

Reference a video by URL

Audio

Transcribe a standalone audio file

Speech from a video

Transcription tips

Supported formats

Next steps

Please confirm your identity

Video and audio understanding

Video

How it works

Basic usage

Reference a video by URL

Audio

Transcribe a standalone audio file

Speech from a video

Transcription tips

Supported formats

Next steps

Please confirm your identity