Computer use on macOS with Muse Spark

A Meta Model Cookbook recipe — computer use with an agent that sees your screen, moves the mouse, and works out a native macOS app on its own.

metacua is a hackable command-line agent that lets Muse Spark operate a Mac through its GUI. You give it a goal; it captures the screen, sends the image plus GUI tool definitions to the model, executes the returned tool calls with native macOS mouse and keyboard events, then screenshots again and repeats until the task is done — the same look-act-look loop a person runs.

Unlike the sibling computer use recipe, which drives a throwaway Linux desktop in a Cua sandbox, metacua drives your actual machine. That makes it a faithful example of native computer use, and it makes the safety notes below non-negotiable.

What you'll learn

1. How to build a look-act-look computer-use loop against POST {base}/responses.
2. How to expose a CUA-guide-aligned computer.computer tool plus a terminal computer.stop tool through OpenAI-style function calling.
3. How coordinate conventions differ: normalized 0-1000 tool coordinates vs. real display pixels vs. fixed 1920x1080 OSWorld pixels.
4. How the same agent is built twice — once in Swift (SwiftPM) and once in Python — against the identical Muse Spark contract.

How it works

1. Describe the goal in plain language.
2. See — the agent captures a screenshot of the primary display.
3. Send — the screenshot and tool definitions go to Muse Spark at POST {base}/responses.
4. Act — the model returns a computer.computer tool call; the agent executes it with native macOS events (Quartz / AppKit / Accessibility).
5. Look again — it screenshots the new state and reacts.
6. Repeat until the model calls computer.stop with a final answer, or the step cap is hit.

The model never runs code on your machine on its own. It sees PNG screenshots and returns mouse and keyboard actions, which the agent translates into synthetic events.

Canonical configuration

This recipe uses the same setup as every other cookbook recipe:

The agent posts to {base}/responses (the Responses API) with parallel_tool_calls: false and store: false, so computer actions stay serial and stateful.

Tools

metacua advertises two model-facing tools, using the same dotted names as the Muse Spark reference agent. Tool calls run one at a time.

computer.computer mirrors Anthropic's computer_20251124 interface (clicks, drag, mouse_move, scroll, key, type, hold_key/release_key, wait, screenshot), except coordinates are normalized to 0...1000 and mapped to primary-display pixels before execution. Pass --batched-actions to advertise an actions array instead; the batch is validated up front, executed in order, and returns a single screenshot at the end.

The Python implementation additionally supports an OSWorld pyautogui mode (--syntax pyautogui): the model returns pyautogui calls as text, which metacua parses with ast and never execs.

Run it

Both implementations read the same config (flags → environment → ~/.metacua/config.json → defaults) and require macOS Accessibility and Screen Recording permissions for the terminal that launches them.

Swift

sh
cd 13_macos_cua
make permissions ARGS="--prompt"          # grant Accessibility + Screen Recording
make configure ARGS="--api-key $MODEL_API_KEY"
make run                                   # interactive metacua > shell

Or run a single goal:

sh
swift run -c release metacua agent --goal "Open Safari and search for the weather in Tokyo"

Full build, flags, and interactive commands are in metacua.md(opens in new tab).

Python

sh
cd 13_macos_cua/python
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
metacua permissions --prompt
metacua configure --api-key "$MODEL_API_KEY"
metacua agent --goal "Open Notes and write today's date"

Full options, OSWorld mode, and the API request shape are in python/README.md(opens in new tab).

Coordinate conventions

Agent mode defaults to Muse Spark's normalized 0-1000 convention: (0, 0) is the top-left of the primary display and (1000, 1000) is the bottom-right. The agent converts to primary-display pixels before executing. Screenshots may be downscaled (--screenshot-scale), but the action coordinate space always uses the display's full logical size. Scroll direction is normalized from the user's point of view: down reveals lower content regardless of the macOS natural-scrolling preference.

Files in this recipe

13_macos_cua/
├── README.md          ← this recipe
├── metacua.md         ← full Swift build/run reference (make targets, flags, commands)
├── assets/            ← images referenced above (tic_tac_toe.gif)
├── Makefile           ← build-and-run targets for the Swift CLI
├── Package.swift      ← SwiftPM manifest
├── src/metacua/       ← Swift implementation
├── python/
│   ├── README.md      ← full Python reference (install, OSWorld mode, API shape)
│   ├── metacua/       ← Python implementation
│   ├── tests/         ← pytest suite
│   └── tools/         ← trace_to_html.py
└── ...

Prompting tips

• Give a goal, not a script. A concrete objective surfaces real problem-solving.
• Let it screenshot after every action. The loop reacts to the real screen state instead of stacking blind clicks on a stale picture.
• Keep goals narrow and reversible, especially while you are getting a feel for it.
• The type action does not submit — follow it with a key action (Return) when needed.