Metadata-Version: 2.5
Name: axpilot
Version: 0.1.0a1
Summary: AXPilot: an MCP server that lets AI agents see and operate macOS, safely.
Project-URL: Homepage, https://github.com/vansh3503/Mac-Mcp
Project-URL: Documentation, https://github.com/vansh3503/Mac-Mcp/tree/main/docs
Project-URL: Issues, https://github.com/vansh3503/Mac-Mcp/issues
Project-URL: Changelog, https://github.com/vansh3503/Mac-Mcp/blob/main/CHANGELOG.md
Author: AXPilot contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: accessibility,ai-agents,automation,computer-use,macos,mcp,model-context-protocol
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: MacOS X
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Desktop Environment
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: anyio>=4.11
Requires-Dist: mcp<3,>=2.3
Requires-Dist: pillow>=10.4
Requires-Dist: pydantic>=2.12
Requires-Dist: pyobjc-core>=11.0; sys_platform == 'darwin'
Requires-Dist: pyobjc-framework-applicationservices>=11.0; sys_platform == 'darwin'
Requires-Dist: pyobjc-framework-cocoa>=11.0; sys_platform == 'darwin'
Requires-Dist: pyobjc-framework-quartz>=11.0; sys_platform == 'darwin'
Requires-Dist: pyobjc-framework-screencapturekit>=11.0; sys_platform == 'darwin'
Requires-Dist: pyobjc-framework-vision>=11.0; sys_platform == 'darwin'
Requires-Dist: tomli-w>=1.0
Requires-Dist: typer>=0.16
Description-Content-Type: text/markdown

# AXPilot

**The MCP server for macOS.** AXPilot lets any AI agent see and operate the apps on your Mac,
through the macOS Accessibility API first and screenshots second, with safety controls that are
on by default.

[![CI](https://github.com/vansh3503/Mac-Mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/vansh3503/Mac-Mcp/actions/workflows/ci.yml)
[![Status: alpha](https://img.shields.io/badge/status-alpha-orange)](#roadmap)
[![PyPI](https://img.shields.io/pypi/v/axpilot?include_prereleases&label=PyPI)](https://pypi.org/project/axpilot/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://github.com/vansh3503/Mac-Mcp/blob/main/pyproject.toml)
[![macOS 14+](https://img.shields.io/badge/macOS-14%2B-black)](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/install.md)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-green)](https://github.com/vansh3503/Mac-Mcp/blob/main/LICENSE)

<!-- mcp-name: io.github.vansh3503/axpilot -->

<p align="center">
  <img src="docs/assets/axpilot-demo.gif" width="900"
       alt="Claude Code uses AXPilot to type a checklist in TextEdit, make its title bold and centred from the Format menu, and turn the other lines into a bulleted list">
</p>
<p align="center"><sub>A real recording on macOS 26: Claude Code (left) drives TextEdit (right) through AXPilot. Sped up in places; the speed is shown on screen.</sub></p>

> **Project status: alpha.** The first alpha, 0.1.0a1, is [on PyPI](#install-from-pypi-alpha); you
> can also [run it from source](#run-from-source-today). It is verified on macOS 26 on Apple
> silicon; macOS 14 and 15 and Intel Macs are not tested yet. Expect breaking changes until 1.0. See the [roadmap](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/PLAN.md#15-roadmap-and-milestones-one-maintainer--ai-assistance).

## Why AXPilot

- **Sees the UI the way macOS does.** A compact text snapshot of the accessibility tree gives every
  button, field, list and menu a short ref (`e1`, `e2`, …), so agents act on elements instead of
  guessing pixels. Text-only and local models work; vision is optional.
- **Acts reliably.** Each action tries the most robust strategy first (an Accessibility action),
  falls back to synthetic mouse and keyboard events, and returns a diff of what changed.
- **Safe by default.** Scripting is off, destructive actions need your confirmation, password
  managers and the Privacy & Security settings are off limits, secure fields are never read, and
  there is **no telemetry**.
- **Works with any MCP client or agent framework.** Standard MCP over stdio or Streamable HTTP:
  Claude Desktop, Claude Code, Cursor, VS Code, Codex CLI, Gemini CLI, OpenAI Agents SDK,
  LangGraph, Pydantic AI and more.
- **Permissions that just work.** A tiny frozen `AXPilot.app` launcher holds the macOS
  permissions, so you grant Accessibility once, to AXPilot, and every client can use it, across updates.

## Features

The feature set planned for 1.0. Every tool below is implemented and tested against the simulated
desktop. On a real Mac (macOS 26), the perception and input tools are verified, including
dropdowns, menus and dialogs; the app, system, scripting and browser tools, the kill switch and the
on-screen indicator are not yet. The [tool reference](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/tools.md) documents each one.

| Area | What you get |
|---|---|
| **Perception** | Accessibility snapshots with refs, element search, element inspection, stable locators for automation scripts, diffs after every action, screenshots with a `screenshot_id` for coordinate mapping, Vision OCR, app/window/display lists |
| **Actions** | Click, type (layout-independent), keyboard shortcuts (layout-aware), scroll, drag, set values, raw AX actions, **menu-bar commands by path**, forms, apps, windows, opening files and URLs |
| **Safety** | Profiles and toolsets, app and window deny-lists, Terminal rule, secure-field protection, confirmations, rate limit, audit log, yield-to-user, kill switch, client pairing |
| **Integrations** | Any MCP client; tested framework examples; an [Agent Skill](https://github.com/vansh3503/Mac-Mcp/blob/main/skills/axpilot/SKILL.md); a `computer` profile for vision-first loops; a Windows-MCP naming profile |

## How it works

```
 MCP clients and agent frameworks
 (Claude, Cursor, VS Code, Codex, OpenAI Agents SDK, LangGraph, …)
        │ spawn `axpilot stdio`                │ Streamable HTTP (`axpilot http`)
        ▼                                      │ 127.0.0.1 + bearer token
 ┌────────────────┐   Unix socket (0600)       │
 │ axpilot bridge │──────────────┐             │
 │ (no perms)     │              ▼             ▼
 └────────────────┘   ┌──────────────────────────────────────────┐
                      │ AXPilot.app: tiny frozen launcher        │
                      │   └─ Python host (child process)         │
                      │       ├ MCP server: tools and schemas    │
                      │       ├ Policy and safety layer          │
                      │       └ Engine: Accessibility, input,    │
                      │         ScreenCaptureKit, Vision OCR     │
                      └──────────────────────────────────────────┘
        macOS permissions (Accessibility, Screen Recording) → AXPilot.app
```

[Spike S1](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/spikes/S1-launcher-tcc.md) confirmed
this design on macOS 26 (Apple silicon): the permission prompts name AXPilot, and one grant covers
every client. **Direct mode** (`axpilot stdio --direct`) runs everything in one process instead;
macOS then attributes permissions to whichever app started the server (your terminal, Claude, Cursor, …).

## Install

**Requirements:** macOS 14 Sonoma or later, Apple silicon or Intel.

### Run from source (today)

You need [uv](https://docs.astral.sh/uv/) and the Xcode Command Line Tools
(`xcode-select --install`), which build the launcher.

```sh
git clone https://github.com/vansh3503/Mac-Mcp.git axpilot
cd axpilot
uv sync
launcher/build.sh                      # builds launcher/build/AXPilot.app
uv run axpilot setup --dev-source .    # installs ~/Applications/AXPilot.app and its runtime
uv run axpilot connect claude-desktop  # or claude-code, cursor, vscode, … (see --list)
uv run axpilot doctor                  # checks the installation
```

Restart your MCP client, then ask it to use AXPilot ("take a snapshot of Finder"):

1. The first time a client connects, AXPilot asks you whether to allow it. Allow it.
2. The first tool call makes macOS ask whether AXPilot may control your computer. Open System
   Settings, switch on **AXPilot** under Accessibility, and ask again; no restart is needed.
   Screenshots and OCR ask for Screen Recording the same way.

Other ways to run it, without the launcher:

```sh
uv run axpilot stdio --engine fake   # the simulated desktop (any OS, no permissions needed)
uv run axpilot stdio --direct        # direct mode: grant Accessibility to the app that starts it
```

`axpilot stdio` speaks MCP on stdin/stdout, so it waits for a client. To poke at it by hand, use
the [MCP Inspector](https://github.com/modelcontextprotocol/inspector):
`npx @modelcontextprotocol/inspector uv run axpilot stdio --engine fake`.

### Install from PyPI (alpha)

```sh
uv tool install axpilot     # or: pipx install axpilot
axpilot setup               # checks your Mac and walks you through permissions
axpilot connect claude-desktop
```

Details, the one-line installer and other channels: [docs/install.md](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/install.md).
Permissions explained: [docs/permissions.md](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/permissions.md).

## Connect a client

### Today, from source

In the clone, `uv run axpilot connect <client>` writes the client's config for you (with a
backup), pointing it at the checkout's `axpilot` by absolute path. Restart the client afterwards.
For direct mode, or to write the config by hand, see
[docs/clients.md](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/clients.md#from-source-today).

### Installed from PyPI

`axpilot connect <client>` writes the config for you (with a backup). The snippets it writes:

**Claude Desktop, Cursor** (`~/.cursor/mcp.json`) and most other clients:

```json
{ "mcpServers": { "axpilot": { "command": "axpilot", "args": ["stdio"] } } }
```

(Claude Desktop needs the absolute path of `axpilot`, e.g. `/Users/you/.local/bin/axpilot`.)

**Claude Code**:

```sh
claude mcp add axpilot -- axpilot stdio
```

**VS Code** (`.vscode/mcp.json` in a workspace, or your user `mcp.json`):

```json
{ "servers": { "axpilot": { "type": "stdio", "command": "axpilot", "args": ["stdio"] } } }
```

More clients (Windsurf, Zed, Codex CLI, Gemini CLI, LM Studio, Goose, Cline, Continue):
[docs/clients.md](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/clients.md).

## Use it from an agent framework

Anything that speaks MCP works. Two examples (APIs as of writing; one for each of eleven
frameworks is in [`examples/frameworks/`](https://github.com/vansh3503/Mac-Mcp/tree/main/examples/frameworks)):

**OpenAI Agents SDK**

```python
from agents import Agent, Runner
from agents.mcp import MCPServerStdio

async with MCPServerStdio(params={"command": "axpilot", "args": ["stdio"]}, cache_tools_list=True) as mac:
    agent = Agent(
        name="Mac operator",
        instructions="Call snapshot before acting, act on element refs, and verify the result.",
        mcp_servers=[mac],
    )
    result = await Runner.run(agent, "Open Notes and create a note titled 'Groceries'")
```

**LangChain / LangGraph**

```python
from langchain.agents import create_agent
from langchain_mcp_adapters.client import MultiServerMCPClient

client = MultiServerMCPClient({"axpilot": {"command": "axpilot", "args": ["stdio"], "transport": "stdio"}})
agent = create_agent("<your-model>", await client.get_tools())
```

More frameworks: [docs/frameworks.md](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/frameworks.md).

## Tools

The 1.0 tool surface, implemented. Profiles pick a starting set: `minimal` (10 tools, for small or
local models), `standard` (default), `full`, and `computer` (one vision-first `computer` tool). A
`windows-mcp` naming exposes the same tools under Windows-MCP's names and parameters.

| Toolset | Tools | Default |
|---|---|---|
| `observe` | `snapshot`, `screenshot`, `find`, `explore`, `read`, `inspect`, `locate`, `ocr`, `list_apps`, `list_windows`, `list_displays`, `wait_for`, `status` | on (read-only) |
| `input` | `click`, `type_text`, `press_keys`, `scroll`, `drag`, `move_mouse`, `set_value`, `select_option`, `perform_action`, `select_menu`, `fill_form`, `handle_dialog` | on |
| `apps` | `app`, `window`, `open` | on |
| `system` | `clipboard`, `notify`, `spotlight_search`, `system_info`, `system_control`, `processes` | `clipboard` on |
| `scripting` | `shell`, `applescript`, `shortcuts`, `defaults` | **off** |
| `browser` | `browser_tabs`, `browser_read`, `browser_eval_js` | **off** |
| `computer` | `computer` | `computer` profile only |

Every tool has an input and output schema, MCP annotations (`readOnlyHint`, `destructiveHint`, …)
and stable error codes such as `STALE_REF`, `BLOCKED_BY_POLICY` or `USER_ACTIVE`. Full reference,
generated from the code: [docs/tools.md](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/tools.md).

## Safety defaults

| Control | Default |
|---|---|
| Profile | `standard`; `scripting` and `browser` toolsets off |
| Confirmations | Required for destructive actions (shell, AppleScript, JS eval, running a shortcut, force quit, `defaults` write, process kill), showing the exact command |
| App deny-list | Keychain Access, Passwords, 1Password, Bitwarden, LastPass, Dashlane, KeePassXC |
| Window deny-list | The Privacy & Security pane of System Settings |
| Terminal rule | Typing into Terminal, iTerm2, Ghostty, Warp and similar apps is blocked unless the `shell` tool is enabled |
| Secure fields | Values never returned; typing refused unless `allow_secure_fields = true` |
| Rate limit | 120 input actions per minute |
| Yield to user | On: input pauses while you are using the mouse or keyboard |
| Audit log | `~/Library/Logs/AXPilot/audit.jsonl`, arguments redacted |
| Client pairing | The host asks once per client app before it may connect |
| Kill switch, indicator | ⌃⌥⌘. stops the agent (press again to resume); a frame and pill show while it acts *(implemented, not yet run on a Mac)* |
| Telemetry | **None** |

Screen content is untrusted input for an agent. Read the
[security model](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/security.md) before enabling
`scripting` or `browser`. To report a vulnerability, see [SECURITY.md](https://github.com/vansh3503/Mac-Mcp/blob/main/SECURITY.md).

## How AXPilot compares

AXPilot is the macOS counterpart of [Windows-MCP](https://github.com/CursorTouch/Windows-MCP), with
a tool surface (and an optional naming profile) that lets Windows-MCP prompts carry over.
Other good projects in this space include [MacOS-MCP](https://github.com/CursorTouch/MacOS-MCP)
(Python + PyObjC, the closest peer), [Peekaboo](https://github.com/steipete/Peekaboo) (Swift),
[mcp-server-macos-use](https://github.com/mediar-ai/mcp-server-macos-use) (Swift) and
[cua-driver](https://github.com/trycua/cua). AXPilot's focus:

1. **Permissions that survive updates**: a frozen launcher holds the grant for every client.
2. **Safety and privacy as the headline**: no telemetry, pairing, kill switch, confirmations, audit.
3. **Spec-first MCP**: schemas, annotations, structured output, deterministic tool order, stateless handles.
4. **Integration kit and evals**: framework examples tested in CI, an Agent Skill, published eval results.

## Roadmap

About 12 weeks to 1.0: `v0.1` alpha (engine and core tools in direct mode; 0.1.0a1 is out),
`v0.2` beta (launcher, host app, safety), `v0.3` public preview (one-line installer, `connect` for
12 clients, MCP Registry), `v0.9` (full toolsets, OCR, frameworks, evals), then `v1.0`. Details and the open questions are in the
[project plan](https://github.com/vansh3503/Mac-Mcp/blob/main/docs/PLAN.md); decisions are recorded as
[ADRs](https://github.com/vansh3503/Mac-Mcp/tree/main/docs/adr).

## Contributing

Contributions are welcome, and most of the code (server, policy, perception) runs and is tested on
Linux. Start with [CONTRIBUTING.md](https://github.com/vansh3503/Mac-Mcp/blob/main/CONTRIBUTING.md);
AI coding agents should read [AGENTS.md](https://github.com/vansh3503/Mac-Mcp/blob/main/AGENTS.md).
Please follow the [Code of Conduct](https://github.com/vansh3503/Mac-Mcp/blob/main/CODE_OF_CONDUCT.md).

## License

[Apache-2.0](https://github.com/vansh3503/Mac-Mcp/blob/main/LICENSE).

## Acknowledgements

AXPilot is inspired by [Windows-MCP](https://github.com/CursorTouch/Windows-MCP) by CursorTouch,
whose tool design and install experience set the bar for desktop MCP servers. Thanks also to the
authors of [PyObjC](https://github.com/ronaldoussoren/pyobjc), the
[MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) and the projects listed above.

"Mac" and "macOS" are trademarks of Apple Inc. Claude and Claude Code are trademarks of Anthropic.
Other product and framework names mentioned here (such as Cursor, VS Code, LangChain and CrewAI) are
trademarks of their respective owners and are used only to describe compatibility. AXPilot is an
independent project and is not affiliated with or endorsed by any of them.

The demo video's voiceover was made with ElevenLabs and its music comes from the HeyGen audio library.
