Metadata-Version: 2.4
Name: omnipilot-agent
Version: 0.1.0
Summary: Your computer, piloted by any LLM. A cross-platform, provider-agnostic, permission-safe computer-use agent (OpenAI, Anthropic, Gemini, NVIDIA NIM and 10+ more).
Author: xrchris7
License: MIT
Project-URL: Homepage, https://github.com/xrchris7/OmniPilot
Project-URL: Repository, https://github.com/xrchris7/OmniPilot
Project-URL: Documentation, https://github.com/xrchris7/OmniPilot#-documentation
Project-URL: Bug Tracker, https://github.com/xrchris7/OmniPilot/issues
Project-URL: Changelog, https://github.com/xrchris7/OmniPilot/blob/main/CHANGELOG.md
Keywords: ai,agent,llm,openai,anthropic,claude,gemini,nvidia,nim,computer-use,automation,open-interpreter,assistant
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Desktop Environment
Classifier: Topic :: Office/Business
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai>=1.40.0
Requires-Dist: anthropic>=0.34.0
Requires-Dist: requests>=2.31.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: rich>=13.7.0
Requires-Dist: typer>=0.12.0
Requires-Dist: pillow>=10.0.0
Provides-Extra: gui
Requires-Dist: pyautogui>=0.9.54; extra == "gui"
Requires-Dist: pyperclip>=1.8.2; extra == "gui"
Requires-Dist: pygetwindow>=0.0.9; platform_system == "Windows" and extra == "gui"
Provides-Extra: system
Requires-Dist: psutil>=5.9.0; extra == "system"
Requires-Dist: plyer>=2.1.0; extra == "system"
Provides-Extra: server
Requires-Dist: fastapi>=0.110.0; extra == "server"
Requires-Dist: uvicorn>=0.29.0; extra == "server"
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.6; extra == "mcp"
Provides-Extra: all
Requires-Dist: pyautogui>=0.9.54; extra == "all"
Requires-Dist: pyperclip>=1.8.2; extra == "all"
Requires-Dist: pygetwindow>=0.0.9; platform_system == "Windows" and extra == "all"
Requires-Dist: psutil>=5.9.0; extra == "all"
Requires-Dist: plyer>=2.1.0; extra == "all"
Requires-Dist: fastapi>=0.110.0; extra == "all"
Requires-Dist: uvicorn>=0.29.0; extra == "all"
Requires-Dist: mcp<2,>=1.6; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: ruff>=0.5.0; extra == "dev"
Dynamic: license-file

<div align="center">

# 🛩️ OmniPilot

### Your computer, piloted by any LLM.

A cross-platform, open-source **computer-use agent** — think [Open Interpreter](https://github.com/OpenInterpreter/open-interpreter) meets Anthropic "computer use" — that lets an LLM **open apps, control the mouse and keyboard, read the screen, run shell commands, write and execute Python, manage files, browse the web and automate your whole machine**… behind a human-in-the-loop **permission system ("allowances")** and through **one unified interface** for OpenAI, Anthropic, Google Gemini, NVIDIA NIM and a dozen more providers, including local models.

[![CI](https://github.com/xrchris7/OmniPilot/actions/workflows/ci.yml/badge.svg)](https://github.com/xrchris7/OmniPilot/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![Platform Windows | macOS | Linux](https://img.shields.io/badge/platform-Windows%20%7C%20macOS%20%7C%20Linux-lightgrey.svg)](#-installation)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
[![PyPI](https://img.shields.io/pypi/v/omnipilot-agent?color=blue&label=PyPI)](https://pypi.org/project/omnipilot-agent/)

</div>

---

## 🎬 See it in action

No API key needed to try it — OmniPilot ships with a fully offline, scripted demo:

```bash
pip install omnipilot-agent
omnipilot demo
```

The agent plans a task, asks for approval before writing files or running shell
commands, executes them and reports back — all rendered locally with **zero network
calls**:

<p align="center">
  <img src="assets/terminal-demo.gif" alt="OmniPilot terminal demo: the agent scaffolds a Python quizzer project, showing approval prompts, tool output and the final summary" width="900"/>
</p>

Prefer a browser? `omnipilot serve` gives you a dark chat UI with **live streaming**
tool activity, approval decisions and token/cost accounting:

<p align="center">
  <img src="assets/web-ui.png" alt="OmniPilot web UI showing the agent's plan for organizing a Downloads folder" width="900"/>
</p>

<p align="center">
  <img src="assets/web-ui-activity.png" alt="OmniPilot web UI mid-run, streaming tool activity as it happens" width="900"/>
</p>

You can even drive the web UI offline with the bundled streaming demo server:

```bash
python examples/demo_streaming_server.py --port 8801
# then open http://127.0.0.1:8801
```

---

## 📖 Table of contents

- [See it in action](#-see-it-in-action)
- [Why OmniPilot](#-why-omnipilot)
- [Features](#-features)
- [Demos / example tasks](#-example-tasks)
- [MCP: use any Model Context Protocol server](#-mcp-use-any-model-context-protocol-server)
- [Installation](#-installation)
- [Getting an API key](#-getting-an-api-key)
- [Quickstart](#-quickstart)
- [Supported LLM providers](#-supported-llm-providers)
- [The permission system (allowances)](#-the-permission-system-allowances)
- [Tool catalog](#-tool-catalog)
- [CLI reference](#-command-line-reference)
- [Configuration](#-configuration)
- [Local web/API server](#-local-webapi-server)
- [Using OmniPilot from Python](#-using-omnipilot-from-python)
- [Writing your own tools (plugins)](#-writing-your-own-tools-plugins)
- [Architecture](#-architecture)
- [Safety & security](#-safety--security)
- [Documentation index](#-documentation-index)
- [Roadmap](#-roadmap)
- [Contributing](#-contributing)
- [License & disclaimer](#-license--disclaimer)

---

## 💡 Why OmniPilot?

Most agent projects are locked to **one** model vendor, run on **one** OS, or execute
commands with **no safety layer**. OmniPilot gives you:

- 🤝 **Bring your own model** — 13 providers out of the box and *any* OpenAI-compatible
  endpoint. Switch mid-conversation with `/provider groq` or `/model ...`.
- 🖥️ **Real computer control** — shell, persistent Python, files, apps, mouse, keyboard,
  screenshots, clipboard, processes, notifications and web requests.
- 🙋 **You stay in charge** — every risky action is classified (`low / medium / high /
  critical`) and either auto-approved, prompted, or blocked. Critical patterns
  (`rm -rf /`, drive formatting, fork bombs…) are blocked **even in YOLO mode**.
- 🪟🍎🐧 **Genuinely cross-platform** — Windows (cmd + PowerShell), macOS and Linux each
  get first-class command and application handling.
- 👁️ **Vision-based computer use** — screenshots are sent to vision-capable models
  (GPT-4o, Claude, Gemini, vision NIMs…) so the agent can *see* what it is doing.
- 🔌 **Hackable** — tiny plugin API, clean provider adapter layer, JSON API server,
  100% Python, MIT licensed.

---

## ✨ Features

### Agents
- A mature tool-calling loop (plan → observe → act → verify → summarize)
- Native function-calling adapters — **no prompt-parser hacks** — for each vendor dialect
- Multiple tool calls per turn, automatic context compaction, token & cost accounting
- Persistent sessions (auto-saved, resumable across restarts) with `/save`, `/load`,
  `/sessions`, `/undo`
- Slash-command REPL plus one-shot mode for scripts/CI
- Per-session permission grants ("always allow this tool", "always allow this exact action")
- Built-in local **web UI + REST API** (`omnipilot serve`) with SSE streaming
- Live provider/model switching without losing the conversation
- 🔌 **MCP client built in** — connect any [Model Context Protocol](https://modelcontextprotocol.io/)
  server (stdio, streamable-HTTP or SSE) and use its tools as first-class OmniPilot
  tools, behind the same permission system

### Computer control (41 built-in tools)
| Domain | Examples |
|---|---|
| Shell | `shell.run_command` (cmd / PowerShell / bash / zsh, timeouts, background processes) |
| Python | `python.run_python` — persistent interpreter, variables survive between calls |
| Files | read, write, targeted find/replace edit, list, move, copy, delete, glob search, content grep |
| Apps | open/close/list applications, open files in default apps (Start Menu, `open -a`, xdg) |
| Screen | screenshots → vision models, screen size, image-on-screen lookup |
| GUI | click, double/right click, move, drag, scroll, type (unicode-safe), keys, hotkeys |
| Browser/ web | open URLs & searches in your browser, fetch pages, arbitrary HTTP requests, downloads, keyless web search |
| System | system info, drives, env vars (secret masking), process list/kill, native notifications |
| Clipboard | read & write |

See the full catalog with arguments in [docs/tools.md](docs/tools.md).

### Safety
- Dynamic risk classification per command, code snippet, path and process name
  (60+ patterns, OS-aware — see [`danger.py`](src/omnipilot/permission/danger.py))
- Three modes: **ask** (default), **auto** (safe reads run unattended), **yolo**
- Allow / deny rules with globs and command prefixes (`shell:git *`, `fs.write_file:~/Documents/*`)
- Filesystem **sandbox roots** restricting where the agent may write
- Non-interactive / server mode **fails closed**
- Full **audit log** of every approved/denied action (`~/.omnipilot/logs/audit.log`)
- PyAutoGUI failsafe: slam the mouse into a screen corner to abort a GUI runaway

---

## 🤖 Example tasks

```
# Productivity
"Open Chrome, go to Gmail, and tell me how many unread emails I have"
"Find every PDF in my Downloads folder, move them into Documents\PDFs by year"
"Open the spreadsheet Q3-sales.xlsx and sum column D"

# Development
"Look at this repo, run the tests, fix whatever is failing, and commit"
"Create a FastAPI todo service in ./todo-api, install deps, run it, and screenshot it"
"Refactor every print() in this project into logging calls, then show me the diff"

# System administration
"Show me what's using all my RAM and close the worst offender"
"Install VLC with winget/brew/apt and pin it to my taskbar/dock"
"Rename all photos in this folder to the pattern YYYY-MM-DD_NNN.jpg"

# Research
"Search the web for the latest NVIDIA driver, download it to Downloads"
"Read the docs at <url> and write a quickstart example using requests"
"Set up a new Python project called inventory-service with pytest and ruff"
```

One-shot mode is great in scripts:

```bash
omnipilot --provider anthropic "compress every PNG in this folder that's over 2 MB"
omnipilot -p ollama -m llama3.1 --mode auto "summarize today's git log"
```

---

## 📦 Installation

You need **Python 3.10 or newer** ([python.org](https://www.python.org/downloads/),
or `winget install Python.Python.3.12` / `brew install python` / `apt install python3 python3-pip python3-venv`).

```bash
# Core install (everything except GUI automation extras)
pip install omnipilot-agent

# Everything — mouse/keyboard, notifications, system stats, web server
pip install "omnipilot-agent[all]"
```

Install from source (this repository):

```bash
git clone https://github.com/xrchris7/OmniPilot.git
cd OmniPilot
pip install -e ".[all]"          # or: make install-all
```

Optional extras: `[gui]` (pyautogui/pyperclip), `[system]` (psutil/plyer),
`[server]` (FastAPI/uvicorn), `[dev]` (pytest/ruff).

**Platform notes:**

- **Windows 11/10** — nothing extra needed; run PowerShell/cmd *as yourself*.
- **macOS** — grant **Accessibility** and **Screen Recording** permission to your
  terminal app (System Settings → Privacy & Security) for GUI/screen tools.
- **Linux** — screenshots/GUI need an X11/Wayland desktop plus
  `sudo apt install python3-tk scrot` (Wayland users may need `ydotool`/pipewire portal support).
- Headless/SSH servers: shell, Python, files, web and system tools all work;
  GUI/screen tools need a desktop session.

Detailed walkthrough: [docs/installation.md](docs/installation.md).

---

## 🔑 Getting an API key

You only need a key for the provider(s) you use. Quick links:

| Provider | Get a key |
|---|---|
| OpenAI | https://platform.openai.com/api-keys |
| Anthropic Claude | https://console.anthropic.com/settings/keys |
| Google Gemini (free tier!) | https://aistudio.google.com/app/apikey |
| NVIDIA NIM (free credits) | https://build.nvidia.com/explore/discover |
| Groq (free tier, very fast) | https://console.groq.com/keys |
| OpenRouter | https://openrouter.ai/keys |
| DeepSeek | https://platform.deepseek.com/api_keys |
| Mistral | https://console.mistral.ai/api-keys |
| xAI Grok | https://console.x.ai/ |
| Together AI | https://api.together.xyz/settings/api-keys |
| Ollama (local, free) | https://ollama.com/download |
| LM Studio (local, free) | https://lmstudio.ai/ |

Then either export the variable…

```bash
export ANTHROPIC_API_KEY=sk-ant-...        # Windows PowerShell: $env:ANTHROPIC_API_KEY = "..."
```

…or copy `.env.example` to `.env`, fill in keys, and OmniPilot loads it automatically.

Run the built-in doctor at any time:

```bash
omnipilot doctor
```

---

## ⚡ Quickstart

No API key yet? Try the fully offline demo first (scripted agent, real tools, real
approval flow):

```bash
omnipilot demo
```

Then connect a model:

```bash
# 1. Create a starter config (~/.omnipilot/config.yaml)
omnipilot config init

# 2. Pick a provider/model for this session and start the REPL
omnipilot --provider anthropic --model claude-sonnet-4-5-20250929
# …or…
omnipilot -p gemini -m gemini-2.0-flash
# …or fully local…
ollama pull llama3.1
omnipilot -p ollama -m llama3.1
```

You'll see the banner and a prompt. Try:

```
❯ What can you do on this machine?
❯ Take a screenshot and describe what's on my screen
❯ Create a folder called Reports on the Desktop and put a file in it that says hello
❯ Open Calculator and type 123 * 456
❯ /mode auto
❯ list all python processes
❯ /save
❯ /exit
```

Run one task and exit (great for automation):

```bash
omnipilot "create a 7-day markdown study plan for the AWS SAA exam in ./study-plan.md"
```

**First run tip:** keep `--mode ask` (the default). Watch the approval prompts; answer:

- `y` — yes, once
- `n` — no
- `a` — always allow **this tool** for the session
- `e` — always allow **this exact action** for the session

---

## 🔌 Supported LLM providers

| Key | Provider | Vision | Key variable | Default base URL |
|---|---|:-:|---|---|
| `openai` | OpenAI GPT-4o / 4.1 / o-series | ✅ | `OPENAI_API_KEY` | api.openai.com |
| `anthropic` | Claude Opus/Sonnet/Haiku | ✅ | `ANTHROPIC_API_KEY` | api.anthropic.com |
| `gemini` | Google Gemini 2.x / 1.5 (REST) | ✅ | `GEMINI_API_KEY` | generativelanguage.googleapis.com |
| `nvidia` | NVIDIA NIM catalog (Llama, Qwen, Nemotron, vision models) | ✅ | `NVIDIA_API_KEY` | integrate.api.nvidia.com/v1 |
| `groq` | Groq ultra-fast inference | ✅ | `GROQ_API_KEY` | api.groq.com/openai/v1 |
| `openrouter` | Hundreds of models via one key | ✅ | `OPENROUTER_API_KEY` | openrouter.ai/api/v1 |
| `deepseek` | DeepSeek V3/R1 | ❌ | `DEEPSEEK_API_KEY` | api.deepseek.com |
| `mistral` | Mistral Large/Ministral | ❌ | `MISTRAL_API_KEY` | api.mistral.ai/v1 |
| `xai` | Grok 2 / Grok vision | ✅ | `XAI_API_KEY` | api.x.ai/v1 |
| `together` | Together AI catalog | ✅ | `TOGETHER_API_KEY` | api.together.xyz/v1 |
| `ollama` | Local Ollama models | depends | — | localhost:11434/v1 |
| `lmstudio` | Local LM Studio server | depends | — | localhost:1234/v1 |
| `custom` | **Any** OpenAI-compatible endpoint | configurable | `CUSTOM_API_KEY` | `CUSTOM_BASE_URL` |

Change provider/model in the REPL at any time:

```
/provider nvidia
/model meta/llama-3.3-70b-instruct
```

List models and even fetch a provider's live catalog:

```bash
omnipilot models
omnipilot models --provider nvidia --remote
```

Full per-provider setup: [docs/providers.md](docs/providers.md).

---

## 🔌 MCP: use any Model Context Protocol server

OmniPilot is an **MCP client**. Connect external
[MCP](https://modelcontextprotocol.io/) servers and the model can call their tools
exactly like the 41 built-ins — each one is named `mcp.<server>.<tool>` and flows
through the **same permission engine** (risk modes, allow/deny rules, approval
prompts, audit log). All three transports are supported: local **stdio**
subprocesses, **streamable HTTP**, and legacy **SSE**.

```bash
pip install "omnipilot-agent[mcp]"
```

```yaml
# ~/.omnipilot/config.yaml
mcp:
  enabled: true
  servers:
    fs:                                                  # -> mcp.fs.*
      command: npx
      args: ["-y", "@modelcontextprotocol/server-filesystem", "/home/me/projects"]
      risk: medium
    fetch:                                               # -> mcp.fetch.*
      url: https://example.com/mcp-server/mcp
      transport: streamable-http
      headers:
        Authorization: "Bearer ${MCP_TOKEN}"
```

Then inspect and smoke-test from the CLI:

```bash
omnipilot mcp list          # show configured servers, status and tool counts
omnipilot mcp test fs       # start the server and list its tools
omnipilot mcp test fs read_file '{"path": "README.md"}'   # call one tool
```

Remote-server failures never abort startup — the offending server is reported as
`error` while every other server and the built-in tools keep working.
Full guide: [docs/mcp.md](docs/mcp.md).

---

## 🛡️ The permission system (allowances)

### Modes

| Mode | Behavior |
|---|---|
| **`ask`** (default) | Prompts for everything not explicitly allowed; low-risk read-only tools auto-run if `auto_approve_low: true` |
| **`auto`** | Low-risk actions run automatically; medium/high/critical prompt |
| **`yolo`** | Nothing prompts — except the critical safety list (which still blocks unless you explicitly set `allow_critical: true`) |

```bash
omnipilot --mode auto
omnipilot --yolo          # maximum autonomy, maximum responsibility
```

### Rules (in `~/.omnipilot/config.yaml`)

```yaml
permission:
  mode: ask
  allow:
    - fs.read_file              # exact tool name
    - fs.*                      # globs
    - "shell:git status"        # exact-ish command prefix
    - "shell:npm run *"         # wildcard command prefixes
    - "python:print(*"          # python prefix
    - "fs.write_file:~/Documents/*"
  deny:
    - "shell:*format *"
    - "shell:*mkfs*"
  sandbox_paths:                # restrict writes to these roots
    - ~/projects
    - ~/Downloads
  allow_critical: false         # keep false!
```

- **Deny rules win** over allow rules; the built-in critical list wins over everything.
- Rules you grant during a session (`a` / `e`) are session-scoped and clearly logged.
- The server / `--no-input` mode cannot answer prompts, so unlisted risky actions are
  denied automatically — **fail closed**.
- Every decision is written to `~/.omnipilot/logs/audit.log`.

Read more: [docs/permissions.md](docs/permissions.md) and [docs/safety.md](docs/safety.md).

---

## 🧰 Tool catalog

<details>
<summary><b>41 tools — click to expand the full list</b></summary>

| Tool | What it does | Base risk |
|---|---|:-:|
| `shell.run_command` | Run a shell command (cmd/PowerShell/bash/zsh), sync or background | high |
| `python.run_python` | Persistent in-process Python interpreter | high |
| `fs.read_file` | Read a text file | low |
| `fs.write_file` | Create/overwrite/append files (creates parent dirs) | medium |
| `fs.edit_file` | Unique/find-replace edits with ambiguity protection | medium |
| `fs.list_directory` | Directory listing with sizes/dates | low |
| `fs.make_directory` | mkdir -p | low |
| `fs.move_file` | Move/rename | medium |
| `fs.copy_file` | Copy files or directories | low |
| `fs.delete_file` | Delete files/trees (permanent) | medium |
| `fs.search_files` | Recursive glob/name search | low |
| `fs.grep` | Search file contents (regex/literal) | low |
| `apps.open_application` | Launch apps by friendly name or path | low |
| `apps.close_application` | Quit apps (graceful/force) | medium |
| `apps.open_file` | Open a file/folder in its default app | low |
| `apps.list_applications` | Running apps/processes | low |
| `browser.open_url` | Open URLs/web searches in the default browser | low |
| `web.fetch_url` | Fetch a page and convert HTML → readable text | low |
| `web.http_request` | Any HTTP method with JSON/form bodies | medium |
| `web.download_file` | Stream a download to disk | medium |
| `web.search_web` | Keyless DuckDuckGo web search | low |
| `screen.screenshot` | Capture screen (→ vision model) as PNG | low |
| `screen.get_screen_size` | Screen resolution | low |
| `screen.locate_image` | Find a PNG template on screen | low |
| `gui.click` / `gui.double_click` / `gui.right_click` | Mouse clicks at x,y | medium |
| `gui.move_mouse` / `gui.drag` / `gui.scroll` | Pointer motion | medium |
| `gui.type_text` | Type (unicode-safe paste path) | medium |
| `gui.press_key` / `gui.hotkey` | Keys and chords (ctrl+c, alt+tab…) | medium |
| `clipboard.get_clipboard` / `set_clipboard` | Clipboard access | medium / low |
| `processes.list_processes` | Process table (CPU/mem) | low |
| `processes.kill_process` | Terminate by PID/name | high |
| `system.get_system_info` | OS/CPU/memory/disk/battery summary | low |
| `system.get_environment_variable` | Env vars with secret masking | medium |
| `system.list_drives` | Drives/volumes | low |
| `notify.send_notification` | Native toast notifications | low |

</details>

Every tool also carries a full JSON-Schema argument description — inspect with
`omnipilot serve` → `GET /v1/tools`, or type `/tools` in the REPL.
Full argument reference: [docs/tools.md](docs/tools.md).

---

## 🖥️ Command line reference

```
omnipilot [OPTIONS] [TASK ...]            # REPL, or one-shot when you type a task
omnipilot chat "your task" --provider ... # explicit one-shot command
omnipilot doctor                          # check keys, packages, OS features
omnipilot models [--provider X] [--remote]
omnipilot config init | path
omnipilot sessions list | resume <id>
omnipilot serve [--host 127.0.0.1] [--port 8765] [--mode auto]
omnipilot version
```

Key flags: `-p/--provider`, `-m/--model`, `--mode {ask,auto,yolo}`, `--yolo`, `--safe`,
`--vision {on,off,auto}`, `-c/--config`, `-w/--workdir`, `--max-iterations`,
`-i/--image`, `-r/--resume`, `--system`, `--no-input`.

REPL slash commands: `/help /mode /yolo /safe /allow /clear /undo /provider /model
/vision /tools /image /screenshot /save /load /sessions /usage /doctor /exit`.

Full reference: [docs/cli.md](docs/cli.md).

---

## ⚙️ Configuration

Resolution order (highest wins): **CLI flags → environment variables → `--config`
file → `./omnipilot.yaml` → `~/.omnipilot/config.yaml` → built-in defaults**.

The fully commented reference is [`config.example.yaml`](config.example.yaml). Highlights:

```yaml
default_provider: openai
default_model: gpt-4o
vision: auto                 # auto = only send screenshots to vision-capable models

agent:
  max_iterations: 25
  temperature: 0.2
  tool_output_limit: 12000

permission: {mode: ask, sandbox_paths: [], allow_critical: false}

tools:
  enabled: ["*"]
  disabled: ["gui.*"]        # e.g. disable GUI automation entirely

providers:
  nvidia: {model: meta/llama-3.3-70b-instruct}
  ollama: {model: llama3.1, base_url: http://localhost:11434/v1}
```

Details: [docs/configuration.md](docs/configuration.md).

---

## 🌐 Local web/API server

```bash
pip install "omnipilot-agent[server]"
omnipilot serve --host 127.0.0.1 --port 8765
```

Open http://127.0.0.1:8765 for the built-in chat page, or use the JSON API:

```bash
curl -X POST http://127.0.0.1:8765/v1/agent/run \
  -H "Content-Type: application/json" \
  -d '{"message": "create a file hello.txt with content hi", "mode": "yolo"}'
```

```json
{
  "ok": true,
  "answer": "I created hello.txt …",
  "session_id": "20260912-091500-ab12cd",
  "provider": "openai", "model": "gpt-4o",
  "tool_runs": 3, "input_tokens": 1840, "output_tokens": 212,
  "estimated_cost_usd": 0.0067
}
```

Prefer streaming? `POST /v1/agent/stream` returns a `text/event-stream` (SSE) feed
emitting `assistant_text`, `tool_started`, `tool_finished`, `tool_denied`, `info`,
`error` and a final `done` event — the bundled web UI consumes exactly this stream
to render live tool activity.

Endpoints: `GET /health`, `GET /v1/tools`, `POST /v1/agent/run`,
`POST /v1/agent/stream`, `GET /v1/sessions`, `GET /v1/sessions/{id}`.
Interactive OpenAPI docs at `/docs`.

> ⚠️ Bind to `0.0.0.0` only on trusted networks; without a terminal the server fails
> closed — unresolved approvals are denied.

---

## 🐍 Using OmniPilot from Python

```python
from omnipilot.config.settings import load_settings
from omnipilot.providers.registry import build_provider
from omnipilot.tools.registry import ToolRegistry
from omnipilot.permission.policy import PermissionPolicy
from omnipilot.agent.loop import Agent, NullUI

settings = load_settings()
registry = ToolRegistry.build_default(settings)
provider = build_provider(settings, "openai", "gpt-4o-mini")
policy = PermissionPolicy(settings, non_interactive=True,
                          prompt=lambda **kw: "y")   # auto-confirm in scripts
agent = Agent(
    settings=settings,
    provider=provider,
    provider_name="openai",
    registry=registry,
    policy=policy,
    model=provider.model,
    ui=NullUI(),
)

print(agent.run("Create a file notes.txt containing today's date."))
print("tokens:", agent.usage)
```

A complete scripted example ships in [`examples/use_as_library.py`](examples/use_as_library.py).

---

## 🧩 Writing your own tools (plugins)

A tool is a small class; the entry-point group `omnipilot.tools` auto-registers plugins.

```python
# my_slack_tool.py
from omnipilot.tools.base import Tool, ToolContext, ToolResult

class SlackPost(Tool):
    name = "slack.post"
    description = "Post a message to a Slack channel via webhook."
    risk = "medium"
    parameters = {
        "channel": {"type": "string"},
        "text": {"type": "string"},
    }
    required = ["channel", "text"]

    def run(self, context: ToolContext, *, channel: str, text: str) -> ToolResult:
        import requests, os
        requests.post(os.environ["SLACK_WEBHOOK"],
                      json={"channel": channel, "text": text})
        return ToolResult.ok(f"Posted to {channel}")
```

Register it in your package's `pyproject.toml`:

```toml
[project.entry-points."omnipilot.tools"]
slack = "my_slack_tool:SlackPost"
```

See [`examples/custom_tool.py`](examples/custom_tool.py) and
[docs/architecture.md](docs/architecture.md).

---

## 🏛️ Architecture

```mermaid
flowchart TD
    U[User · REPL / CLI / Web API] --> CLI[cli.py · sessions · REPL]
    CLI --> AGENT[Agent loop · agent/loop.py]
    AGENT -->|normalized messages + tool schemas| AD[Provider adapter]
    subgraph Providers
        AD --> OAI[OpenAI-compatible<br/>OpenAI · NVIDIA · Groq · OpenRouter<br/>DeepSeek · Mistral · xAI · Together · Ollama]
        AD --> ANT[Anthropic Claude]
        AD --> GEM[Gemini REST]
    end
    AGENT --> POL{PermissionPolicy<br/>risk assessment · rules · prompts}
    POL -->|approved| TOOLS[ToolRegistry · 41 tools]
    TOOLS --> SHELL[shell / python]
    TOOLS --> FS[files / apps / browser]
    TOOLS --> GUI[screen / mouse / keyboard / clipboard]
    TOOLS --> WEB[web / processes / system / notify]
    TOOLS -->|mcp.&lt;server&gt;.&lt;tool&gt;| MCP[MCP client · stdio / HTTP / SSE]
    TOOLS -->|ToolResult + optional image| AGENT
    POL --> AUDIT[(audit.log)]
    AGENT --> SESS[(sessions/*.json)]
```

The agent uses a **single internal message format**; provider adapters translate to each
vendor's dialect (OpenAI tools, Anthropic tool_use blocks, Gemini functionDeclarations),
including multimodal screenshot messages. Adding a provider = one ~150-line adapter.

---

## 🧯 Safety & security

- OmniPilot can do **anything your user account can do**. Treat it like giving someone
  remote shell access — because it effectively is.
- Start in `ask` mode. Switch to `auto` only for routine work; reserve `yolo` for
  disposable VMs, containers or throwaway accounts.
- Don't paste long-lived secrets into tasks; env vars containing `KEY`/`TOKEN`/`SECRET`
  are masked when read by the agent and redacted from logs.
- Review `~/.omnipilot/logs/audit.log`. Use `sandbox_paths` for untrusted tasks.
- Consider running inside a VM/container (Windows Sandbox, macOS VM, Docker with a
  desktop) for fully autonomous experiments.
- Report vulnerabilities privately — see [SECURITY.md](SECURITY.md).

---

## 📚 Documentation index

| Document | Contents |
|---|---|
| [docs/installation.md](docs/installation.md) | Per-OS install, extras, permissions, virtualenvs |
| [docs/quickstart.md](docs/quickstart.md) | Guided first session with examples |
| [docs/providers.md](docs/providers.md) | Every provider, key setup, model IDs, local models, custom gateways |
| [docs/configuration.md](docs/configuration.md) | All YAML keys and env variables explained |
| [docs/permissions.md](docs/permissions.md) | Modes, rule grammar, sandboxes, audit logs |
| [docs/tools.md](docs/tools.md) | Complete tool & argument reference |
| [docs/mcp.md](docs/mcp.md) | Connecting MCP servers (stdio / HTTP / SSE), risk, troubleshooting |
| [docs/cli.md](docs/cli.md) | Every command, flag and slash command |
| [docs/architecture.md](docs/architecture.md) | Internals, message format, writing adapters & plugins |
| [docs/safety.md](docs/safety.md) | Threat model, safe operating procedures |
| [docs/faq.md](docs/faq.md) | Troubleshooting |

---

## 🗺️ Roadmap

- [x] 13 providers + custom OpenAI-compatible endpoints
- [x] 41 cross-platform computer-use tools incl. vision + GUI
- [x] Risk-aware permission engine, rules, sandboxes, audit log
- [x] Sessions, REPL, CLI, web UI + JSON API + SSE streaming, plugin entry points
- [x] MCP client (stdio / streamable-HTTP / SSE), tools gated by the permission engine
- [x] Offline `omnipilot demo` for zero-key evaluation
- [ ] Optional Docker / Windows-Sandbox execution backends
- [ ] MCP (Model Context Protocol) client for external MCP tools
- [ ] Voice I/O and accessibility-oriented automation
- [ ] Workflow recording (watch → generate a reusable script)
- [ ] Multi-agent planning and parallel tool execution
- [ ] Windows UI Automation / macOS Accessibility element tree (selector-based, coordinate-free)

Issues/PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).

## 🤝 Contributing

Bug reports, new danger-pattern rules, provider adapters and tools all welcome.
Quick local loop: `make dev` → `make test` → `make lint`. Please add tests and run
ruff. By contributing you agree your work ships under the MIT license.

---

## 📄 License & disclaimer

MIT © 2026 [xrchris7](https://github.com/xrchris7).

**OmniPilot executes model-generated commands on real hardware.** It is provided
"as is", without warranty. You are responsible for reviewing actions, securing your
keys, and for anything the agent does on your machines. It is intended for legitimate
automation on systems you own or are authorized to administer.
