Metadata-Version: 2.4
Name: eye2byte
Version: 0.7.0
Summary: Desktop automation platform — screen capture, OCR, interaction, macros, events, and integrations
Author: wolverin0
License-Expression: MIT
Project-URL: Homepage, https://github.com/wolverin0/Eye2byte
Project-URL: Changelog, https://github.com/wolverin0/Eye2byte/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/wolverin0/Eye2byte/issues
Keywords: screen-capture,mcp,coding-agent,vision,context,desktop-automation,ocr,macros,events,powertoys
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: Pillow
Requires-Dist: fastmcp>=2.10
Provides-Extra: voice
Requires-Dist: openai-whisper; extra == "voice"
Provides-Extra: ui
Requires-Dist: customtkinter>=5.0; extra == "ui"
Requires-Dist: pystray; extra == "ui"
Provides-Extra: ocr
Requires-Dist: easyocr>=1.7; extra == "ocr"
Provides-Extra: capture
Requires-Dist: mss>=9.0; extra == "capture"
Provides-Extra: find
Requires-Dist: rapidfuzz>=3.0; extra == "find"
Provides-Extra: interact
Requires-Dist: pyautogui>=0.9; extra == "interact"
Provides-Extra: win32
Requires-Dist: comtypes>=1.2; extra == "win32"
Provides-Extra: bridges
Requires-Dist: websocket-client>=1.6; extra == "bridges"
Provides-Extra: uia
Requires-Dist: uiautomation>=2.0; extra == "uia"
Provides-Extra: all
Requires-Dist: openai-whisper; extra == "all"
Requires-Dist: customtkinter>=5.0; extra == "all"
Requires-Dist: pystray; extra == "all"
Requires-Dist: easyocr>=1.7; extra == "all"
Requires-Dist: mss>=9.0; extra == "all"
Requires-Dist: rapidfuzz>=3.0; extra == "all"
Requires-Dist: pyautogui>=0.9; extra == "all"
Requires-Dist: comtypes>=1.2; extra == "all"
Requires-Dist: websocket-client>=1.6; extra == "all"
Requires-Dist: uiautomation>=2.0; extra == "all"

<p align="center">
  <img src="eye2byte.png" width="120" alt="Eye2byte icon">
  <h1 align="center">Eye2byte</h1>
  <p align="center"><strong>Desktop automation platform for AI agents.<br>See the screen. Find elements. Click, type, record, automate.</strong></p>
</p>

<p align="center">
  <a href="https://pypi.org/project/eye2byte/"><img src="https://img.shields.io/pypi/v/eye2byte?color=blue&logo=pypi&logoColor=white" alt="PyPI"></a>
  <a href="#quick-start"><img src="https://img.shields.io/badge/python-3.10+-blue?logo=python&logoColor=white" alt="Python 3.10+"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License"></a>
  <a href="#platforms"><img src="https://img.shields.io/badge/platform-Windows-lightgrey" alt="Windows"></a>
  <a href="CHANGELOG.md"><img src="https://img.shields.io/badge/changelog-CHANGELOG.md-orange" alt="Changelog"></a>
  <a href="docs/USE_CASES.md"><img src="https://img.shields.io/badge/use_cases-guide-brightgreen" alt="Use Cases"></a>
  <a href="docs/CAPABILITIES_MAP.md"><img src="https://img.shields.io/badge/capabilities-map-informational" alt="Capabilities Map"></a>
  <img src="https://img.shields.io/badge/tests-passing-success" alt="Tests">
</p>

---

Eye2byte is a desktop automation platform that exposes MCP tools to AI coding agents. It captures your screen, finds UI elements via OCR/VLM, clicks and types, records macros, monitors system events, and integrates with PowerToys and EventGhost — all through the [Model Context Protocol](https://modelcontextprotocol.io).

```
Screen / Voice / Annotations  ──>  Vision Model + Whisper  ──>  Context Pack  ──>  Coding Agent
                                   (Ollama, Gemini,              (goal, errors,    (Claude Code,
                                    OpenRouter, Hyperbolic)       signals, next)     Codex, Gemini CLI)
```

Current reference docs:

- Capability maturity map: `docs/CAPABILITIES_MAP.md`
- Testing and CI contracts: `docs/testing.md`
- Release gate checklist: `docs/release-checklist.md`

---

## What Can Eye2byte Do?

### Perception — See the Screen
- **Screenshot capture** — full screen, active window, region, all monitors, or specific monitor
- **AI analysis** — sends screenshots to vision models, produces structured Context Packs
- **OCR** — extract text with coordinates via WinOCR (150ms) or EasyOCR fallback
- **Voice** — record narration, transcribe locally via Whisper, bundle with captures
- **Screen clips** — record video, extract keyframes, summarize sequences

### Interaction — Control the Desktop
- **Click, type, scroll, keypress** — full mouse/keyboard control at coordinates
- **Find and click** — describe what you want to click in plain text, OCR+VLM finds and clicks it
- **Confidence-scored matching** — retry logic, fail-safe abort below threshold, deterministic ranking

### Window Management — Arrange Your Workspace
- **List, focus, minimize** any window by name or process
- **Pin/unpin** windows as always-on-top
- **Move and resize** to exact coordinates
- **Set transparency** on any window (0-255 alpha)
- **Get pixel color** at any screen coordinate

### Desktop Automation — Fill PowerToys Gaps
- **Hot corners** — trigger actions when mouse hits screen corners
- **Text expander** — type abbreviations, get expanded text (supports `{today}`, `{time}`, `{clipboard}`)
- **Focus lock** — prevent apps from stealing focus
- **Distraction dimmer** — dim all windows except the active one
- **Auto-tile** — i3-style window tiling (columns, rows, grid, master+stack)
- **App-specific hotkeys** — remap keys only when a specific app is focused
- **Key-to-mouse** — map any keyboard key to mouse clicks

### Macro Recording — Record and Replay
- **Record** mouse clicks, keyboard input, and scroll events with timestamps
- **Replay** using the find engine (resilient to window repositioning) or raw coordinates
- **Visual assertions** — verify expected state after each step
- **Audit logs** — before/after screenshots with confidence scores for every action

### Event System — React to System Changes
- **10 event types**: window focus, window open/close, clipboard, global hotkeys, file system changes, USB devices, process start/stop, display changes, power state, volume
- **Webhook delivery** — POST events to any URL (Node-RED, custom server)
- **MCP polling** — buffer events and query them from your agent
- **Configurable** — listen to specific events, ignore the rest

### Integrations — Extend with PowerToys & EventGhost
- **PowerToys** — detect installation, control FancyZones layouts, check file locks, resize images, quick-preview files
- **EventGhost** — trigger events, read/write variables, execute scripts, bridge events via WebSocket (GPL-safe HTTP API, no imports)
- **Node-RED** — custom node for visual workflow automation, 7 example flows, UIBUILDER dashboard
- **CmdPal** — PowerToys Command Palette extension (C#/.NET 9) exposes all tools to millions of users

### System Control
- **Set volume** (0-100)
- **Text-to-speech** via Windows SAPI
- **Keep awake** — prevent sleep/display-off for a duration
- **Search history** — full-text search across all past Context Packs

---

## Quick Start

### 1. Install

```bash
pip install eye2byte[all]
```

Alternative installation methods:

```bash
# Isolated app install (recommended for CLI use)
pipx install "eye2byte[all]"

# uv (fast Python package manager)
uv tool install "eye2byte[all]"

# Latest main branch (before PyPI release catches up)
pip install "git+https://github.com/wolverin0/Eye2byte.git@main"
```

<details>
<summary>Granular install options</summary>

```bash
pip install eye2byte             # Core + MCP server (Pillow + fastmcp)
pip install eye2byte[voice]      # + local voice transcription (openai-whisper)
pip install eye2byte[ui]         # + control panel (customtkinter + pystray)
pip install eye2byte[ocr]        # + coordinate-aware OCR (winocr on Windows, easyocr fallback)
pip install eye2byte[capture]    # + optional mss in-memory capture backend
pip install eye2byte[find]       # + rapidfuzz matching acceleration
pip install eye2byte[interact]   # + mouse/keyboard automation (pyautogui)
pip install eye2byte[win32]      # + volume control, text-to-speech (comtypes)
pip install eye2byte[bridges]    # + EventGhost WebSocket bridge (websocket-client)
pip install eye2byte[uia]        # + Windows UI Automation helpers
pip install eye2byte[all]        # Everything
```

System prerequisites:

- `ffmpeg` is required for voice and clip workflows.
- Linux: OCR and screenshot stacks may require additional desktop packages (`tesseract`, `scrot`, compositor-specific deps).
- macOS: grant Accessibility and Screen Recording permissions for interaction/capture tools.
- Windows: PowerToys/EventGhost integrations are optional and can be added later.

</details>

### 2. Configure a vision provider

| Provider | Setup | Cost |
|----------|-------|------|
| **Ollama** | [Install Ollama](https://ollama.com), `ollama pull qwen3-vl:8b` | Free (local) |
| **Gemini** | Set `GEMINI_API_KEY` in `.env` | Free tier |
| **OpenRouter** | Set `OPENROUTER_API_KEY` in `.env` | Free models available |
| **Hyperbolic** | Set `HYPERBOLIC_API_KEY` in `.env` | Pay per use |

### 3. Run

```bash
eye2byte capture                   # Screenshot + analysis
eye2byte capture --voice           # + voice narration
eye2byte capture --mode window     # Active window only
eye2byte-ui                        # Launch control panel
```

---

## MCP Tools

Eye2byte exposes a broad MCP tool catalog. Any MCP-compatible agent can use them.

### Perception (8 tools)

| Tool | What it does |
|------|-------------|
| `capture_and_summarize` | Screenshot + vision AI analysis → Context Pack |
| `capture_with_voice` | Screenshot + voice recording + transcription |
| `record_clip_and_summarize` | Screen clip with keyframe extraction |
| `summarize_screenshot` | Analyze an existing image file |
| `transcribe_audio` | Local Whisper transcription |
| `get_screen_elements` | OCR with coordinates — find text positions |
| `get_recent_context` | Retrieve recent Context Pack summaries |
| `search_context_history` | Full-text search past observations |

### Interaction (6 tools)

| Tool | What it does |
|------|-------------|
| `find_element` | Find UI element by text/description (OCR+VLM, confidence scored) |
| `find_and_click` | Find element + click in one call |
| `click_element` | Click at screen coordinates |
| `type_text` | Type text at cursor position |
| `press_key` | Press keyboard key or combo |
| `scroll_screen` | Scroll at a screen position |

### Window Management (9 tools)

| Tool | What it does |
|------|-------------|
| `list_windows` | List all visible windows with titles and positions |
| `focus_window` | Bring window to foreground |
| `minimize_window` | Minimize by title or process |
| `pin_window` | Set always-on-top |
| `unpin_window` | Remove always-on-top |
| `move_window` | Move and resize to exact coordinates |
| `set_window_opacity` | Set transparency (0-255) |
| `get_pixel_color` | Get RGB + hex color at coordinates |
| `keep_awake` | Prevent sleep/display-off |

### Desktop Automation (7 tools)

| Tool | What it does |
|------|-------------|
| `hot_corners` | Trigger actions on screen corner hover |
| `text_expander` | Type abbreviation → expanded text |
| `focus_lock` | Prevent focus stealing |
| `distraction_dimmer` | Dim inactive windows |
| `auto_tile` | i3-style window tiling |
| `app_specific_hotkeys` | Per-app key remapping |
| `key_to_mouse` | Map keyboard key to mouse action |

### Macro Recording (6 tools)

| Tool | What it does |
|------|-------------|
| `start_macro_recording` | Begin recording mouse/keyboard actions |
| `stop_macro_recording` | Save recorded macro |
| `replay_macro` | Replay with find-engine or raw coordinates |
| `list_macros` | List saved macros with metadata |
| `add_macro_assertion` | Add visual assertion to macro step |
| `delete_macro` | Delete macro and associated files |

### Event System (4 tools)

| Tool | What it does |
|------|-------------|
| `start_event_listener` | Monitor: focus, clipboard, hotkeys, files, USB, process, display, power, volume |
| `stop_event_listener` | Stop specific or all listeners |
| `get_events` | Poll buffered events with filters |
| `get_event_listeners` | List active listeners |

### System (2 tools)

| Tool | What it does |
|------|-------------|
| `set_volume` | Set system volume (0-100) |
| `speak` | Text-to-speech via Windows SAPI |

### PowerToys Integration (5 tools)

| Tool | What it does |
|------|-------------|
| `powertoys_status` | Check installation, version, available CLIs |
| `powertoys_fancy_zones` | Control FancyZones layouts |
| `powertoys_file_locksmith` | Detect processes locking files |
| `powertoys_image_resize` | Resize images via PowerToys |
| `powertoys_peek` | Quick file preview |

### EventGhost Integration (7 tools)

| Tool | What it does |
|------|-------------|
| `eventghost_status` | Check if EG Webserver is running |
| `eventghost_trigger` | Trigger EventGhost event via HTTP |
| `eventghost_get_value` | Read EG variable |
| `eventghost_set_value` | Write EG variable |
| `eventghost_get_all_values` | Get all EG variables |
| `eventghost_execute_script` | Run Python 2.7 in EG context |
| `eventghost_bridge` | Manage WebSocket event bridge |

---

## Use Cases

**"Debug what I'm looking at"** — Capture your screen + voice-describe the bug. Your agent gets full visual context.

**"Click what you see"** — `find_and_click("Submit button")` — OCR finds it, confidence scores it, clicks it. No coordinates needed.

**"Record my workflow"** — `start_macro_recording("login")` — record clicks and keystrokes, then `replay_macro("login")` re-locates elements visually.

**"Dim distractions"** — `distraction_dimmer(180)` dims everything except your active window. `focus_lock("VSCode")` prevents apps from stealing focus.

**"Auto-tile my windows"** — `auto_tile("master+stack")` for i3-style tiling. Rebalances automatically when windows open/close.

**"React to system events"** — `start_event_listener(["clipboard", "focus", "hotkey:ctrl+shift+c"])` — get notified when clipboard changes, windows gain focus, or custom hotkeys fire.

**"Automate with Node-RED"** — Visual workflow: Timer → capture → OCR → if error keyword → send alert. Drag-and-drop automation.

**"Control PowerToys"** — `powertoys_fancy_zones(command="set-layout", layout="Coding")` — assign windows to zones programmatically.

**"Bridge EventGhost"** — `eventghost_trigger("IR.VolumeUp")` — trigger IR remote commands, control media players, read hardware events.

**"Give remote agents eyes"** — SSE server lets cloud agents see your local screen. Bearer token auth included.

**"Search past observations"** — `search_context_history("CORS error login")` — find when you last saw this bug and what the fix was.

---

## MCP Integration

### Local agents (stdio)

```json
{
  "mcpServers": {
    "eye2byte": {
      "command": "python",
      "args": ["C:/path/to/eye2byte_mcp.py"]
    }
  }
}
```

### Remote agents (SSE)

```bash
python eye2byte_mcp.py --sse                         # No auth (LAN only)
python eye2byte_mcp.py --sse --token mysecret123     # Bearer token auth
```

```json
{
  "mcpServers": {
    "eye2byte": {
      "url": "http://YOUR_LOCAL_IP:8808/sse",
      "headers": {"Authorization": "Bearer mysecret123"}
    }
  }
}
```

### Node-RED

Install the custom node:
```bash
cd ~/.node-red
npm install /path/to/node-red-contrib-eye2byte
```

### PowerToys CmdPal

Build and install the extension:
```bash
cd eye2byte-cmdpal
dotnet build
```

---

## Control Panel

```bash
eye2byte-ui          # or: python eye2byte_ui.py
```

A small always-on-top floating panel with global hotkeys.

### Global Hotkeys (Windows)

| Hotkey | Action |
|--------|--------|
| `Ctrl+Shift+1` | Capture screenshot |
| `Ctrl+Shift+2` | Annotate (freeze screen, draw overlay) |
| `Ctrl+Shift+3` | Toggle voice recording |
| `Ctrl+Shift+5` | Grab clipboard image |

### Annotation Tools

| Key | Tool |
|-----|------|
| `X` | Arrow |
| `C` | Circle |
| `V` | Rectangle |
| `B` | Freehand |
| `T` | Text |

`Enter` saves, `Escape` cancels, right-click to undo.

---

## Architecture

```
eye2byte.py              Core engine — capture, voice, clip, summarize
eye2byte_ui.py           Control panel with hotkeys and annotation overlay
eye2byte_mcp.py          MCP server (FastMCP)
eye2byte_ocr.py          OCR via WinOCR (150ms) with EasyOCR fallback
eye2byte_find.py         Hybrid element finding — OCR fast path + VLM fallback
eye2byte_interact.py     Mouse/keyboard automation via pyautogui
eye2byte_windows.py      Win32 window management + system control (ctypes)
eye2byte_desktop.py      Desktop automation — hot corners, tiling, text expander
eye2byte_macro.py        Macro recording and replay with visual assertions
eye2byte_events.py       Native Win32 event system with webhook delivery
eye2byte_history.py      Searchable context history via SQLite FTS5
eye2byte_powertoys.py    PowerToys CLI wrappers (FancyZones, Locksmith, etc.)
eye2byte_eventghost.py   EventGhost HTTP/WebSocket bridge (GPL-safe)

node-red-contrib-eye2byte/   Node-RED custom node + 7 example flows + dashboard
eye2byte-cmdpal/             PowerToys CmdPal extension (C#/.NET 9)
```

---

## Product Tracks

- **eye2byte-annotator** (`v0.4.0`) — lightweight screenshot annotation tool. Tagged at `v0.4.0-annotator`.
- **eye2byte-platform** (`v0.5.0`) — full desktop automation suite with the complete toolset.

---

## License

MIT


