Metadata-Version: 2.4
Name: robotframework-desktop
Version: 0.1.3
Summary: Robot Framework library for Windows desktop automation with hybrid image (pyautogui) and UI Automation (pywinauto) engines.
License: Apache-2.0
Keywords: robotframework,desktop,automation,pyautogui,pywinauto,uia,wpf,winforms
Classifier: Framework :: Robot Framework :: Library
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: robotframework>=6.1
Requires-Dist: robotframework-pythonlibcore>=4.2
Requires-Dist: pyautogui>=0.9.54
Requires-Dist: opencv-python>=4.8
Requires-Dist: pillow>=10.0
Requires-Dist: mss>=9.0
Requires-Dist: pywinauto>=0.6.8
Requires-Dist: pyperclip>=1.8
Requires-Dist: rapidocr>=3.0
Requires-Dist: onnxruntime>=1.17
Provides-Extra: ocr
Requires-Dist: rapidocr>=3.0; extra == "ocr"
Requires-Dist: onnxruntime>=1.17; extra == "ocr"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: rapidocr>=3.0; extra == "dev"
Requires-Dist: onnxruntime>=1.17; extra == "dev"
Requires-Dist: PyYAML>=6.0; extra == "dev"
Requires-Dist: blue>=0.9; extra == "dev"

<div align="center">

# 🖥️ robotframework-desktop

**Hybrid Windows desktop automation for [Robot Framework](https://robotframework.org)**
Image recognition · UI Automation · OCR — one keyword set, chosen by the locator.

![Python](https://img.shields.io/badge/python-3.10%2B-3776AB?logo=python&logoColor=white)
![Robot Framework](https://img.shields.io/badge/Robot%20Framework-6.1%2B-000000?logo=robotframework&logoColor=white)
![Platform](https://img.shields.io/badge/platform-Windows-0078D6?logo=windows&logoColor=white)
![License](https://img.shields.io/badge/license-Apache--2.0-green)
![Code style: blue](https://img.shields.io/badge/code%20style-blue-1f6feb)

[Install](#-install) · [Quick start](#-quick-start) · [Locators](#-locators) · [YAML locators](#-locators-in-a-yaml-file) · [OCR](#-ocr) · [Keywords](docs/DesktopLibrary.html)

</div>

---

## ✨ Features

- 🔀 **Hybrid keywords** – `Click`, `Input Text`, `Drag And Drop`, ... work with images, UI elements or regions; the engine is picked from the `element` argument.
- 🖼️ **Image recognition** – pyautogui + OpenCV, per-call similarity, multiple candidate images, offsets and search regions.
- 🌳 **UI Automation** – pywinauto (`uia` and `win32` backends) for WPF, WinForms, UWP and Win32 apps.
- 🔤 **OCR** – [RapidOCR](https://github.com/RapidAI/RapidOCR) on onnxruntime, models bundled, **no external binary**; accent and case insensitive text search.
- ⏳ **Busy cursor aware** – actions wait automatically while the app shows the busy cursor.
- 🧩 **SikuliLibrary compatible** – implements all its actions, plus UIA keywords (`Attach To Window`, `Get Element Text`, `Set Element Value`, `Select Combobox Item`, ...).

## 📦 Install

```bash
pip install robotframework-desktop          # core
pip install robotframework-desktop[ocr]     # + OCR (rapidocr + onnxruntime)
```

## 🚀 Quick start

```robot
*** Settings ***
Library    DesktopLibrary    image_paths=${CURDIR}/images;${CURDIR}/shared    similarity=0.6

*** Test Cases ***
Write In Notepad
    Open Application     notepad.exe
    Attach To Window     name_re:.*Notepad
    Input Text           class:Edit    Hello
    Click                ${{ {"path": ["save.png", "save_dark.png"], "similarity": 0.85} }}
    Wait Until Screen Contain    done.png    10s
    [Teardown]    Close Application    notepad.exe
```

> [!TIP]
> Relative image paths are searched in the `image_paths` folders (in order, extendable with `Add Image Path`) and then in the current working directory. The extension can be omitted.

## 🎯 Locators

| Locator | Engine |
|---|---|
| `save.png` · `save.png=0.85` · `image:C:/refs/save.png` | 🖼️ Image |
| `{"path": [...], "similarity": 0.8, "scales": "1, 1.1-1.6", "offset": {x, y}, "region": {x, y, width, height}}` | 🖼️ Image |
| `automation_id:btnSave` · `name:OK AND control_type:Button` | 🌳 UIA |
| `win32:class:Button` · `{"name": "Save", "backend": "win32"}` | 🌳 Win32 |
| `{"automation_id": "btnSave", "offset": {"x": 5, "y": 0}}` | 🌳 UIA |
| `[x, y, w, h]` · `{"region": {...}}` · region/match returned by keywords | 📐 Coordinates |

## 🗂️ Locators in a YAML file

Keep all element locators in one variable file ([full example](tests/atest/elements.yml)):

```yaml
APP:
  BTN_SAVE: "automation_id:btnSave"
  TXT_NAME:
    automation_id: txtName
    control_type: Edit
  BTN_SAVE_WIN32:
    name: Save
    class_name_re: ".*BUTTON.*"
    backend: win32
  IMG_TARGET_OFFSET:
    path: target.png
    offset:
      x: 30
      y: 20
  IMG_IN_REGION:
    path: save.png
    region:
      x: 0
      y: 0
      width: 800
      height: 600
```

```robot
*** Settings ***
Library      DesktopLibrary
Variables    elements.yml          # requires PyYAML

*** Test Cases ***
Save
    Input Text    ${APP.TXT_NAME}    Robot
    Click         ${APP.BTN_SAVE}
    Click         ${APP.IMG_TARGET_OFFSET}
```

## 🔤 OCR

No language setup needed — the default RapidOCR models read Latin text (accented letters may come out unaccented, so matching ignores accents and case).

```robot
${text}=    Get Text    automation_id:lblStatus
${text}=    Read Text From Region    [10, 400, 300, 30]
Click Text    Clear
Set ROI    [0, 0, 600, 400]
Region Click Text    Clear
```

## 🔍 Scaled images (DPI / font scaling)

Remote sessions (Citrix, RDP) may render menus, window titles and message boxes with a different font scaling (e.g. 125%) than when the reference images were captured. Enable multi-scale search globally or per locator:

```robot
Library    DesktopLibrary    image_scales=1, 1.1-1.6

Set Image Scales    1.0, 0.8-0.9, 1.1-1.6
Click    ${{ {"path": "menu_file.png", "scales": "1, 125%"} }}
```

Scale `1` is tried first; when it finds nothing, the other scales are tried and the best score wins. Ranges (`1.1-1.6`, step `0.02`, or `1.1-1.6:0.05`) are recommended because fonts are rounded to whole pixels, so the real factor varies per text. More scales mean slower searches when the image is not on screen, so combine it with `Set ROI`.

> [!NOTE]
> Matches at scales other than `1` must reach at least `0.8` similarity (`image_scales_similarity=` on import or `Set Image Scales    ...    min_similarity=0.85`), even if the locator uses a lower one. Resized images are approximations, and a low similarity across many scales easily matches a similar element, such as another window title with the same icon.

## 📋 Clipboard in remote sessions

`Paste Text` (and `Input Text` with non-ASCII text) waits `clipboard_delay` (default `0.5s`) before and after `Ctrl+V`, so remote sessions receive the new clipboard content before the previous one is restored. Tune it with `clipboard_delay=` on import or `Set Clipboard Delay`.

## 🔁 Looping over elements

```robot
@{items}=    Find Elements    control_type:ListItem          # [] when nothing is found
FOR    ${item}    IN    @{items}
    ${text}=    Get Element Text    ${item}
    IF    $text == "Item 3"    Click    ${item}
END
@{icons}=    Find Elements    icon.png    sort_by=column    min_count=3
@{rows}=     Find Child Elements    automation_id:grid    control_type:DataItem    depth=1
```

## ⏳ Busy cursor handling

Action keywords wait while the cursor is busy (`wait` / `appstarting`) until it stays normal for 0.5 s — **before** searching, before acting and **after** the action (watched for 0.2 s). On timeout (default 30 s) the keyword fails, or only warns with `busy_cursor_fail=False` / `ignore_busy_error=True`. Use `show_warnings=False` to log warnings as INFO.

```robot
Library    DesktopLibrary    busy_cursor_timeout=1 min    busy_cursor_after_delay=0.3s

Click    automation_id:btnLoad    ignore_busy_error=True
Set Busy Cursor Wait     False
Set Busy Cursor Types    wait,appstarting,custom
Wait Until Cursor Is Not Busy    2 min
${cursor}=    Get Current Cursor Type
```

## 📚 Documentation

Full keyword reference: **[docs/DesktopLibrary.html](docs/DesktopLibrary.html)** (`just docs` to regenerate).

## 🛠️ Development

Requires [uv](https://docs.astral.sh/uv/) and [just](https://github.com/casey/just) (PowerShell recipes).

| Command | Description |
|---|---|
| `just setup` | Create `.venv` and install the library (editable + dev deps) |
| `just fmt` | Format code with [blue](https://github.com/grantjenks/blue) |
| `just test` | Unit tests (pytest) |
| `just atest` | Acceptance tests (Robot Framework) |
| `just docs` | Regenerate libdoc |
| `just build` | Build both wheels |
| `just build-full` | `dist/full` – OCR deps required |
| `just build-no-ocr` | `dist/no-ocr` – OCR optional (`[ocr]` extra) |
| `just check` | Show dependencies of the built wheels |

### Release

CI ([.github/workflows/ci.yml](.github/workflows/ci.yml)) runs lint, unit and acceptance tests on every push and pull request.
To publish a release, bump `version` in `pyproject.toml`, then:

```powershell
git tag v0.2.0
git push --tags
```

[.github/workflows/release.yml](.github/workflows/release.yml) runs the tests, builds both wheels and creates a GitHub Release with them attached:

- `robotframework-desktop` - OCR optional (`[ocr]` extra)
- `robotframework-desktop-full` - OCR deps required

> [!WARNING]
> The acceptance suite launches a small WinForms app ([tests/app/SampleApp.ps1](tests/app/SampleApp.ps1)) and generates its reference images at runtime. **Don't use the mouse or keyboard while it runs.**

## 📄 License

[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
