Metadata-Version: 2.5
Name: ui-locator-cli
Version: 0.1.2
Summary: Locate precise click coordinates of UI elements in screenshots via natural-language queries (UI-TARS / Qwen2.5-VL grounding).
Project-URL: Homepage, https://github.com/applica-software-guru/ui-locator
Project-URL: Repository, https://github.com/applica-software-guru/ui-locator
Author: ui-locator contributors
License: Apache-2.0
License-File: LICENSE
Keywords: computer-use,gui-agent,mlx,qwen-vl,screenshot,ui-grounding,ui-tars,vision-language-model
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: anthropic>=0.111.0
Requires-Dist: openai>=2.43.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: typer>=0.16
Provides-Extra: cloud
Requires-Dist: httpx>=0.27; extra == 'cloud'
Requires-Dist: openai>=1.50; extra == 'cloud'
Provides-Extra: cua
Requires-Dist: anthropic>=0.40; extra == 'cua'
Provides-Extra: demo
Requires-Dist: gradio>=4.0; extra == 'demo'
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: mlx
Requires-Dist: huggingface-hub>=0.25; extra == 'mlx'
Requires-Dist: mlx-vlm>=0.1.20; extra == 'mlx'
Requires-Dist: mlx>=0.22.0; (platform_machine == 'arm64' and sys_platform == 'darwin') and extra == 'mlx'
Provides-Extra: uitars
Requires-Dist: ui-tars>=0.5; extra == 'uitars'
Description-Content-Type: text/markdown

# ui-locator

Locate **precise click coordinates** of UI elements in screenshots via natural-language queries, for use with computer-use agents.

> "Where is the 'Invia' button?" → `(742, 511)` (absolute pixels) + bounding box.

It wraps **GUI grounding models** — primarily ByteDance **UI-TARS-1.5-7B** (Apache-2.0) running locally via **MLX** on Apple Silicon, or any **OpenAI-compatible** cloud endpoint (a local `mlx-vlm` server, `vLLM`, a HuggingFace Inference Endpoint, or the Volcengine/Doubao Ark API for the proprietary **UI-TARS-2**).

## Quick start

The package is published as **`ui-locator-cli`**; the command it installs is
**`ui-locator`**.

```bash
# Run it without installing anything
uvx --from "ui-locator-cli[cloud,uitars]" ui-locator --help

# Or install the command on your PATH
uv tool install "ui-locator-cli[cloud,uitars]"   # cloud-only, any OS
uv tool install "ui-locator-cli[mlx,uitars]"     # local inference, Apple Silicon

# As a library in a project
uv add "ui-locator-cli[cloud,uitars]"
```

From a clone, for development:

```bash
uv pip install -e ".[mlx,uitars]"    # Apple Silicon
uv pip install -e ".[cloud,uitars]"  # cloud-only, any OS
```

For a cloud endpoint (Qwen, UI-TARS, ...), set the project up once:

```bash
ui-locator init                                   # writes .ui-locator/config.toml
echo 'UI_LOCATOR_OPENAI_API_KEY=<key>' >> .ui-locator/.env
ui-locator config show                            # check what resolved
```

```python
from ui_locator import Locator

loc = Locator()  # auto-selects MLX on Apple Silicon
res = loc.locate("screen.png", "where is the 'Invia' button?")

print(res.click_point)   # Coordinates(x=742, y=511)
print(res.bbox)          # BoundingBox(...) or None
print(res.best.confidence)
print(res.best.thought)  # model reasoning (agent mode)
```

Intent-style queries work too — the model resolves *which* element to click:

```python
loc.locate("screen.png", "chiudi il terminale")      # finds the close button
loc.locate("screen.png", "minimizza il browser")     # finds the minimise button
loc.locate("screen.png", "clicca sul tab Google")    # finds the browser tab
```

## Command line

```bash
ui-locator screen.png "where is the 'Invia' button?"
ui-locator screen.png "close the terminal" --think          # ambiguous/intent query
ui-locator screen.png "Submit" --annotate                   # saves screen_test.png
ui-locator screen.png "Submit" out.png --annotate           # custom output path
ui-locator screen.png "Submit" --json                       # machine-readable
```

| Option | Meaning |
| --- | --- |
| `--id <dotted.id>` | Element id — reuse a known position instead of calling the model ([below](#element-map---id)) |
| `--session <id>` | Tag learned entries so one run can be dropped later |
| `--no-cache` | Re-ask the model and overwrite the stored entry |
| `--verify` | Re-check a crop around the stored point before trusting it |
| `--fill id=query ...`, `--prefix` | Populate several ids in one model load |
| `--map-dir <path>` | Element-map directory (default: the project's, see [below](#configuration-ui-locator)) |
| `--use <profile>` | Model profile from `[models.<name>]` in the project config |
| `--provider {auto,mlx,qwen,qwen-local,openai,anthropic}` | Force a backend |
| `--model <hf-id>` | Local MLX model id (default `mlx-community/UI-TARS-1.5-7B-4bit`) |
| `--think` | Enable UI-TARS reasoning (agent mode) |
| `--base-url`, `--api-key`, `--openai-model`, `--openai-prompt {uitars,qwen}` | Cloud endpoint settings |
| `--annotate PATH` | Save an annotated copy of the image |
| `--json` | Emit JSON |

## Configuration (`.ui-locator/`)

A project is any directory containing `.ui-locator/`, found by walking up from
the current directory the way git finds `.git`. That one root holds both the
configuration and the element map, so the coordinates you have already
established travel with the automation that uses them:

```
progetto/
├── .ui-locator/
│   ├── config.toml          # committed: providers, endpoints, models
│   ├── .env                 # gitignored: API keys only
│   └── map/
│       └── sap.screen.json  # committed: known coordinates
└── automazioni/sap/         # any depth: the same root is found
```

```bash
ui-locator init            # create .ui-locator/ here, with a config template
ui-locator config show     # the resolved settings, and where each came from
ui-locator config path     # the project root, if any
```

### Model profiles

Adding a backend is a TOML block, not a code change:

```toml
default_model = "qwen-cloud"      # used when --use is not given

[models.qwen-cloud]
provider = "openai"
openai_base_url = "https://api.fireworks.ai/inference/v1"
openai_model = "accounts/fireworks/models/qwen3p7-plus"
openai_prompt = "qwen"

[models.uitars-local]
provider = "mlx"
model = "mlx-community/UI-TARS-1.5-7B-4bit"
```

```bash
ui-locator screen.png "Invia" --use uitars-local    # or UI_LOCATOR_USE=...
```

Any `Settings` field works as a top-level key (shared baseline) or inside a
profile (specialisation). Unknown keys are reported on stderr — including those
in profiles you have not selected, so a typo surfaces before the day you switch
to it.

### Precedence

Highest first:

1. CLI flags
2. environment variables (`UI_LOCATOR_*`, and the plain `OPENAI_*` the cloud provider also honours)
3. `.env` — `./.env`, then `.ui-locator/.env`
4. the selected `[models.<name>]` block
5. top-level keys of `.ui-locator/config.toml`
6. the same of `$XDG_CONFIG_HOME/ui-locator/config.toml`
7. field defaults

`ui-locator config show` prints the winner of each field **and** the layer it
came from (`model:qwen-cloud`, `env:OPENAI_MODEL`, `config:/path`, `default`),
naming the exact environment variable — which is the only quick answer to "why
is it calling the wrong endpoint". API keys are shown as `***`.

Keep secrets out of `config.toml` so it can be committed: `ui-locator init`
writes a `.gitignore` for `.env` alone — `config.toml` and `map/` are meant to be
committed. Positions the model guessed during a run land in the same files as the
ones you declared, so drop them with `ui-locator map forget --session <id>`
before committing if you only want to keep what you have verified.


## Element map (`--id`)

Grounding a screenshot costs seconds. Most agent loops ask the same question
about the same screen over and over, so `--id` lets **the caller** name an
element once and reuse its position:

```bash
# first time: the model runs, and the position is stored under that id
ui-locator screen.png --id sap.screen.submit "il bottone Invia"    # ~3 s

# afterwards: answered from the map — no model, no image decode
ui-locator screen.png --id sap.screen.submit "il bottone Invia"    # ~0.13 s
```

The id is an **assertion by the caller**, not a guess from the image: passing
`--id sap.screen.submit` states that this element sits at a stable position on
that screen, whatever data the screen happens to be showing. That is exactly the
case a screenshot fingerprint gets wrong — a data-heavy screen looks different on
every visit while its buttons never move — and it is why a hit needs no image
analysis at all: only the PNG header is read, to get `(width, height)`.

On a hit the query is ignored, so rephrasing it (`"Invia"` /
`"clicca su Invia"`) still hits. It is only used on a miss, and can be omitted
entirely once an entry exists.

### What takes part in the lookup

The map lives in the project (`.ui-locator/map/`) when there is one, otherwise
in `$XDG_DATA_HOME/ui-locator/map` — a durable location, not a cache directory.

| | In the key? | Why |
| --- | --- | --- |
| element id | yes | the caller's assertion |
| screenshot size | yes | a resized window genuinely moves elements |
| backend / model | **no** | recorded as provenance only: which model *found* the button does not change where it *is*, so switching provider does not cold-start the map |
| the screenshot itself | **no** | the whole point of `--id` |

### The map is a file you can edit

One small JSON file per id prefix (last segment = element, the rest = group):

```jsonc
// .ui-locator/map/sap.screen.json
{
  "submit": [
    { "click_point": [742, 511], "bbox": [700, 495, 790, 530],
      "size": [2560, 1440], "source": "model", "hits": 12 }
  ]
}
```

So when you already know a coordinate, declare it and the model is never
involved. Declared entries win over learned ones and are never overwritten; one
with `"size": null` applies at any resolution.

```bash
ui-locator map set --id sap.screen.submit --point 742,511
ui-locator map set --id sap.screen.submit --point 742,511 --size 2560x1440
```

### Managing it

```bash
ui-locator map show                          # everything
ui-locator map show --id sap.screen          # one screen (dotted prefix)
ui-locator map show --json                   # machine-readable
ui-locator map miss --id sap.screen.submit   # the click failed: drop what was learned
ui-locator map forget --id sap               # app updated: wipe the whole prefix
ui-locator map forget --session run-42       # drop just what one run learned
ui-locator map path
```

Populate a whole screen in one model load:

```bash
ui-locator screen.png --prefix sap.screen \
  --fill submit=Invia --fill cancel=Annulla --fill search="campo di ricerca"
```

### Using it from an agent

The `--json` output says where the answer came from, so the caller can decide
how much to trust it:

```json
{ "click_point": {"x": 742, "y": 511}, "provider": "map",
  "map": {"element_id": "sap.screen.submit", "hit": true,
          "source": "declared", "hits": 12} }
```

Two rules make the loop reliable:

1. **When a click does not have the expected effect, say so** —
   `ui-locator map miss --id <id>` — or re-run with `--no-cache`, which re-asks
   the model and overwrites the entry. A stored position is only ever corrected
   by feedback: nothing else can tell that it went stale.
2. `--session <run-id>` tags what a run learns, so a whole run's guesses can be
   dropped without touching hand-declared entries.

The residual risk of `--id` is not that the button moved — you asserted it does
not — but that the caller **misidentified which screen it is on** (a modal is
up, or it is on a different screen than it thinks). `--verify` covers that: it
compares a small stored crop around the point against the screenshot and falls
back to the model on mismatch. It costs one image decode and is insensitive to
data changing elsewhere on the screen.

### Python

```python
from ui_locator import Locator

loc = Locator()
res = loc.locate("screen.png", "il bottone Invia",
                 element_id="sap.screen.submit", session="run-42")
res.map.hit        # False on the first call, True afterwards
```

Settings also come from the environment: `UI_LOCATOR_MAP_DIR`,
`UI_LOCATOR_SESSION`, `UI_LOCATOR_USE_MAP=0`, `UI_LOCATOR_VERIFY=1`.


## Use it from an agent

A ready-made skill for a coding / computer-use agent ships with the package:

```bash
ui-locator skill install                    # ~/.claude/skills/ui-locator
ui-locator skill install --scope project    # the layout the project already uses
ui-locator skill install --scope agents     # ./.agents/skills/ui-locator
ui-locator skill install --scope claude     # ./.claude/skills/ui-locator
ui-locator skill install --dir path/skills  # anywhere
ui-locator skill install --link             # symlink, so edits apply immediately
ui-locator skill status                     # where it is, and whether it is current
ui-locator skill update                     # refresh after upgrading ui-locator
ui-locator skill remove
```

`SKILL.md` is the same file for any agent — only the directory is
vendor-specific, so it is not hardcoded. `--scope project` installs into
whichever of `.agents/skills` or `.claude/skills` the project already uses, and
into the vendor-neutral `.agents/` when neither exists yet.

The agent picks it up in a new session. The skill covers the `--id` workflow,
**when a stable id is a valid assertion and when it is not**, and how to report
a bad position back — which is what keeps the map honest.

`install` refuses to overwrite an existing install (use `update`, or `--force`),
and neither `install` nor `remove` will touch a directory that is not this
skill.

## Providers

| Provider | Engine | When to use |
| --- | --- | --- |
| **`mlx`** | UI-TARS-1.5-7B via `mlx-vlm` | **Default on Apple Silicon.** On-device, private, free. Returns a click point. |
| **`qwen`** | Qwen2.5-VL, cloud or local | **Bounding-box** grounding (native `<box>` tokens). Uses the configured endpoint when there is one, the local MLX checkpoint otherwise, and always the Qwen prompt style. |
| **`qwen-local`** | Qwen2.5-VL via `mlx-vlm` | Force the on-device checkpoint even when an endpoint is configured. |
| **`openai`** | Any OpenAI-compatible endpoint | Cloud; the only way to use **UI-TARS-2** (Doubao/Ark). |
| `omniparser` | Microsoft OmniParser v2 | *Planned* — note its detector is **AGPL-3.0**. |
| `cua` | Anthropic Claude Computer Use | *Planned fallback* — easy but least precise. |

### Provider selection

`Locator()` auto-selects: MLX (UI-TARS) on Apple Silicon, otherwise an OpenAI-compatible endpoint if configured. Override with `UI_LOCATOR_PROVIDER` (or `--provider`), or name a profile with `--use`.

`qwen` means *Qwen grounding wherever it is available*: the configured endpoint if there is one, the local MLX checkpoint otherwise. Either way it forces `openai_prompt = "qwen"`, because the default `uitars` style parses `<point>` and finds nothing in a `<box>` answer — a mismatch that produces no error, just no coordinates. When the two disagree anyway (say `--provider openai` with a Qwen model), ui-locator warns on stderr rather than letting it look like a grounding failure. Use `qwen-local` to pin the on-device checkpoint (`--qwen-model` to change it).

### Cloud endpoints

OpenRouter UI-TARS-1.5:

```bash
export UI_LOCATOR_PROVIDER=openai
export UI_LOCATOR_OPENAI_BASE_URL=https://openrouter.ai/api/v1
export UI_LOCATOR_OPENAI_API_KEY=<your-openrouter-key>
export UI_LOCATOR_OPENAI_MODEL=bytedance/ui-tars-1.5-7b
export UI_LOCATOR_OPENAI_PROMPT=uitars
```

Fireworks Qwen bbox grounding:

```bash
export UI_LOCATOR_PROVIDER=openai
export UI_LOCATOR_OPENAI_BASE_URL=https://api.fireworks.ai/inference/v1
export UI_LOCATOR_OPENAI_API_KEY=<your-fireworks-key>
export UI_LOCATOR_OPENAI_MODEL=accounts/fireworks/models/qwen3p7-plus
export UI_LOCATOR_OPENAI_PROMPT=qwen
```

UI-TARS-2 via Volcengine/Doubao Ark:

```bash
export UI_LOCATOR_PROVIDER=openai
export UI_LOCATOR_OPENAI_BASE_URL=https://ark.cn-beijing.volces.com/api/v3
export UI_LOCATOR_OPENAI_API_KEY=<your-ark-key>
export UI_LOCATOR_OPENAI_MODEL=<doubao-ui-tars-endpoint-id>
export UI_LOCATOR_OPENAI_PROMPT=uitars
```

You can equally point `OPENAI_BASE_URL` at a local server (`mlx_vlm.server --model mlx-community/UI-TARS-1.5-7B-4bit`) or a `vLLM` instance.

## Coordinates

UI-TARS-1.5/2 emit click points as **absolute pixels in the "smart-resized" image space** (Qwen2.5-VL preprocessing). `ui-locator` maps these back to the **original screenshot resolution** and always returns absolute pixels. The optional `ui-tars` package is used for parsing when available; a self-contained regex + `smart_resize` fallback is used otherwise.

- `Element.click_point` — where to click, absolute pixels (always present).
- `Element.bbox` — bounding box, absolute pixels (when available).
- `Element.confidence` — a heuristic `[0, 1]` signal (UI-TARS does not emit a score).

## Important notes

- Recommended order: **local UI-TARS-1.5 via MLX** when available; **OpenRouter UI-TARS-1.5** for hosted use; **Qwen bbox models** when you want region grounding; **UI-TARS-2 via Volcengine Ark** if you can access it.
- **UI-TARS-2 is not open-weight.** It is accessible **only** via ByteDance's Volcengine/Doubao Ark API. The local option is **UI-TARS-1.5-7B** (Apache-2.0), which shares UI-TARS-2's architecture and grounding format.
- **MLX is Apple-Silicon only.** On Intel Macs / Linux use the cloud provider.
- **Memory:** UI-TARS-1.5-7B at **4-bit** uses ~4–5 GB (default; good for 8–16 GB Macs). Use `--model mlx-community/UI-TARS-1.5-7B-6bit` (~6–8 GB) or `-8bit` (~9–11 GB) for higher quality on machines with more memory.
- The `ui-tars` PyPI package is ByteDance's action **parser** (Apache-2.0), not the model itself.

## Examples & benchmark

The [`examples/`](examples/) folder contains runnable scripts:

```bash
uv run python examples/locate_button.py screen.png "where is the 'Invia' button?"
uv run python examples/locate_button.py screen.png "the Submit button" --provider qwen
uv run python examples/annotate_demo.py screen.png "the 'Invia' button" out.png  # draw bbox + click
uv run python examples/serve_mlx.py     screen.png "the 'Invia' button"          # local mlx-vlm server + query
```

The [`bench/`](bench/) folder is a **precision benchmark**. It runs a provider over cases
(screenshot + query + ground-truth bounding box) and reports hit-rate, click-to-centre
distance and IoU — useful to validate accuracy on your own UIs.

```bash
uv run python bench/make_fixtures.py          # generate synthetic screenshots + GT
uv run python bench/benchmark.py              # benchmark the default provider
UI_LOCATOR_PROVIDER=qwen uv run python bench/benchmark.py   # compare with Qwen2.5-VL
uv run python bench/benchmark.py --models bench/fixtures/model_matrix.json   # multi-model benchmark
uv run python bench/compare_models.py ~/Desktop/screen.png "locate refresh button in active window"  # annotated per model
```

For multi-model runs, edit `bench/fixtures/model_matrix.json`.
`bench/benchmark.py` reports scores per model; `bench/compare_models.py` saves one annotated image per model.

Add your own benchmark cases by editing `bench/fixtures/benchmark.json`
(`{"image": "...", "query": "...", "bbox": [x1,y1,x2,y2]}`).

## Development

```bash
uv sync --extra dev --extra mlx --extra uitars
uv run ruff check
uv run mypy
uv run pytest

# Live model test (Apple Silicon, downloads checkpoint on first run):
UI_LOCATOR_RUN_MODEL=1 uv run pytest -m integration
```

`uv.lock` is not committed, so CI resolves dependencies the way a user
installing from PyPI does. The supported `typer` range is tested at both ends
(0.16 and latest) because 0.27 stopped being click-based.

### Releasing

The distribution is named `ui-locator-cli` on PyPI while `[project.scripts]`
keeps the command itself `ui-locator`, so extras are addressed by the
distribution name (`ui-locator-cli[mlx]`) and `uvx` needs `--from`.

`.github/workflows/publish.yml` uploads to PyPI when a GitHub release is
published, gated on lint, types, the test matrix, and two release-only checks:
the tag must match the version in `pyproject.toml` (PyPI never lets a version
be reused), and `SKILL.md` must be present in the wheel.

```bash
# 1. bump version = "x.y.z" in pyproject.toml, commit
# 2. rehearse: builds and round-trips the artifacts, uploads nothing
gh workflow run publish.yml -f target=testpypi
# 3. release (the tag must be vx.y.z)
gh release create v0.1.0 --generate-notes
```

Authentication uses the `PYPI_API_TOKEN` secret. [Trusted
Publishing](https://docs.pypi.org/trusted-publishers/) is preferable — GitHub
mints short-lived OIDC credentials per run, so there is no long-lived secret to
leak or rotate. Configure a publisher for this repo and `publish.yml`, then
delete the `password:` line from the PyPI step.

## License

Apache-2.0. The optional OmniParser detector model is AGPL-3.0 (evaluate separately if used).
