Local development preview Structured v1 API

Find the pixels.
Keep the evidence.

Locate, click, and fill interface elements from screenshots across Selenium, Playwright, and Appium. The structured API explains every match—and refuses uncertain actions before they touch the page.

Get started Read the full contract
  • Python 3.10+
  • No selector required
  • MIT licensed
safe_click.py Python 
from pyxelator import click_result

result = click_result(
    driver,
    "templates/save.png",
    confidence=0.80,
    verify_image="templates/saved.png",
)

if result.ok:
    print("Saved safely")
else:
    print(result.reason)
MatchResult(ok=True, score=0.94)
One visual API for Selenium Playwright Appium beta

Quick start

From install to evidence in minutes.

This is an unreleased local development preview. The structured APIs shown here are not yet available from the published PyPI package. For this preview, install the development checkout with pip install -e '.[selenium]'. Capture a tightly cropped element image, then choose the simple or structured API.

01

Install

Add the adapter extra for the automation framework you use.

pip install pyxelator[selenium]

Also available: playwright, appium, or dev.

02

Capture

Crop a distinctive element from a screenshot. Keep the crop tight and avoid surrounding layout.

Save
03

Act

Pass your driver, template, and confidence. Check result.ok before continuing.

from pyxelator import click_result

result = click_result(
    driver, "save.png",
    confidence=0.8
)
assert result.ok, result.reason

Structured v1 API

Three calls, one explicit contract.

Match results preserve confidence, bounds, scale, ranked candidates, and refusal reasons. Action results add what happened after the match.

match_result()

Inspect screenshot bytes without acting.

match_result(
  screenshot_bytes,
  template_path,
  confidence=0.7,
  grayscale=True,
  scales=None,
  ambiguity_margin=0.02,
) → MatchResult
  • Ranks spatially distinct candidates
  • Reports score, rectangle, scale, and margin
  • Truthful only when ok is true

click_result()

Validate once, then click safely.

click_result(
  driver, image,
  confidence=0.7, *,
  ambiguity_margin=0.02,
  verify_image=None,
  verification_confidence=None,
  verification_expected_visible=True,
  debug=False,
) → ActionResult

One pre-action screenshot. No click occurs unless the match is both confident and unambiguous.

fill_result()

Validate once, then write text.

fill_result(
  driver, image, text,
  confidence=0.7, *,
  ambiguity_margin=0.02,
  verify_image=None,
  verification_confidence=None,
  verification_expected_visible=True,
  debug=False,
) → ActionResult

The accepted coordinates go directly to the framework adapter, avoiding a second match against a changed screen.

Result objects

Evidence you can log, test, and inspect.

Complete field-by-field reference
MatchResult
ok, found, score, threshold, location, scale, ambiguous, reason, candidates
ActionResult
ok, action, match, optional verification, reason, artifacts
VerificationResult
ok, expected_visible, observed_visible, match, reason

Safety behavior

Uncertain pixels produce zero actions.

Structured actions fail closed. A refusal still returns useful evidence so you can improve the template or choose a safer fallback.

1

Capture

Take one fresh screenshot from the driver.

2

Validate

Require confidence and a clear winner.

3A

Act

Click or fill only when match.ok is true.

3B

Refuse

Return a reason. Do not touch the UI.

Ambiguity rejection

If another passing candidate falls within the default 0.02 score margin, the target is marked ambiguous_match. No action occurs.

Threshold gate

A best candidate below confidence returns below_threshold with its score and bounds for diagnostics.

Invalid input handling

Unreadable screenshots, invalid or flat templates, unusable scales, and matcher errors become explicit results—not accidental actions.

Optional verification

Match a fresh screenshot after the action. You can require an image to appear or prove disappearance with a valid below-threshold result.

Check ok, not only found.

found means the winner passed the score threshold. ok also means the winner was not made unsafe by a similarly scored, spatially distinct candidate.

Keep exploring

Examples, details, and measurements.

Everything here works from the repository. The example and benchmark generate their own deterministic images and require no browser or network.

Reference

Structured results

Full signatures, fields, reasons, refusal behavior, and verification flows.

Open reference Runnable example

Offline structured flow

Generate fixtures and exercise match, click, fill, and ambiguity refusal.

View example Benchmark

Compare match paths

Measure legacy and structured matching with reproducible local inputs.

View benchmark
Run locally python examples/structured_results.py python benchmarks/benchmark_matching.py --iterations 20