Metadata-Version: 2.4
Name: adscrawl
Version: 0.1.0
Summary: Python SDK for AdsCrawl: rendered content, screenshots, structured extraction, remote CDP sessions, and cloud browsers.
Author-email: AdsCrawl <support@adscrawl.net>
License-Expression: MIT
Project-URL: Homepage, https://www.adscrawl.net/?utm_source=pypi&utm_medium=sdk&utm_campaign=adscrawl-python
Project-URL: Documentation, https://www.adscrawl.net/docs/
Project-URL: Repository, https://github.com/AdsCrawl/adscrawl-python
Project-URL: Issues, https://github.com/AdsCrawl/adscrawl-python/issues
Project-URL: Changelog, https://github.com/AdsCrawl/adscrawl-python/blob/main/CHANGELOG.md
Keywords: adscrawl,web-scraping,browser,markdown,screenshot,playwright,cdp,python,ai,extraction
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6; extra == "dev"
Provides-Extra: playwright
Requires-Dist: playwright>=1.49; extra == "playwright"
Dynamic: license-file

<p align="center">
  <a href="https://www.adscrawl.net/?utm_source=github&utm_medium=sdk&utm_campaign=adscrawl-python">
    <img src="https://raw.githubusercontent.com/AdsCrawl/adscrawl-python/main/assets/adscrawl-logo.svg" alt="AdsCrawl" width="360" />
  </a>
</p>

<p align="center"><strong>Real browsers. Structured extraction. Screenshots and automation.</strong></p>

<h1 align="center">Python SDK</h1>

Turn a URL into rendered HTML, readable Markdown, structured data, or a PNG screenshot. Connect a remote browser with Playwright when your workflow needs interaction.

[Website](https://www.adscrawl.net/?utm_source=github&utm_medium=sdk&utm_campaign=adscrawl-python) · [API documentation](https://www.adscrawl.net/docs/) · [Get an API key](https://app.adscrawl.net/register/?utm_source=pypi&utm_medium=sdk&utm_campaign=adscrawl-python) · [中文](./README.zh-CN.md)

- Python 3.9+ with synchronous and asynchronous clients.
- Zero runtime dependencies; uses the Python standard library.
- Explicit request timeouts, response validation, and typed errors.
- No automatic retries of metered requests or browser creation.

## Install

```bash
pip install adscrawl
```

## Your first request

[Create an account](https://app.adscrawl.net/register/?utm_source=pypi&utm_medium=sdk&utm_campaign=adscrawl-python), create an API key, and set `ADSCRAWL_API_KEY` in the server environment.

```python
from adscrawl import AdsCrawl

client = AdsCrawl()
markdown = client.markdown({
    "url": "https://www.adscrawl.net",
    "waitUntil": "domcontentloaded",
})
print(markdown)
```

You can also use `AdsCrawl(api_key="...")`. Keep API keys in server-side code. Request dictionaries use the same camelCase fields as the HTTP API and TypeScript SDK.

## Rendered content and screenshots

```python
from pathlib import Path
from adscrawl import AdsCrawl

client = AdsCrawl()
html = client.html({"url": "https://www.adscrawl.net"})
article = client.article({"url": "https://www.adscrawl.net"})
print(article["title"], article["textContent"])

png = client.screenshot({
    "url": "https://www.adscrawl.net",
    "viewport": {"width": 1440, "height": 900},
    "fullPage": True,
    "waitUntil": "load",
})
Path("page.png").write_bytes(png)
```

`html()` returns HTML by default. `markdown()` returns Markdown, `article()` returns a dictionary, and `screenshot()` returns PNG bytes. Use a custom `proxy` or managed `countryCode`; they cannot be combined.

## Proxy and fingerprint: BrowserScan screenshot

AdsCrawl uses real browsers with configurable routing and browser fingerprints. This example uses managed `GLOBAL` routing unless proxy environment variables are present, opens [BrowserScan](https://www.browserscan.net/), and saves the returned screenshot. The same browser workflow can access and render [Pixelscan](https://pixelscan.net/) and [IPhey](https://iphey.com/); inspect the returned page for its current score.

```python
import os
from pathlib import Path
from adscrawl import AdsCrawl

client = AdsCrawl()
server = os.getenv("ADSCRAWL_PROXY_SERVER")
username = os.getenv("ADSCRAWL_PROXY_USERNAME")
password = os.getenv("ADSCRAWL_PROXY_PASSWORD")
if bool(username) != bool(password):
    raise RuntimeError("Set both proxy username and password, or neither.")
if not server and (username or password):
    raise RuntimeError("Set ADSCRAWL_PROXY_SERVER with proxy credentials.")

routing = {"countryCode": "GLOBAL"}
if server:
    proxy = {"server": server}
    if username and password:
        proxy.update({"username": username, "password": password})
    routing = {"proxy": proxy}

png = client.screenshot({
    "url": "https://www.browserscan.net/",
    **routing,
    "viewport": {"width": 1440, "height": 900},
    "fullPage": True,
    "waitUntil": "networkidle",
    "timeoutMs": 60_000,
    "userAgentMode": "random",
    "userAgentOs": "windows",
    "fingerprint": {
        "webRtc": "forward", "webGl": "random", "webGpu": "random",
        "webGlImage": "random", "canvas": "random",
        "audioContext": "random", "clientRects": "random",
        "speechVoices": "random", "fonts": "random",
        "hardware": "random", "doNotTrack": "random",
    },
}, timeout_ms=75_000)
Path("browserscan.png").write_bytes(png)
```

## Structured extraction

```python
from adscrawl import AdsCrawl

client = AdsCrawl()
print(client.spa.templates()["templates"])

result = client.spa.extract({
    "url": "https://www.adscrawl.net",
    "fields": {
        "title": {"source": "dom", "selector": "h1", "value": "text", "required": True},
    },
})
print(result["data"]["title"], result["missingFields"])

inspection = client.spa.inspect({"url": "https://www.adscrawl.net"})
print(inspection["candidates"])
```

## Remote browsers with Playwright

```bash
pip install "adscrawl[playwright]"
playwright install chromium
```

```python
from adscrawl import AdsCrawl
from playwright.sync_api import sync_playwright

client = AdsCrawl()
session = client.cdp.create({
    "idleTimeoutMs": 600_000,
    "maxSessionMs": 3_600_000,
    "browserSettings": {"viewport": {"width": 1440, "height": 900}},
})
try:
    with sync_playwright() as playwright:
        browser = playwright.chromium.connect_over_cdp(session["cdpBaseUrl"])
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()
        page.goto("https://www.adscrawl.net")
        print(page.title())
        browser.close()
finally:
    client.cdp.close(session["sessionId"])
```

`cdp.list()` returns `{"ok": true, "data": [...]}`. `cdp.get_version(session)` discovers the WebSocket endpoint without forwarding the API key. Connection URLs contain secrets and should not be logged.

## Persistent cloud browsers

```python
import os
from adscrawl import AdsCrawl

client = AdsCrawl()
server = os.environ["ADSCRAWL_PROXY_SERVER"]
proxy = {"server": server}
launched = client.cloud_browsers.launch({
    "proxy": proxy,
    "tabs": ["https://www.adscrawl.net"],
})
browser_id = launched["id"]
try:
    profile = client.cloud_browsers.get(browser_id)
    print(profile["id"], profile["runtime"]["status"])
finally:
    client.cloud_browsers.stop(browser_id)
```

API-key starts and launches require a top-level custom `proxy` every time. A `stopping` response does not confirm shutdown; poll `get()` until `stopped`. Failed launches can expose a cleanup id as `AdsCrawlAPIError.id`.

## Async client

```python
import asyncio
from adscrawl import AsyncAdsCrawl

async def main():
    async with AsyncAdsCrawl() as client:
        markdown = await client.markdown({"url": "https://www.adscrawl.net"})
        print(markdown)

asyncio.run(main())
```

The async facade runs the dependency-free standard-library HTTP transport in worker threads. Cancelling a coroutine or reaching a timeout does not prove remote browser work stopped.

## Configuration and errors

```python
from adscrawl import AdsCrawl, AdsCrawlAPIError, AdsCrawlTimeoutError

client = AdsCrawl(base_url="https://api.adscrawl.net", timeout_ms=90_000)
try:
    text = client.markdown(
        {"url": "https://www.adscrawl.net", "timeoutMs": 60_000},
        timeout_ms=75_000,
    )
    print(text)
except AdsCrawlAPIError as error:
    print(error.status, error.code, error.trace_id)
except AdsCrawlTimeoutError:
    print("Inspect remote sessions before retrying.")
```

The API key defaults to `ADSCRAWL_API_KEY`; the base URL defaults to `ADSCRAWL_BASE_URL`, `ADSCRAWL_API_URL`, then `https://api.adscrawl.net`. Errors include `AdsCrawlAPIError`, `AdsCrawlTimeoutError`, `AdsCrawlConnectionError`, and `AdsCrawlResponseError`. API response bodies are credential-redacted. Requests are not automatically retried.

## Develop and release

```bash
python -m unittest discover -s tests -v
python -m build
python -m twine check dist/*
```

Tests use fake responses and never consume AdsCrawl credits. See [`RELEASING.md`](./RELEASING.md) for PyPI trusted publishing.

## License

MIT
