Metadata-Version: 2.4
Name: hamro-scrapy-playwright
Version: 0.2.1
Summary: Monkeypatch scrapy-playwright to use a custom stealth Chromium build
License: MIT
Project-URL: Homepage, https://github.com/jyabadesk/hamro-scrapy-playwright
Project-URL: Repository, https://github.com/jyabadesk/hamro-scrapy-playwright
Keywords: scrapy,playwright,chromium,anti-bot,stealth
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.25
Requires-Dist: scrapy>=2.10
Requires-Dist: scrapy-playwright>=0.0.40
Requires-Dist: playwright>=1.35
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"
Dynamic: license-file

# hamro-scrapy-playwright

Use `scrapy-playwright` with a custom stealth Chromium build.

## Features

* Automatic `scrapy-playwright` patching
* Automatic browser download and management
* Drop-in replacement for existing `scrapy-playwright` projects
* Optional Cloudflare Turnstile auto-click integration

## Install

```bash
pip install hamro-scrapy-playwright
```

## Example

```python
import scrapy
from scrapy import Request


class ExampleSpider(scrapy.Spider):
    name = "example"

    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
            "https": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "PLAYWRIGHT_BROWSER_TYPE": "chromium",
        "PLAYWRIGHT_LAUNCH_OPTIONS": {
            "headless": False,
        },
        "PLAYWRIGHT_PROCESS_REQUEST_HEADERS": None,
    }

    async def start(self):
        yield Request(
            "https://www.example.com/",
            meta={
                "playwright": True,
                "playwright_include_page": True,
            },
            callback=self.parse,
        )

    async def parse(self, response):
        page = response.meta["playwright_page"]

        try:
            await page.wait_for_load_state("networkidle")

            await page.evaluate(
                """
                () => {
                    document.body.style.backgroundColor = "#225577";
                }
                """
            )

            await page.wait_for_timeout(4000)

            self.logger.info("Title: %s", await page.title())

        finally:
            await page.close()
```

## Cloudflare Turnstile Auto-Click

To enable automatic Turnstile clicking, enable `HAMRO_CF_AUTOCLICK` and configure a persistent Playwright context.

### Headful mode (recommended for development)

```python
import tempfile

custom_settings = {
    "DOWNLOAD_HANDLERS": {
        "http": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
        "https": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
    },
    "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    "PLAYWRIGHT_BROWSER_TYPE": "chromium",
    "PLAYWRIGHT_LAUNCH_OPTIONS": {
        "headless": False,
    },
    "PLAYWRIGHT_CONTEXTS": {
        "persistent": {
            "user_data_dir": tempfile.mkdtemp(),
        }
    },
    "HAMRO_CF_AUTOCLICK": True,
    "HTTPERROR_ALLOWED_CODES": [403, 429],
}
```

A visible browser window opens and the bundled `cf-autoclick` extension runs, so you can watch the Turnstile widget get clicked in real time.

### Headless mode

If running headless, use Chromium's new headless mode (extensions do not load in the old headless mode):

```python
"PLAYWRIGHT_LAUNCH_OPTIONS": {
    "headless": True,
    "args": ["--headless=new"],
}
```

### Verify the extension is active

The extension registers a background service worker in the browser context. Check it from your spider:

```python
for worker in page.context.service_workers:
    if "chrome-extension" in worker.url:
        self.logger.info("cf-autoclick extension is ACTIVE: %s", worker.url)
```

You should also see `cf-autoclick extension enabled: <path>` in the Scrapy logs at startup.

### Why a persistent context?

The Cloudflare auto-click integration loads as a Chromium extension. Chromium extensions require a persistent browser profile (`user_data_dir`), so `HAMRO_CF_AUTOCLICK` only works when using a persistent Playwright context. `PLAYWRIGHT_LAUNCH_OPTIONS` are automatically merged into the persistent context by this package.
