Metadata-Version: 2.5
Name: scrapy-seleniumbase-cdp
Version: 2.1.1
Summary: Scrapy downloader middleware that uses SeleniumBase's pure CDP mode to make requests.
Project-URL: Homepage, https://github.com/nyg/scrapy-seleniumbase-cdp
Project-URL: Issues, https://github.com/nyg/scrapy-seleniumbase-cdp/issues
Author: nyg
License: MIT
License-File: LICENSE
Keywords: cdp,chrome-devtools-protocol,crawler,middleware,scrapy,seleniumbase,web-scraping
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.11
Requires-Dist: mycdp>=1.0
Requires-Dist: scrapy>=2.0
Requires-Dist: seleniumbase>=4.0
Description-Content-Type: text/markdown

# scrapy-seleniumbase-cdp

[![PyPI](https://img.shields.io/pypi/v/scrapy-seleniumbase-cdp)](https://pypi.org/project/scrapy-seleniumbase-cdp/)
[![Python Versions](https://img.shields.io/pypi/pyversions/scrapy-seleniumbase-cdp)](https://pypi.org/project/scrapy-seleniumbase-cdp/)
[![License](https://img.shields.io/pypi/l/scrapy-seleniumbase-cdp)](https://github.com/nyg/scrapy-seleniumbase-cdp/blob/master/LICENSE)
[![Downloads](https://img.shields.io/pypi/dm/scrapy-seleniumbase-cdp)](https://pypi.org/project/scrapy-seleniumbase-cdp/)

Scrapy downloader middleware that uses [SeleniumBase][4]'s pure CDP mode to make
requests, allowing to bypass most anti-bot protections (e.g. CloudFlare).

Using Selenium's pure CDP mode also makes the middleware more platform
independent as no WebDriver is required.

## Table of contents

- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
  - [Additional arguments](#additional-arguments)
    - [`page_load_timeout`](#page_load_timeout)
    - [Captcha handling](#captcha-handling)
    - [`wait_for_element` / `element_timeout`](#wait_for_element--element_timeout)
    - [`browser_callback`](#browser_callback)
    - [`script`](#script)
    - [`screenshot`](#screenshot)
- [Error handling](#error-handling)
- [Stats](#stats)
- [Enabling debug logs](#enabling-debug-logs)
- [Tips for headless Linux environments](#tips-for-headless-linux-environments)
  - [Recording an Xvfb session with ffmpeg](#recording-an-xvfb-session-with-ffmpeg)
  - [Connecting via VNC to an Xvfb session](#connecting-via-vnc-to-an-xvfb-session)
- [Architecture](#architecture)
- [License](#license)

## Installation

```
pip install scrapy-seleniumbase-cdp
```

## Configuration

1. Add the `SeleniumBaseAsyncCDPMiddleware` to the downloader middlewares:
    ```python
    DOWNLOADER_MIDDLEWARES = {
        'scrapy_seleniumbase_cdp.SeleniumBaseAsyncCDPMiddleware': 800
    }
    ```

2. If needed, configuration can be provided to the SeleniumBase browser instance.
   For example, to enable the built-in ad blocker (blocks 30+ ad and tracking
   domains via CDP):

   ```python
   SELENIUMBASE_BROWSER_OPTIONS = {
       'ad_block': True,
   }
   ```

## Usage

To have SeleniumBase handle requests, use the
`scrapy_seleniumbase_cdp.SeleniumBaseRequest` instead of Scrapy's built-in
`Request`:

```python
from scrapy_seleniumbase_cdp import SeleniumBaseRequest

async def start(self):
    yield SeleniumBaseRequest(url=url, callback=self.parse_result)
```

`response.meta['cookies']` holds the cookies Chrome would send to the page's final URL, as a list of CDP cookie dicts (`name`, `value`, `domain`, `path`, `secure`, `httpOnly`, `expires`, …). Scrapy's `Request(cookies=...)` accepts that list, so a plain request can reuse the browser's session:

```python
def parse_result(self, response):
    yield scrapy.Request(next_url, cookies=response.meta['cookies'])
```

Chrome loads every `SeleniumBaseRequest` as a `GET`: a request's method and body are not sent, and a non-`GET` request logs a warning. Of the request's headers, only `Referer` is passed on to Chrome, which sends its own `User-Agent`, `Cookie` and other headers.

### Additional arguments

The `scrapy_seleniumbase_cdp.SeleniumBaseRequest` accepts additional
arguments. They are executed in the order presented below:

#### `page_load_timeout`

Maximum number of seconds to wait for both the page's document response and the page load event before proceeding. If only the load event is missing when the timeout is reached, a warning is logged and the request continues. If the document response is missing, the request fails with `TimeoutError` (see [Error handling](#error-handling)). Defaults to `10`.

#### Captcha handling

After navigating to a page, the middleware waits for both the HTTP response
status and the page load event. It then attempts to solve any captcha present
on the page using SeleniumBase's built-in solver, retrying up to a configurable
maximum number of attempts.

The delay before each solve attempt depends on the latest HTTP status code of the page:

- **Blocked responses** (status in `captcha_blocked_codes`): wait
  `captcha_blocked_delay` seconds (default `4`)
- **Any other status**: wait `captcha_delay` seconds (default `0`)

So when a click on a captcha reloads the page with a `200`, the check that follows waits `captcha_delay`, not `captcha_blocked_delay`.

```python
yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    captcha_delay=1,
    captcha_blocked_delay=5,
    captcha_blocked_codes=[403, 429, 503],
    captcha_max_attempts=5)
```

Available captcha configuration:

- `captcha_delay`: Seconds to wait before solving when the status is not in `captcha_blocked_codes`. Defaults to `0`.
- `captcha_blocked_delay`: Seconds to wait before solving on a blocked
  response. Defaults to `4`.
- `captcha_blocked_codes`: List of HTTP status codes treated as blocked.
  Defaults to `[403, 429, 503]`.
- `captcha_max_attempts`: Maximum number of solve attempts. Defaults to `3`.
  After exhausting all attempts the middleware continues normally but logs a
  warning. Set it to `0` to skip captcha solving.
- `captcha_only_when_blocked`: Attempt captcha solving only when the page loads with a status in `captcha_blocked_codes`. Defaults to `False`, which checks every page, as some sites show a captcha on a `200` page. Each check reads the whole page and took about 1.5s per page on a Raspberry Pi, so set it to `True` when the site serves its challenges with a blocked status.

A click on a captcha only starts its verification, so after each click the middleware waits up to `page_load_timeout` seconds for the page to reload before the next attempt checks whether the captcha is gone.

`response.meta['captcha_unsolved']` is `True` when every attempt still found a captcha, and `False` otherwise. A challenge page can pass `wait_for_element` (Cloudflare's has an `h1`), so check it to tell a blocked page from the real one:

```python
def parse_result(self, response):
    if response.meta['captcha_unsolved']:
        self.logger.warning(f'blocked by a captcha: {response.url}')
        return
```

#### `wait_for_element` / `element_timeout`

When used, SeleniumBase will wait for the element with the given CSS selector to appear. The default timeout value is of 10 seconds but can be changed if needed. If the element is not found within the timeout, a full-page error screenshot is captured and stored in `request.meta['error_screenshot']`, then a `TimeoutError` is raised. The screenshot image format is taken from the request's `screenshot` configuration if set, otherwise it defaults to PNG.

`TimeoutError` is an `OSError`, so Scrapy's `RetryMiddleware` retries the request (`RETRY_TIMES`, 2 by default; set `meta={'dont_retry': True}` to skip that). Once retries run out, the request's `errback` receives the `TimeoutError`, and the last attempt's error screenshot is in `failure.request.meta['error_screenshot']`:

```python
yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    errback=self.handle_error,
    wait_for_element='h1.some-class',
    element_timeout=5)


def handle_error(self, failure):
    screenshot = failure.request.meta.get('error_screenshot')
    if screenshot:
        with open('error.png', 'wb') as f:
            f.write(screenshot)
```

#### `browser_callback`

If needed, it is possible to provide a callback to interact with the browser
instance and/or its tabs. The return value of the async callback is stored in
`response.meta['callback']`. Requests always load in the tab the middleware set up when the spider opened, so a tab the callback opens does not take them over.

```python
async def start(self):
    async def maximize_window(browser: Browser):
        await browser.main_tab.maximize()

    yield SeleniumBaseRequest(…, browser_callback=maximize_window)
```

#### `script`

When used, SeleniumBase will execute the provided JavaScript code.

```python
yield SeleniumBaseRequest(
    # …
    script='window.scrollTo(0, document.body.scrollHeight)')
```

If the script returns a Promise, it is possible to await its result:

```python
yield SeleniumBaseRequest(
    # …
    script={
        'await_promise': True,
        'script': '''
            document.getElementById('onetrust-accept-btn-handler').click()
            new Promise(resolve => setTimeout(resolve, 1000))
        '''
    })
```

The result of the JavaScript code is stored in `response.meta['script']`.

#### `screenshot`

When used, SeleniumBase will take a screenshot of the page and the binary data
will be stored in `response.meta['screenshot']`:

```python
yield SeleniumBaseRequest(url=url, callback=self.parse_result, screenshot=True)


def parse_result(self, response):
    # …
    with open('image.png', 'wb') as image_file:
        image_file.write(response.meta['screenshot'])
```

You can also specify additional configuration options:

```python
yield SeleniumBaseRequest(…, screenshot={'format': 'jpg', 'full_page': False})
```

Or provide a path to automatically save the screenshot (in this case, the image
data is **not** stored in the response):

```python
yield SeleniumBaseRequest(…, screenshot={'path': 'output/image.png'})
```

Available configuration keys:

- `path`: File path where screenshot will be saved. Use `auto` for
  SeleniumBase default path. Leave empty to return data in response `meta`.
- `format`: Image format, defaults to `png`, `jpg` also available.
- `full_page`: Capture full page or just viewport, defaults to `True`.

## Error handling

The middleware checks the HTTP status code right after loading the page to
determine captcha-solving behaviour (see [Captcha handling](#captcha-handling)
above).

The response returned to Scrapy carries the status of the last document response the tab's main frame received, so a redirect gives the target page's status and an iframe's status is ignored. When a solved captcha reloads the page, that is the reloaded page's status (usually `200`), not the challenge's `403` or `429`, so Scrapy's `RetryMiddleware` does not retry a page that loaded fine.

- **No document response**: if the main frame gets no document response within `page_load_timeout` seconds, for example because the server is too slow or the host cannot be resolved, an error is logged and `TimeoutError` is raised. Scrapy's `RetryMiddleware` retries the request, then passes the `TimeoutError` to the request's `errback`.
- **`wait_for_element` timeout**: if the expected element is not found within `element_timeout` seconds, a full-page error screenshot is captured and stored in `request.meta['error_screenshot']`, then `TimeoutError` is raised. Scrapy's `RetryMiddleware` retries the request, then passes the `TimeoutError` to the request's `errback`, where the screenshot is in `failure.request.meta['error_screenshot']` (see [`wait_for_element`](#wait_for_element--element_timeout) for an example).

When a CDP command fails, for example one sent by a browser callback, SeleniumBase closes the tab's DevTools connection, and the next command opens a new session, which has lost the tab's settings (user agent, locale, timezone, ad block, …). The next request then logs a warning, loads in a new tab set up like the first one, and the old tab is closed. If the session is lost while a captcha is clicked, the page's reload would load without those settings, so an error is logged and `ConnectionError` is raised: Scrapy's `RetryMiddleware` retries the request, in a new tab, then passes the `ConnectionError` to the request's `errback`.

## Stats

The middleware counts captcha outcomes in Scrapy's stats (`crawler.stats`), so they show up in the stats dump at the end of a crawl and in any project that persists stats, whatever the log level:

- `captcha/detected`: requests whose page showed a captcha on the first solve attempt.
- `captcha/solved`: requests where a later attempt found the captcha gone.
- `captcha/unsolved`: requests where every attempt still found a captcha (`captcha_max_attempts` reached).
- `seleniumbase/response_status_count/<code>`: the status the page loaded with, before any captcha was solved. Scrapy's own `downloader/response_status_count/<code>` counts the status returned to Scrapy, which is the reloaded page's after a solved captcha. A request that gets no document response is not counted here, as it fails with `TimeoutError`.

For example, many `seleniumbase/response_status_count/403` with no `captcha/detected` means the site blocks without a captcha the solver recognises, while a high `captcha/unsolved` means the solver runs but fails.

A failing browser callback, script or screenshot is logged and the response still goes through without its result, so each failure is also counted: `seleniumbase/browser_callback_failed`, `seleniumbase/script_failed`, `seleniumbase/screenshot_failed`, and `seleniumbase/error_screenshot_failed` for the screenshot taken when `wait_for_element` times out.

## Tips for headless Linux environments

When running Scrapy with this middleware in headless mode using Xvfb on Linux,
you may want to record or visually inspect browser sessions for debugging
purposes. The examples below assume an Xvfb display at `:1001` — adjust to
match your setup.

### Recording an Xvfb session with ffmpeg

Use `ffmpeg` to capture the virtual display as a video file:

```bash
ffmpeg -f x11grab -r 30 -s 1440x900 -i :1001 \
    -codec:v libx264 -preset ultrafast -pix_fmt yuv420p \
    /home/user/session_$(date +%Y%m%d_%H%M%S).mp4
```

Key flags:
- `-f x11grab` — capture from an X11 display
- `-r 30` — frame rate (30 fps)
- `-s 1440x900` — resolution (must match your Xvfb geometry)
- `-i :1001` — X display to capture

### Connecting via VNC to an Xvfb session

Use `x11vnc` to expose the virtual display over VNC for live inspection:

```bash
x11vnc -display :1001 -passwd secret -forever -xkb
```

Then connect from any VNC client to `<host>:5900`. Key flags:
- `-display :1001` — X display to share
- `-passwd secret` — VNC password
- `-forever` — keep the server running after the first client disconnects
- `-xkb` — use XKEYBOARD extension for better keyboard handling

## Enabling debug logs

The middleware logs operational details (page load events, captcha attempts,
screenshot captures, etc.) at the `DEBUG` level. Log messages are emitted under
the `scrapy_seleniumbase_cdp.middleware_async` logger name. Warnings and errors
(page load timeouts, element wait timeouts, max captcha attempts reached) use
higher log levels and are always visible.

To see all debug output, set Scrapy's global log level in your `settings.py`:

```python
LOG_LEVEL = 'DEBUG'
```

If you prefer to keep Scrapy's own output at a higher level and only enable
debug logging for this middleware, configure the parent logger directly:

```python
# settings.py or spider __init__
import logging
logging.getLogger('scrapy_seleniumbase_cdp').setLevel(logging.DEBUG)
```

This works because the middleware uses a module-level logger named
`scrapy_seleniumbase_cdp.middleware_async`, which inherits settings from the
`scrapy_seleniumbase_cdp` parent logger.

You can also use Scrapy's per-module log configuration via the
[`LOG_CATEGORIES`][6] setting (Scrapy ≥ 2.8):

```python
LOG_CATEGORIES = {
    'scrapy_seleniumbase_cdp': 'DEBUG',
}
```

## Architecture

See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for a detailed overview of
the middleware internals, including a sequence diagram of the request processing
pipeline.

## License

This project is licensed under the MIT License. It is a fork
of [Quartz-Core/scrapy-seleniumbase][1]
which was originally released under the WTFPL.

[1]: https://github.com/Quartz-Core/scrapy-seleniumbase
[4]: https://seleniumbase.io/examples/cdp_mode/ReadMe/
[5]: https://github.com/nyg/autoscout24-trends
[6]: https://docs.scrapy.org/en/latest/topics/settings.html#log-categories
