Metadata-Version: 2.4
Name: pxm2-scrapy
Version: 1.0.0
Summary: Official Scrapy middleware and Playwright rotation helper for PXM2 4G/5G mobile proxies.
Author-email: PXM2 Team <support@pxm2.io>
License-Expression: MIT
Project-URL: Homepage, https://pxm2.io/
Project-URL: Scrapy Guide, https://pxm2.io/blog/scrapy/
Project-URL: Rotating Mobile Proxies, https://pxm2.io/proxy-types/rotating-mobile-proxies/
Project-URL: Documentation, https://pxm2.io/guides/
Project-URL: Bug Tracker, https://pxm2.io/contact/
Keywords: scrapy,playwright,proxy,mobile-proxy,rotating-proxy,pxm2,web-scraping
Classifier: Programming Language :: Python :: 3
Classifier: Framework :: Scrapy
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: Proxy Servers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# PXM2 Scrapy Middleware & Playwright Rotation Helper

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/)
[![Scrapy Compatible](https://img.shields.io/badge/scrapy-%3E%3D2.0-green.svg)](https://scrapy.org/)
[![Playwright Compatible](https://img.shields.io/badge/playwright-compatible-orange.svg)](https://playwright.dev/)

Official Downloader Middleware and browser automation helper for integrating [PXM2 4G/5G Rotating Mobile Proxies](https://pxm2.io/proxy-types/rotating-mobile-proxies/) with **Scrapy** and **Playwright**.

For an in-depth tutorial and production scraping patterns, read our comprehensive [Scrapy Web Scraping Guide](https://pxm2.io/blog/scrapy/).

---

## Features

- **Automatic Proxy Injection**: Route all Scrapy requests through PXM2 high-reputation mobile proxy gateways with authentication.
- **Smart Ban Recovery**: Detect blocks (`403 Forbidden`, `429 Too Many Requests`, or connection timeouts) and trigger instant modem IP rotation before retrying the request.
- **Frequency-Based Rotation**: Rotate cellular IPs automatically every *N* requests.
- **Hardware Cooldown Handling**: Built-in delay allowing cellular modems to complete radio disconnection and obtain a fresh carrier CGNAT IP.
- **Playwright Rotation Helper**: Ready-to-use rotator for headless Chromium/Firefox and `scrapy-playwright`.
- **Zero Heavy Dependencies**: Pure standard library implementation.

---

## Installation

```bash
pip install pxm2-scrapy
```

Or install from source:

```bash
git clone https://github.com/pxm2/pxm2-scrapy-middleware.git
cd pxm2-scrapy-middleware
pip install -e .
```

---

## Scrapy Setup

### 1. Enable Middleware in `settings.py`

Add `PXM2ProxyMiddleware` to your Scrapy project's `DOWNLOADER_MIDDLEWARES` and configure your credentials:

```python
# settings.py

DOWNLOADER_MIDDLEWARES = {
    'pxm2_scrapy.PXM2ProxyMiddleware': 610,
}

# PXM2 Mobile Proxy Credentials
PXM2_PROXY_HOST = 'proxy.pxm2.io'
PXM2_PROXY_PORT = 10001
PXM2_PROXY_USERNAME = 'your_username'
PXM2_PROXY_PASSWORD = 'your_password'

# Instant IP rotation webhook URL from your PXM2 dashboard
PXM2_ROTATION_URL = 'https://api.pxm2.io/rotate/YOUR_API_TOKEN'

# HTTP status codes that trigger automatic IP rotation and request retry
PXM2_ROTATE_ON_STATUS = [403, 429, 503]

# Rotate every N requests (0 to disable scheduled rotation)
PXM2_ROTATE_EVERY = 50

# Hardware reconnection cooldown in seconds (default: 5.0)
PXM2_ROTATION_COOLDOWN = 5.0

# Max retries per request after rotation
PXM2_MAX_RETRIES = 3
```

---

## Scrapy Spider Example

```python
import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = [
        "https://httpbin.org/ip",
        "https://example.com/products",
    ]

    def parse(self, response):
        self.logger.info("Fetched %s via IP: %s", response.url, response.text.strip())
        # Parse product details...
```

---

## Playwright Integration

Use `PXM2PlaywrightRotator` with standalone Playwright or `scrapy-playwright`:

```python
from playwright.sync_api import sync_playwright
from pxm2_scrapy import PXM2PlaywrightRotator

# Initialize rotator
rotator = PXM2PlaywrightRotator(
    host="proxy.pxm2.io",
    port=10001,
    username="your_username",
    password="your_password",
    rotation_url="https://api.pxm2.io/rotate/YOUR_API_TOKEN",
    cooldown=5.0
)

with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy=rotator.get_proxy_config(),
        headless=True
    )
    context = browser.new_context()
    page = context.new_page()

    page.goto("https://api.ipify.org?format=json")
    print("Initial IP:", page.content())

    # Trigger instant carrier IP rotation
    print("Rotating mobile IP...")
    rotator.rotate()

    page.reload()
    print("New IP after rotation:", page.content())

    browser.close()
```

---

## Configuration Reference

| Setting | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `PXM2_PROXY_HOST` | `str` | `'proxy.pxm2.io'` | Hostname or IP of the PXM2 mobile proxy gateway |
| `PXM2_PROXY_PORT` | `int` | `10001` | Gateway port number |
| `PXM2_PROXY_USERNAME` | `str` | `''` | Proxy authentication username |
| `PXM2_PROXY_PASSWORD` | `str` | `''` | Proxy authentication password |
| `PXM2_ROTATION_URL` | `str` | `None` | Webhook URL that triggers hardware modem rotation |
| `PXM2_ROTATE_ON_STATUS` | `list` | `[403, 429]` | HTTP status codes triggering automatic rotation & retry |
| `PXM2_ROTATE_EVERY` | `int` | `0` | Rotate cellular IP every N requests (0 = disabled) |
| `PXM2_ROTATION_COOLDOWN` | `float` | `5.0` | Cooldown wait time in seconds for modem reconnection |
| `PXM2_MAX_RETRIES` | `int` | `3` | Maximum retry attempts per request when rotated |

---

## Why PXM2 4G/5G Mobile Proxies?

Traditional datacenter and residential proxies are increasingly identified and blocked by anti-bot systems (Cloudflare Turnstile, DataDome, PerimeterX, Akamai).

[PXM2 Mobile Proxies](https://pxm2.io/proxy-types/rotating-mobile-proxies/) leverage real SIM modems connected to tier-1 mobile carriers. Because millions of real smartphone users share the same mobile carrier CGNAT IP pools, mobile proxies achieve the highest trust scores and bypass anti-scraping protections effortlessly.

Learn more:
- [PXM2 Rotating Mobile Proxies](https://pxm2.io/proxy-types/rotating-mobile-proxies/)
- [Scrapy Integration Guide](https://pxm2.io/blog/scrapy/)
- [Documentation & Tutorials](https://pxm2.io/guides/)
- [Contact Support](https://pxm2.io/contact/)

---

## License

MIT License. See [LICENSE](LICENSE) for details.
