Metadata-Version: 2.4
Name: scrapy-crawio
Version: 1.2.1
Summary: Scrapy downloader middleware for Crawio: route requests through the Crawio API, with proxies and anti-bot handling included.
Author-email: Crawio <support@crawio.com>
License-Expression: MIT
Project-URL: Homepage, https://crawio.com
Project-URL: Documentation, https://crawio.com/docs
Keywords: scrapy,scraping,proxy,crawler,anti-bot,crawio
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scrapy>=2.6
Requires-Dist: requests>=2.25
Dynamic: license-file

# scrapy-crawio

Scrapy downloader middleware for [Crawio](https://crawio.com).

Your spiders make normal requests; this middleware routes them through the Crawio API, which
fetches each one from a healthy exit — using the cheapest route that works, up to a full browser
when the site's anti-bot needs one — and returns the site's own response. No spider changes, no
per-request proxy juggling. Only successful requests are billed.

## Install

```bash
pip install scrapy-crawio
```

## Configure (`settings.py`)

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_crawio.CrawioMiddleware": 1000,
}

CRAWIO_API_KEY = "rk-..."                   # required — dashboard -> API Key
CRAWIO_DOMAINS = ["example.com"]            # hosts to route; omit to route every request
CRAWIO_API_URL = "https://api.crawio.com"   # optional
CRAWIO_TIMEOUT = 120                        # seconds, optional
CRAWIO_SESSION = "job-1"                    # optional: default sticky session id
CRAWIO_COUNTRY = "DE"                       # optional: fetch every request from this country

# Recommended: let Scrapy retry the transient answers.
RETRY_ENABLED = True
RETRY_HTTP_CODES = [429, 502, 503, 504]
```

## Use

Write spiders exactly as normal — the matched hosts go through Crawio:

```python
import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://example.com/catalogue/page-1.html"]

    def parse(self, response):
        self.logger.info("fetched in %s ms", response.headers.get("X-Crawio-Ms", b"").decode())
        for href in response.css("h3 a::attr(href)").getall():
            yield response.follow(href, callback=self.parse_item)
```

A page or JSON arrives as a `TextResponse`. A file (PDF, image, archive) arrives as a plain
`scrapy.http.Response` with its original bytes in `response.body`.

## Response headers

Every response through Crawio carries these, alongside the site's own headers:

| Header | Meaning |
|---|---|
| `X-Crawio-Request-Id` | The request's reference (`req_...`). Quote it when you contact support. |
| `X-Crawio-Cost` | Credits this request used (most sites `1`), or `0` when it was not billed (blocks, errors, timeouts). |
| `X-Crawio-Ms` | Crawio's own time for the fetch, in milliseconds. |

Error responses (429, 502, 503, 504) carry `X-Crawio-Request-Id` and `X-Crawio-Cost` too, and a 429
carries `Retry-After`.

## Per-request overrides

```python
scrapy.Request(url, meta={"crawio_skip": True})       # download this one directly, not through us
scrapy.Request(url, meta={"crawio_session": "job-2"}) # pin to a sticky session
scrapy.Request(url, meta={"crawio_country": "FR"})    # fetch this one from France
```

A sticky session keeps the same exit and the same cookies across requests: sign in once, reuse the
id, and the following requests stay signed in.

## Countries

Fetch a page the way a visitor in another country sees it: local prices, local search results, pages
that only open inside a country. Set a two-letter ISO 3166-1 code (case-insensitive) for the whole
crawl, for one request, or both:

```python
# settings.py: every request from Germany
CRAWIO_COUNTRY = "DE"
```

```python
# one request from Brazil, one with no country (the best exit for the site)
yield scrapy.Request("https://example.com/precos", meta={"crawio_country": "BR"})
yield scrapy.Request("https://example.com/about", meta={"crawio_country": ""})
```

Leave the country out unless the page depends on it: Crawio then picks the best exit for each site.
A country Crawio cannot serve answers `400` with `error: country_unavailable` before anything runs,
and it is not billed. With a sticky session, keep the same country on every request of the session.
Every country, and how the API answers, is in the [docs](https://crawio.com/docs#countries).

## Retries and billing

Every request carries an `Idempotency-Key`. When the connection to Crawio drops or times out before an
answer arrives, the middleware raises `scrapy_crawio.CrawioUnreachable` (an `OSError`, which Scrapy's
`RetryMiddleware` retries), and the retry carries the same key: it reattaches to the request already
running instead of starting, and billing, a second one. An answer Crawio has returned (a `502` or
`504`, say) is final for its key, so a retry of it is sent as a new request with a new key. A `503`
keeps its key. Errors are never billed, so retrying them never costs a credit.

## What Crawio answers

| code | meaning |
|------|---------|
| 200 | success — the site's response is in the body |
| 400 | `country_unavailable`: the country you asked for cannot be served; not billed, not retried |
| 401 | invalid API key; the request is dropped, not retried |
| 403 | account suspended; fix the account, do not retry |
| 410 | a strict sticky session lost its exit — sign in again on a new session |
| 422 | the request is not valid (for example a country that is not two letters); the body says what to fix |
| 429 | a plan limit: the body names which (`rate_limited`, `concurrency_limit`, `daily_quota_exceeded`, `monthly_quota_exceeded`, `credits_used_up`) and `Retry-After` says when, where the wait is known |
| 502 | the site blocked the request behind its anti-bot |
| 503 | no capacity right now; retry |
| 504 | the fetch timed out |

Everything except 401 comes back as a real `Response`, so `RetryMiddleware` decides what to do with
it and the body tells you why. Until 1.2.0 a retried `502`/`504` was refused with `409` (it reused the
finished request's key) and a dropped connection was not retried at all.

## Scrapy versions

Scrapy 2.6 and later. Since 1.2.1 the middleware's `process_request` is a coroutine and takes no
`spider` argument, so Scrapy 2.13 and later print no deprecation warning for it (1.2.0 printed one when
the crawl started and one per request on Scrapy 2.19).

## License

MIT
