Metadata-Version: 2.5
Name: scrapy-curl
Version: 0.1.1
Summary: Scrapy download handler built on curl_cffi: browser TLS impersonation, exact header control, and proxy credentials that never reach the target site.
Project-URL: Homepage, https://github.com/alikamal-jaffri/scrapy-curl
Project-URL: Repository, https://github.com/alikamal-jaffri/scrapy-curl
Project-URL: Issues, https://github.com/alikamal-jaffri/scrapy-curl/issues
Project-URL: Changelog, https://github.com/alikamal-jaffri/scrapy-curl/blob/main/CHANGELOG.md
Author: Ali Kamal
License: MIT License
        
        Copyright (c) 2026 Ali Kamal
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: curl_cffi,fingerprint,impersonate,ja3,proxy,scrapy,tls
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.9
Requires-Dist: curl-cffi>=0.7
Requires-Dist: pump>=0.1.2
Requires-Dist: pytest>=8.4.2
Requires-Dist: scrapy>=2.13
Provides-Extra: test
Requires-Dist: pytest>=7; extra == 'test'
Description-Content-Type: text/markdown

# scrapy-curl

[![PyPI](https://img.shields.io/pypi/v/scrapy-curl.svg)](https://pypi.org/project/scrapy-curl/)
[![Python](https://img.shields.io/pypi/pyversions/scrapy-curl.svg)](https://pypi.org/project/scrapy-curl/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

A Scrapy download handler built on [curl_cffi](https://github.com/lexiforest/curl_cffi).
You get browser TLS/JA3 impersonation, byte-exact control over the header block,
and proxy credentials that go to your proxy and nowhere else.

It's a drop-in replacement for Scrapy's HTTP handler. Requests you mark go
through curl_cffi; everything else takes the normal path, so you can adopt it
one spider at a time.

- [Why this exists](#why-this-exists)
- [What you get](#what-you-get)
- [Requirements](#requirements)
- [Installation](#installation)
- [Getting started](#getting-started)
- [Headers, in detail](#headers-in-detail)
- [Proxies](#proxies)
- [What stays with Scrapy](#what-stays-with-scrapy)
- [Settings](#settings)
- [Per-request options](#per-request-options)
- [Connection reuse](#connection-reuse)
- [How this differs from similar packages](#how-this-differs-from-similar-packages)
- [Troubleshooting](#troubleshooting)
- [Contributing](#contributing)
- [License](#license)

## Why this exists

Scrapy doesn't keep proxy credentials in `request.meta["proxy"]`. Its
`HttpProxyMiddleware` pulls them out and moves them into a request header:

```python
request.meta["proxy"] = "http://gate.example.com:7000"        # credentials gone
request.headers["Proxy-Authorization"] = b"Basic dXNlcjpwYXNz"  # they live here now
```

`Proxy-Authorization` is a hop-by-hop header. It addresses the proxy, not the
site. So any download handler that takes `request.headers` and passes it
straight to an HTTP client is handing your proxy password to the site you're
crawling. On HTTPS it rides inside the CONNECT tunnel, which means the proxy
never sees it and can't strip it on the way past.

The reason this survives code review is that nothing looks broken. libcurl
defaults to `CURLHEADER_SEPARATE` and authenticates the proxy from a separate
option, so the proxy leg works, pages come back, the crawl is green. The
credentials just also arrive at the destination, on every single request, for
as long as the spider runs.

This package treats "credentials don't reach the origin" as the property to
protect rather than a step to remember. `Proxy-Authorization`,
`Proxy-Authenticate` and `Proxy-Connection` are stripped from every outgoing
request — whether or not a proxy is set, whether or not the credentials parsed,
and whether they came from Scrapy, your own middleware, or `raw_headers`.
Credentials reach libcurl through `CURLOPT_PROXYUSERNAME` and
`CURLOPT_PROXYPASSWORD`, which only ever apply to the proxy leg.

The test suite checks this against a real CONNECT proxy and a real origin
server, asserting on the bytes each one received. There's also a canary test
that deliberately leaks and asserts the harness notices, so the suite can't
quietly stop testing anything.

## What you get

- **Browser TLS fingerprints** — JA3, Akamai HTTP/2 fingerprint, and the rest
  of curl_cffi's `impersonate` targets.
- **Header block you actually control** — your order, your capitalisation,
  no template merged in behind your back.
- **Proxy credentials that stay on the proxy leg** — HTTP, HTTPS and SOCKS5,
  with credentials in the URL or in the header.
- **Scrapy stays in charge** — cookies, redirects and decompression are still
  handled by the middlewares you already have configured.
- **One connection pool for the crawl**, not a fresh session per request.
- **Opt-in per request**, so you can migrate gradually.

## Requirements

- Python 3.9+
- Scrapy 2.13+
- curl_cffi 0.7+
- The asyncio Twisted reactor

Scrapy 2.14 reworked the download handler API: `download_request` became a
coroutine and stopped receiving a `spider` argument, `close` became a
coroutine, and `__init__` started taking just the crawler. Scrapy picks which
convention to use by checking whether a handler's `download_request` is a
coroutine function, so a handler has to commit to one or the other.

This one detects the installed Scrapy from the base class and defines the
matching pair, so the same release works either side of that change. The test
suite runs against both, and asserts the convention matches — see
`TestScrapyApiContract` in `tests/test_handler.py`.

## Installation

```bash
pip install scrapy-curl
```

## Getting started

Point the download handlers at this package and make sure you're on the asyncio
reactor:

```python
# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_curl.CurlDownloadHandler",
    "https": "scrapy_curl.CurlDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
```

Then mark the requests you want impersonated:

```python
import scrapy


class ProductSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/api/product/123",
            meta={"curl": {"impersonate": "chrome"}},
        )

    def parse(self, response):
        yield response.json()
```

Requests without `meta["curl"]` go through Scrapy's normal handler, so nothing
else in your project changes.

If you'd rather impersonate everything, set a default and forget about the meta
key:

```python
CURL_IMPERSONATE = "chrome"
```

Individual requests can still opt out with `meta["curl"] = False` — handy for
hitting an internal API where a Chrome fingerprint would just be noise.

## Headers, in detail

This is the part that usually costs people a weekend, so it's worth being
explicit about.

### `default_headers`: three states

curl_cffi can supply the impersonated browser's own header block for you. That
is genuinely useful, and it's also the thing that silently rewrites a header
block you spent time building — when it's on, your headers get *merged* into
the template: names the template knows about inherit the template's position,
and names it doesn't know about get appended at the end.

So the flag has three states rather than two:

| Value | Behaviour |
| --- | --- |
| `True` | Use curl_cffi's template for the impersonated browser. Your headers merge into it. |
| `False` | Send exactly the headers you gave, in the order you gave them. |
| unset (default) | Decide per request: `False` when the request has headers, `True` when it has none. |

The default is the interesting one. Most of the time you have headers and you
want them sent as-is, so it behaves like `False`. But a request with no headers
and no template goes out looking like nothing on the internet — a worse
fingerprint than the template it was avoiding. In that case it falls back to
the template.

Set it globally:

```python
CURL_DEFAULT_HEADERS = True   # always use curl_cffi's browser headers
```

or per request, which wins over the setting:

```python
meta={"curl": {"impersonate": "chrome", "default_headers": False}}
```

Leave both unset to get the per-request behaviour described above.

### Capitalisation, and why `raw_headers` exists

`scrapy.http.Headers` is a case-insensitive dict that stores every name as
`name.title()`. That turns `sec-ch-ua` into `Sec-Ch-Ua` and `DNT` into `Dnt`.
No browser sends those. Scrapy's own HTTP/1.1 handler has the same behaviour,
so it stays invisible until you diff against a real capture and wonder why a
site is treating you differently.

The case is destroyed at assignment time, so there's nothing to recover later.
`raw_headers` skips the dict:

```python
yield scrapy.Request(
    url,
    meta={"curl": {
        "impersonate": "chrome",
        "raw_headers": [
            ("sec-ch-ua", '"Chromium";v="142", "Not_A Brand";v="99"'),
            ("sec-ch-ua-mobile", "?0"),
            ("sec-ch-ua-platform", '"Windows"'),
            ("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"),
            ("Accept", "application/json"),
            ("Accept-Language", "es-MX"),
        ],
    }},
)
```

`raw_headers` is used verbatim: same order, same capitalisation, duplicates
kept. It replaces `request.headers` entirely for that request.

Ordinary `request.headers` still work fine and still keep their order — only
the capitalisation is Scrapy's rather than yours.

### One thing you can't control

curl drops a `Host` header that matches the request URL's authority and writes
its own, always first in the block. That's what browsers do on HTTP/1.1 anyway,
but it does mean `Host` can't be moved.

## Proxies

`http://`, `https://` and `socks5://` proxies are supported. Credentials can
live in either place, and both work:

```python
# in the URL
request.meta["proxy"] = "http://user:pass@gate.example.com:7000"

# or in the header, which is where Scrapy's HttpProxyMiddleware puts them
request.meta["proxy"] = "http://gate.example.com:7000"
request.headers["Proxy-Authorization"] = b"Basic dXNlcjpwYXNz"
```

Percent escapes are decoded from URLs, because that's URL syntax, and *not*
decoded from the header, because Scrapy already decoded them before
base64-encoding. Decoding twice quietly corrupts any password containing a
literal `%`.

If a `Proxy-Authorization` header can't be decoded — a Digest or NTLM
challenge, say — the request goes out with no proxy credentials at all and the
proxy answers `407`. That's deliberate. Failing loudly beats falling back to
sending credentials somewhere they don't belong.

The session runs with `trust_env=False`, so proxies come from Scrapy only. A
stray `HTTP_PROXY` in the environment won't quietly reroute your crawl.

## What stays with Scrapy

Redirects, cookies and decompression stay where you already configured them
(`allow_redirects=False`, `discard_cookies=True`, `accept_encoding=None`):

- `RedirectMiddleware` sees the 3xx, so `dont_redirect`, redirect limits and
  redirect stats all behave normally.
- `CookiesMiddleware` keeps the jar and decides where the `Cookie` header sits
  in the block — which matters if you care about header order.
- `HttpCompressionMiddleware` does the decoding, so `Content-Encoding` is
  passed through to it intact.

Turning off curl's own `Accept-Encoding` also stops it contradicting the one
you're impersonating.

## Settings

| Setting | Default | What it does |
| --- | --- | --- |
| `CURL_IMPERSONATE` | unset | Browser target for every request, e.g. `"chrome"`. Unset means opt-in per request. |
| `CURL_DEFAULT_HEADERS` | unset | Whether curl_cffi supplies the browser's default headers. Unset decides per request — see [above](#default_headers-three-states). |
| `CURL_MAX_CLIENTS` | `CONCURRENT_REQUESTS` | Size of the curl_cffi connection pool. |
| `CURL_VERIFY` | `False` | Verify TLS certificates. `False` matches Scrapy's own default context factory. |

`DOWNLOAD_TIMEOUT` and `HTTPPROXY_AUTH_ENCODING` are honoured too.

## Per-request options

Everything goes in `meta["curl"]`:

| Key | What it does |
| --- | --- |
| `impersonate` | Browser target, e.g. `"chrome"`, `"chrome131"`, `"safari"`. |
| `ja3` | Custom JA3 string. |
| `akamai` | Custom Akamai HTTP/2 fingerprint. |
| `extra_fp` | curl_cffi's extra fingerprint options. |
| `http_version` | Force an HTTP version. |
| `verify` | Per-request TLS verification. |
| `raw_headers` | Ordered `(name, value)` pairs, used verbatim. |
| `default_headers` | Override the setting for this request. |

Unknown keys raise `ValueError` instead of being ignored. A typo in
`impersonate` shouldn't quietly cost you the fingerprint you thought you had.

`download_timeout` and `bindaddress` are read from `meta` directly, as usual.

## Connection reuse

One `AsyncSession` is shared across the crawl rather than built per request.
curl reuses its connections and keeps TLS sessions warm, so you're not
re-running the handshake that impersonation exists to shape — which is both
slower and more conspicuous. The pool is sized from `CONCURRENT_REQUESTS` and
closed with the handler.

## How this differs from similar packages

There are two other curl_cffi handlers for Scrapy, and both are worth knowing
about. Observations below are from `scrapy-impersonate` 1.6.3 and
`scrapy-curl-cffi` 0.2.0.

- **`scrapy-impersonate`** pops `Proxy-Authorization` from a *copy* of the
  request while building curl options, then reads headers from the untouched
  original — so the header still goes to the origin. It also leaves
  `default_headers` on, which merges your headers into the browser template.
- **`scrapy-curl-cffi`** handles the credentials correctly. It applies
  `unquote()` to them a second time after decoding, though, which corrupts
  passwords containing a literal `%`, and it doesn't offer header-order
  control.

If you don't use authenticated proxies and don't care about exact header
order, any of the three will serve you. This one exists for the case where you
do.

## Troubleshooting

**`ValueError: ... requires the asyncio Twisted reactor`**
Set `TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"`.

**The proxy returns 407.**
Check the logs for a warning about an undecodable `Proxy-Authorization` header.
If the proxy uses Digest or NTLM rather than Basic, put the credentials in
`meta["proxy"]` as a URL instead.

**My headers come out in a different order.**
Something turned `default_headers` on — check `CURL_DEFAULT_HEADERS` and your
`meta["curl"]`. If the request had no headers of its own, the per-request
default enables the template on purpose.

**A header name comes out capitalised differently than I set it.**
Scrapy title-cases names in `request.headers`. Use `raw_headers`.

**Responses look like binary garbage.**
`HttpCompressionMiddleware` is probably disabled. This package leaves bodies
encoded on purpose and relies on that middleware to decode them.

## Contributing

```bash
git clone https://github.com/alikamal-jaffri/scrapy-curl
cd scrapy-curl
pip install -e ".[test]"
pytest
```

The tests spin up a local origin server and a local CONNECT proxy on ephemeral
ports and assert on what each of them received. No network access needed.

If you're touching anything near proxy handling, keep
`test_harness_would_notice_a_leak` passing — it's what stops the security tests
from becoming decorative.

## License

MIT — see [LICENSE](LICENSE).
