Metadata-Version: 2.4
Name: graph-crawler
Version: 5.2.0
Summary: Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
Author: 0-EternalJunior-0
Maintainer: 0-EternalJunior-0
License-Expression: MIT
Project-URL: Homepage, https://gitlab.com/demoprogrammer/web_graf
Project-URL: Documentation, https://gitlab.com/demoprogrammer/web_graf/-/blob/main/web_graf/README.md
Project-URL: Repository, https://gitlab.com/demoprogrammer/web_graf
Project-URL: Changelog, https://gitlab.com/demoprogrammer/web_graf/-/blob/main/web_graf/CHANGELOG.md
Project-URL: Bug Tracker, https://gitlab.com/demoprogrammer/web_graf/-/issues
Keywords: web,crawler,scraper,graph,spider,playwright,anti-bot,tls-fingerprint
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp>=3.13.0
Requires-Dist: backports.zstd>=1.6.0; python_version < "3.14"
Requires-Dist: Brotli
Requires-Dist: selectolax>=0.4.0
Requires-Dist: defusedxml>=0.7.0
Requires-Dist: pydantic>=2.12.0
Requires-Dist: orjson>=3.11.0
Requires-Dist: aiofiles>=24.1.0
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: typing_extensions>=4.5; python_version < "3.12"
Requires-Dist: xxhash>=3.0.0
Provides-Extra: native
Requires-Dist: cython>=3.0.0; extra == "native"
Requires-Dist: mmh3>=5.0.0; extra == "native"
Provides-Extra: browser
Requires-Dist: playwright>=1.58.0; extra == "browser"
Provides-Extra: playwright
Requires-Dist: graph-crawler[browser]; extra == "playwright"
Provides-Extra: tls
Requires-Dist: curl_cffi>=0.7; extra == "tls"
Provides-Extra: stealth
Requires-Dist: patchright>=1.52; extra == "stealth"
Requires-Dist: playwright-stealth>=1.0; extra == "stealth"
Provides-Extra: cloak
Requires-Dist: cloakbrowser>=0.5; extra == "cloak"
Requires-Dist: playwright>=1.58.0; extra == "cloak"
Provides-Extra: undetected
Requires-Dist: nodriver>=0.38; extra == "undetected"
Provides-Extra: antidetect
Requires-Dist: graph-crawler[browser,cloak,stealth,tls,undetected]; extra == "antidetect"
Provides-Extra: server
Requires-Dist: fastapi>=0.135.0; extra == "server"
Requires-Dist: uvicorn>=0.30; extra == "server"
Requires-Dist: websockets>=13; extra == "server"
Provides-Extra: text
Requires-Dist: tiktoken>=0.13.0; extra == "text"
Requires-Dist: stop-words>=2018.7.23; extra == "text"
Provides-Extra: captcha
Provides-Extra: mongodb
Requires-Dist: pymongo[srv]>=4.9.0; extra == "mongodb"
Provides-Extra: postgresql
Requires-Dist: asyncpg>=0.30.0; extra == "postgresql"
Provides-Extra: newspaper
Requires-Dist: newspaper3k>=0.2.8; extra == "newspaper"
Provides-Extra: goose
Requires-Dist: goose3>=3.1.0; extra == "goose"
Provides-Extra: readability
Requires-Dist: readability-lxml>=0.8.0; extra == "readability"
Provides-Extra: markdown
Requires-Dist: html2md-clean>=0.5.0; extra == "markdown"
Requires-Dist: markdown-it-py>=4.0.0; extra == "markdown"
Provides-Extra: sitemap-fast
Requires-Dist: lxml>=5.1.0; extra == "sitemap-fast"
Provides-Extra: articles
Requires-Dist: newspaper3k>=0.2.8; extra == "articles"
Requires-Dist: goose3>=3.1.0; extra == "articles"
Requires-Dist: readability-lxml>=0.8.0; extra == "articles"
Provides-Extra: viz
Requires-Dist: pyvis>=0.3.0; extra == "viz"
Requires-Dist: networkx>=3.6; extra == "viz"
Provides-Extra: celery
Requires-Dist: celery>=5.5.0; extra == "celery"
Requires-Dist: redis>=5.2.0; extra == "celery"
Provides-Extra: performance
Requires-Dist: aiodns>=3.6.0; extra == "performance"
Requires-Dist: uvloop>=0.22.0; platform_system != "Windows" and extra == "performance"
Provides-Extra: fuzzy
Requires-Dist: rapidfuzz>=3.6.0; extra == "fuzzy"
Provides-Extra: dev
Requires-Dist: pytest>=9.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
Requires-Dist: pytest-cov>=7.0.0; extra == "dev"
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: ruff>=0.8.0; extra == "dev"
Requires-Dist: mypy>=1.13.0; extra == "dev"
Requires-Dist: psutil; extra == "dev"
Requires-Dist: rapidfuzz>=3.6.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.3.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.5.0; extra == "dev"
Requires-Dist: build>=1.2.0; extra == "dev"
Requires-Dist: twine>=5.0.0; extra == "dev"
Requires-Dist: playwright>=1.58.0; extra == "dev"
Requires-Dist: curl_cffi>=0.7; extra == "dev"
Requires-Dist: html2md-clean>=0.5.0; extra == "dev"
Requires-Dist: markdown-it-py>=4.0.0; extra == "dev"
Requires-Dist: tiktoken>=0.13.0; extra == "dev"
Requires-Dist: stop-words>=2018.7.23; extra == "dev"
Requires-Dist: fastapi>=0.135.0; extra == "dev"
Requires-Dist: httpx>=0.28; extra == "dev"
Provides-Extra: system
Requires-Dist: psutil>=5.9.0; extra == "system"
Provides-Extra: all
Requires-Dist: graph-crawler[antidetect,articles,celery,fuzzy,markdown,mongodb,performance,postgresql,server,sitemap-fast,system,text,viz]; extra == "all"
Dynamic: license-file

# graph-crawler

Web crawler that builds a graph of a site: pages are nodes, links are edges.
You get a `Graph` object you can query, export, or feed into another tool.

Fetching goes through a chain of drivers with escalation: plain HTTP first,
then TLS-fingerprint HTTP, then a real browser, then an "undetected" browser.
The browser step accepts any engine (CloakBrowser, patchright, stock
Playwright, or your own binary).

## Install

```bash
pip install graph-crawler                 # HTTP only (aiohttp)
pip install "graph-crawler[browser]"      # + browser (stock Playwright)
pip install "graph-crawler[antidetect]"   # + TLS, patchright, CloakBrowser, nodriver
```

| extra                                                                                         | what it adds                                                       |
| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `browser`                                                                                     | `driver="playwright"` with bundled Chromium (`playwright` = alias) |
| `tls`                                                                                         | `curl_cffi` — TLS fingerprint tier (`driver="tls"`)                |
| `stealth`                                                                                     | `patchright` + `playwright-stealth` — engine `"patchright"`        |
| `cloak`                                                                                       | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)   |
| `undetected`                                                                                  | `nodriver` — last tier (`driver="nodriver"`)                       |
| `antidetect`                                                                                  | all four above                                                     |
| `server`                                                                                      | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)  |
| `text`                                                                                        | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`     |
| `markdown`                                                                                    | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`              |
| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `fuzzy`, `performance`, `system`, `all` | storage backends, distributed mode, extras                         |

After installing a browser extra, run once:

```bash
playwright install chromium            # stock Playwright / patchright
python -m cloakbrowser install         # CloakBrowser binary (~200 MB)
# or point to an existing binary: export CLOAKBROWSER_BINARY=/opt/cloak/chrome
```

Python 3.11+.

## Quickstart

```python
import graph_crawler as gc

graph = gc.crawl("https://example.com", max_depth=2, max_pages=50)

print(len(graph.nodes), "pages")
for node in graph.nodes.values():
    print(node.depth, node.response_status, node.url)

hits = graph.search("pricing")
tree = graph.url_tree()
graph.export_nodes("nodes.json")
graph.export_edges("edges.json")
```

Async:

```python
graph = await gc.async_crawl("https://example.com", driver="escalating")
```

Reuse one crawler for many runs (keeps the driver and storage open):

```python
with gc.Crawler(driver="escalating", storage="sqlite", storage_config={"path": "site.db"}) as c:
    g1 = c.crawl("https://a.example")
    g2 = c.crawl("https://b.example")
```

Only some pages, with rules:

```python
from graph_crawler import URLRule

graph = gc.crawl(
    "https://shop.example",
    url_rules=[URLRule(pattern=r"/product/", should_scan=True),
               URLRule(pattern=r"/cart", should_scan=False)],
)
```

Sitemap first:

```python
graph = gc.crawl_sitemap("https://example.com/sitemap.xml", max_urls=500)
```

CLI:

```bash
graph-crawler crawl https://example.com --max-depth 2 --driver escalating --save example
```

## Drivers

`driver=` accepts a name or a driver instance.

| name                | tier        | needs                 | notes                                           |
| ------------------- | ----------- | --------------------- | ----------------------------------------------- |
| `http` / `async`    | http        | —                     | aiohttp, default                                |
| `tls` / `curl_cffi` | tls         | `[tls]`               | Chrome TLS fingerprint, no JS                   |
| `playwright`        | browser     | `[playwright]`        | real browser; engine chosen by `browser_engine` |
| `nodriver`          | undetected  | `[undetected]`        | CDP without webdriver, async                    |
| `escalating`        | all of them | whatever is installed | http → tls → browser → undetected               |

`escalating` builds tiers lazily: a browser is only started when a cheaper
tier is blocked. Tiers whose dependency is missing are skipped.

## Anti-detect: escalation and browser engines

### How escalation decides

Every response goes through `classify_block()`:

| result       | when                                                                         |
| ------------ | ---------------------------------------------------------------------------- |
| `none`       | normal page                                                                  |
| `soft_block` | 429, plain 503, or 200 with an empty body                                    |
| `challenge`  | Cloudflare "Just a moment", `cf-mitigated: challenge`, Incapsula, DDoS-Guard |
| `hard_block` | 403 "Access Denied", Akamai `_abck`, DataDome headers                        |
| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |

Anything other than `none` (and network errors) moves to the next tier.

The tier that worked is remembered per host, and the memory is adaptive:

- it expires after `GC_TIER_TTL` seconds (default 1200 — about the lifetime of
  a `cf_clearance` cookie); after that the host is probed from the cheap tier
  again, so you do not stay on a browser once the protection is gone;
- a block on the remembered tier drops the entry and escalates from the next tier;
- only one request per host escalates at a time, the others wait for its result
  (no herd of browsers against one challenge);
- the memory is per process. Workers do not share it.

Each response carries what happened in `FetchResponse.extras`:

```python
resp.extras["bot_block_status"]  # "none" | "soft_block" | "challenge" | "hard_block" | "captcha"
resp.extras["channel"]           # "http" | "tls" | "browser" | "undetected"
resp.extras["driver"]            # "aiohttp" | "tls" | "playwright" | "nodriver"
resp.extras["engine"]            # "cloakbrowser" | "patchright" | "playwright" | None
resp.extras["escalations"]       # how many tiers were tried before this one
```

### Using it

```python
import graph_crawler as gc

# default chain, browser tier uses CloakBrowser if its binary is present
graph = gc.crawl("https://protected.example", driver="escalating")

# pick the browser engine, stop after the browser tier (0=http 1=tls 2=browser 3=undetected)
graph = gc.crawl(
    "https://protected.example",
    driver="escalating",
    driver_config={"browser_engine": "patchright", "max_tier": 2},
)
```

Directly as a driver:

```python
from graph_crawler.infrastructure.transport.escalation import EscalatingDriver, EscalationPolicy

async with EscalatingDriver(policy=EscalationPolicy.default(browser_engine="cloakbrowser")) as d:
    resp = await d.fetch("https://protected.example")
    print(resp.status_code, resp.extras["channel"], resp.extras["engine"])
```

Your own chain:

```python
from graph_crawler.infrastructure.transport import DriverType
from graph_crawler.infrastructure.transport.escalation import EscalationPolicy, EscalationTier

policy = EscalationPolicy(
    tiers=[
        EscalationTier(channel="tls", driver=DriverType.TLS, config={"impersonate": "chrome"}),
        EscalationTier(channel="browser", driver=DriverType.PLAYWRIGHT, engine="cloakbrowser"),
    ],
    remember_per_host=False,
)
```

### Browser engines

The browser tier is stock `PlaywrightDriver`; the _engine_ decides which
Playwright module and which Chromium binary it launches.

| engine         | what it is                                                    | falls back to |
| -------------- | ------------------------------------------------------------- | ------------- |
| `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
| `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
| `playwright`   | stock                                                         | —             |

Fallback is automatic (one `INFO` log line with the reason). A silent downgrade
to a weaker stealth is dangerous for anti-detect profiles, so there is a strict
mode: `GC_ENGINE_STRICT=1` (or `{"browser_engine": {"strict": True}}`,
`driver_config={"engine_strict": True}` for `escalating`) raises `ImportError`
instead of falling back. Choose the engine:

```python
gc.crawl(url, driver="playwright", driver_config={"browser_engine": {"type": "cloakbrowser"}})
```

or via env `GC_BROWSER_ENGINE=cloakbrowser` (applies when no engine is set in config).

Plug in your own binary or Playwright fork without touching the library:

```python
from graph_crawler.infrastructure.transport.playwright.engine import register_engine

register_engine(
    "my-chrome",
    executable_resolver=lambda: "/opt/my-chrome/chrome",   # None → fall back
    launch_hook=lambda args: ([*args, "--my-flag"], {"ignore_default_args": ["--enable-automation"]}),
    fallback="playwright",
)

gc.crawl(url, driver="escalating", driver_config={"browser_engine": "my-chrome"})
```

`playwright_module="patchright.async_api"` can be passed to `register_engine`
if your engine needs a different Playwright implementation.

Stealth lives in the engine (CloakBrowser / patchright). The library only
decides when to escalate and which engine to launch.

## Storage

`storage="memory"` (default), `"json"`, `"sqlite"`, `"postgresql"`, `"mongodb"`.
Large crawls: use `sqlite` or a server backend — nodes are evicted from RAM
and loaded back on access.

## Tests

```bash
pip install -e ".[browser,tls,dev]"
pytest                       # offline, no browser needed
pytest -m manual tests/manual   # real sites, network required
```
