Metadata-Version: 2.4
Name: graph-crawler
Version: 5.3.7
Summary: Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
Author: 0-EternalJunior-0
Maintainer: 0-EternalJunior-0
License-Expression: MIT
Project-URL: Homepage, https://gitlab.com/demoprogrammer/web_graf
Project-URL: Documentation, https://gitlab.com/demoprogrammer/web_graf/-/blob/main/web_graf/README.md
Project-URL: Repository, https://gitlab.com/demoprogrammer/web_graf
Project-URL: Changelog, https://gitlab.com/demoprogrammer/web_graf/-/blob/main/web_graf/CHANGELOG.md
Project-URL: Bug Tracker, https://gitlab.com/demoprogrammer/web_graf/-/issues
Keywords: web,crawler,scraper,graph,spider,playwright,anti-bot,tls-fingerprint
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp>=3.13.0
Requires-Dist: backports.zstd>=1.6.0; python_version < "3.14"
Requires-Dist: Brotli
Requires-Dist: selectolax>=0.4.0
Requires-Dist: defusedxml>=0.7.0
Requires-Dist: pydantic>=2.12.0
Requires-Dist: orjson>=3.11.0
Requires-Dist: aiofiles>=24.1.0
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: typing_extensions>=4.5; python_version < "3.12"
Requires-Dist: xxhash>=3.0.0
Provides-Extra: native
Requires-Dist: cython>=3.0.0; extra == "native"
Requires-Dist: mmh3>=5.0.0; extra == "native"
Provides-Extra: browser
Requires-Dist: playwright>=1.58.0; extra == "browser"
Provides-Extra: playwright
Requires-Dist: graph-crawler[browser]; extra == "playwright"
Provides-Extra: tls
Requires-Dist: curl_cffi>=0.7; extra == "tls"
Provides-Extra: stealth
Requires-Dist: patchright>=1.52; extra == "stealth"
Requires-Dist: playwright-stealth>=1.0; extra == "stealth"
Provides-Extra: cloak
Requires-Dist: cloakbrowser>=0.5; extra == "cloak"
Requires-Dist: playwright>=1.58.0; extra == "cloak"
Provides-Extra: undetected
Requires-Dist: nodriver>=0.38; extra == "undetected"
Provides-Extra: antidetect
Requires-Dist: graph-crawler[browser,cloak,stealth,tls,undetected]; extra == "antidetect"
Provides-Extra: server
Requires-Dist: fastapi>=0.135.0; extra == "server"
Requires-Dist: uvicorn>=0.30; extra == "server"
Requires-Dist: websockets>=13; extra == "server"
Provides-Extra: text
Requires-Dist: tiktoken>=0.13.0; extra == "text"
Requires-Dist: stop-words>=2018.7.23; extra == "text"
Provides-Extra: captcha
Provides-Extra: mongodb
Requires-Dist: pymongo[srv]>=4.9.0; extra == "mongodb"
Provides-Extra: postgresql
Requires-Dist: asyncpg>=0.30.0; extra == "postgresql"
Provides-Extra: newspaper
Requires-Dist: newspaper3k>=0.2.8; extra == "newspaper"
Provides-Extra: goose
Requires-Dist: goose3>=3.1.0; extra == "goose"
Provides-Extra: readability
Requires-Dist: readability-lxml>=0.8.0; extra == "readability"
Provides-Extra: markdown
Requires-Dist: html2md-clean>=0.5.0; extra == "markdown"
Requires-Dist: markdown-it-py>=4.0.0; extra == "markdown"
Provides-Extra: sitemap-fast
Requires-Dist: lxml>=5.1.0; extra == "sitemap-fast"
Provides-Extra: articles
Requires-Dist: newspaper3k>=0.2.8; extra == "articles"
Requires-Dist: goose3>=3.1.0; extra == "articles"
Requires-Dist: readability-lxml>=0.8.0; extra == "articles"
Provides-Extra: viz
Requires-Dist: pyvis>=0.3.0; extra == "viz"
Requires-Dist: networkx>=3.6; extra == "viz"
Provides-Extra: celery
Requires-Dist: celery>=5.5.0; extra == "celery"
Requires-Dist: redis>=5.2.0; extra == "celery"
Provides-Extra: performance
Requires-Dist: aiodns>=3.6.0; extra == "performance"
Requires-Dist: uvloop>=0.22.0; platform_system != "Windows" and extra == "performance"
Provides-Extra: fuzzy
Requires-Dist: rapidfuzz>=3.6.0; extra == "fuzzy"
Provides-Extra: dev
Requires-Dist: pytest>=9.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
Requires-Dist: pytest-cov>=7.0.0; extra == "dev"
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: ruff>=0.8.0; extra == "dev"
Requires-Dist: mypy>=1.13.0; extra == "dev"
Requires-Dist: psutil; extra == "dev"
Requires-Dist: rapidfuzz>=3.6.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.3.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.5.0; extra == "dev"
Requires-Dist: build>=1.2.0; extra == "dev"
Requires-Dist: twine>=5.0.0; extra == "dev"
Requires-Dist: playwright>=1.58.0; extra == "dev"
Requires-Dist: curl_cffi>=0.7; extra == "dev"
Requires-Dist: html2md-clean>=0.5.0; extra == "dev"
Requires-Dist: markdown-it-py>=4.0.0; extra == "dev"
Requires-Dist: tiktoken>=0.13.0; extra == "dev"
Requires-Dist: stop-words>=2018.7.23; extra == "dev"
Requires-Dist: fastapi>=0.135.0; extra == "dev"
Requires-Dist: httpx>=0.28; extra == "dev"
Provides-Extra: system
Requires-Dist: psutil>=5.9.0; extra == "system"
Provides-Extra: all
Requires-Dist: graph-crawler[antidetect,articles,celery,fuzzy,markdown,mongodb,performance,postgresql,server,sitemap-fast,system,text,viz]; extra == "all"
Dynamic: license-file

# graph-crawler

A web crawler that builds a graph of a site: pages are nodes, links are edges.
The result is a `Graph` object you can query, export, save and load again.

Pages are fetched through a chain of drivers with escalation: plain HTTP first,
then HTTP with a browser TLS fingerprint, then a real browser, then an
"undetected" browser. The browser step can use any engine: CloakBrowser,
patchright, stock Playwright or your own binary.

Python 3.11+.

## Install

```bash
pip install graph-crawler                 # HTTP only (aiohttp)
pip install "graph-crawler[browser]"      # + browser (stock Playwright)
pip install "graph-crawler[antidetect]"   # + TLS, patchright, CloakBrowser, nodriver
```

| extra          | what it adds                                                                    |
| -------------- | ------------------------------------------------------------------------------- |
| `browser`      | `driver="playwright"`, stock Playwright (`playwright` is an alias of this extra) |
| `tls`          | `curl_cffi` — TLS fingerprint tier, `driver="tls"`                              |
| `stealth`      | `patchright` + `playwright-stealth` — engine `"patchright"`                     |
| `cloak`        | `cloakbrowser` — engine `"cloakbrowser"` (stealth in the binary)                |
| `undetected`   | `nodriver` — last tier, `driver="nodriver"`                                     |
| `antidetect`   | the five extras above                                                           |
| `server`       | FastAPI dashboard / REST / WebSocket (`graph_crawler.api.server`)               |
| `text`         | `tiktoken` + `stop-words` — `count_tokens`, `extract_keywords`                  |
| `markdown`     | `html2md-clean` + `markdown-it-py` — `MarkdownPlugin`                           |
| `fuzzy`        | `rapidfuzz` — `graph.search(..., mode="fuzzy")`                                 |
| `sitemap-fast` | `lxml` sitemap parser                                                           |
| `mongodb`, `postgresql`, `celery`, `viz`, `articles`, `performance`, `system`, `all` | storage backends, distributed mode and other extras |

After installing a browser extra, download the browser once:

```bash
playwright install chromium            # stock Playwright / patchright
python -m cloakbrowser install         # CloakBrowser binary (~200 MB)
# or point to an existing binary: export CLOAKBROWSER_BINARY=/opt/cloak/chrome
```

## Quickstart

```python
import graph_crawler as gc

graph = gc.crawl("https://example.com", max_depth=2, max_pages=50)

print(gc.quick_stats(graph))           # "12 nodes (12 scanned) | 31 edges"
for node in graph.nodes.values():
    print(node.depth, node.response_status, node.url)

hits = graph.search("pricing")         # [SearchHit(node, score, matched_fields, snippet), ...]
tree = graph.url_tree()
graph.export_nodes("nodes.json")
graph.export_edges("edges.json")
```

`graph.nodes` also holds pages that were discovered but not fetched (for example,
beyond `max_pages`); `node.scanned` tells them apart.

Async:

```python
graph = await gc.async_crawl("https://example.com", driver="escalating")
```

Reuse one crawler for several runs (the driver and storage stay open):

```python
with gc.Crawler(driver="escalating", storage="sqlite", storage_config={"path": "site.db"}) as c:
    g1 = c.crawl("https://a.example")
    g2 = c.crawl("https://b.example")
```

Crawl only some pages:

```python
from graph_crawler import URLRule

graph = gc.crawl(
    "https://shop.example",
    url_rules=[URLRule(pattern=r"/product/", should_scan=True),
               URLRule(pattern=r"/cart", should_scan=False)],
)
```

Start from a sitemap:

```python
graph = gc.crawl_sitemap("https://example.com/sitemap.xml", max_urls=500)
```

Save and load:

```python
gc.save_graph(graph, "site.json")      # format="json" (default) or "sqlite"
graph = gc.load_graph("site.json")
```

CLI:

```bash
graph-crawler crawl https://example.com --max-depth 2 --driver escalating --save example
graph-crawler list
graph-crawler info example
```

## Inspecting pages

Every node records how it was fetched:

```python
node = graph.get_node_by_url("https://example.com/")

node.ok                 # scanned, 2xx/3xx, not blocked, no error
node.is_blocked         # classified as an anti-bot block
node.channel            # "http" | "tls" | "browser" | "undetected"
node.bot_block_status   # None | "none" | "soft_block" | "challenge" | "hard_block" | "captcha"
node.served_by_driver   # "aiohttp" | "tls" | "playwright" | "nodriver"
node.served_by_engine   # "cloakbrowser" | "patchright" | "playwright" | None
node.escalations        # how many tiers were tried before this one
node.fetch_info         # all of the above as one dict
```

A response with status 400 or higher keeps `metadata["error"] = "HTTP <code>"`,
so `node.ok` is `False` for error pages.

## Drivers

`driver=` accepts a name or a driver instance.

| name                          | tier        | needs                 | notes                                           |
| ----------------------------- | ----------- | --------------------- | ----------------------------------------------- |
| `aiohttp` / `http` / `async`  | http        | —                     | default                                         |
| `tls` / `curl_cffi`           | tls         | `[tls]`               | Chrome TLS fingerprint, no JavaScript           |
| `playwright`                  | browser     | `[browser]`           | real browser; engine chosen by `browser_engine` |
| `nodriver`                    | undetected  | `[undetected]`        | CDP without webdriver                           |
| `escalating`                  | all of them | whatever is installed | http → tls → browser → undetected               |

`escalating` creates tiers lazily: a browser starts only when a cheaper tier is
blocked. Tiers whose dependency is not installed are skipped.

## Anti-detect: escalation and browser engines

### How escalation decides

Every response goes through `classify_block()`:

| result       | when                                                                         |
| ------------ | ---------------------------------------------------------------------------- |
| `none`       | normal page                                                                  |
| `soft_block` | 429, plain 503, or 200 with an empty body                                    |
| `challenge`  | Cloudflare "Just a moment", `cf-mitigated: challenge`, Incapsula, DDoS-Guard |
| `hard_block` | 403/401/405 **with** a vendor or "Access Denied" marker (Akamai `_abck`, DataDome/Incapsula headers, `x-iinfo`, Cloudflare `server`) |
| `captcha`    | PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup        |

Anything other than `none`, and network errors, move the request to the next tier.

A bare 403/401/405 without a vendor marker is classified as `none` (authorization
or method, not a bot block), so `escalating` does not open a browser for it and
logs the decision at `INFO`. If such a page is a block, use `driver="playwright"`.

The tier that worked is remembered per host:

- the entry expires after `GC_TIER_TTL` seconds (default 1200, about the lifetime
  of a `cf_clearance` cookie); after that the host is probed from the cheapest tier again;
- a block on the remembered tier drops the entry and escalates from the next tier;
- only one request per host escalates at a time, the others wait for its result;
- the memory is per process; workers do not share it.

Each `FetchResponse` describes what happened in `extras` (on a node — see
[Inspecting pages](#inspecting-pages)):

```python
resp.extras["bot_block_status"]  # "none" | "soft_block" | "challenge" | "hard_block" | "captcha"
resp.extras["channel"]           # "http" | "tls" | "browser" | "undetected"
resp.extras["driver"]            # "aiohttp" | "tls" | "playwright" | "nodriver"
resp.extras["engine"]            # "cloakbrowser" | "patchright" | "playwright" | None
resp.extras["escalations"]       # how many tiers were tried before this one
```

### Using it

```python
import graph_crawler as gc

# default chain; the browser tier uses CloakBrowser if its binary is present
graph = gc.crawl("https://protected.example", driver="escalating")

# choose the browser engine and stop after the browser tier (0=http 1=tls 2=browser 3=undetected)
graph = gc.crawl(
    "https://protected.example",
    driver="escalating",
    driver_config={"browser_engine": "patchright", "max_tier": 2},
)
```

As a driver:

```python
from graph_crawler.infrastructure.transport.escalation import EscalatingDriver, EscalationPolicy

async with EscalatingDriver(policy=EscalationPolicy.default(browser_engine="cloakbrowser")) as d:
    resp = await d.fetch("https://protected.example")
    print(resp.status_code, resp.extras["channel"], resp.extras["engine"])
```

A custom chain:

```python
from graph_crawler.infrastructure.transport import DriverType
from graph_crawler.infrastructure.transport.escalation import EscalationPolicy, EscalationTier

policy = EscalationPolicy(
    tiers=[
        EscalationTier(channel="tls", driver=DriverType.TLS, config={"impersonate": "chrome"}),
        EscalationTier(channel="browser", driver=DriverType.PLAYWRIGHT, engine="cloakbrowser"),
    ],
    remember_per_host=False,
)
```

### Browser engines

The browser tier is `PlaywrightDriver`; the _engine_ decides which Playwright
module and which Chromium binary it launches.

| engine         | what it is                                                    | falls back to |
| -------------- | ------------------------------------------------------------- | ------------- |
| `cloakbrowser` | Chromium with fingerprint patches compiled into the binary    | `patchright`  |
| `patchright`   | Playwright fork without automation leaks (`Runtime.enable` …) | `playwright`  |
| `playwright`   | stock Playwright                                              | —             |

A fallback is logged once at `INFO` with the reason. To get an `ImportError`
instead of a silent downgrade, set `GC_ENGINE_STRICT=1`, or pass
`{"browser_engine": {"strict": True}}` (`driver_config={"engine_strict": True}`
for `escalating`).

Choose the engine in the config:

```python
gc.crawl(url, driver="playwright", driver_config={"browser_engine": {"type": "cloakbrowser"}})
```

or with `GC_BROWSER_ENGINE=cloakbrowser` (used when the config sets no engine).

Register your own binary or Playwright fork:

```python
from graph_crawler.infrastructure.transport.playwright.engine import register_engine

register_engine(
    "my-chrome",
    executable_resolver=lambda: "/opt/my-chrome/chrome",   # None → fall back
    launch_hook=lambda args: ([*args, "--my-flag"], {"ignore_default_args": ["--enable-automation"]}),
    fallback="playwright",
)

gc.crawl(url, driver="escalating", driver_config={"browser_engine": "my-chrome"})
```

Pass `playwright_module="patchright.async_api"` to `register_engine` if the engine
needs a different Playwright implementation.

Stealth lives in the engine. The library decides when to escalate and which
engine to launch.

### Page actions

The Playwright driver can click, type and wait before the HTML snapshot is taken:

```python
from graph_crawler import PageAction

graph = gc.crawl(
    "https://shop.example/search",
    max_depth=0,
    driver="playwright",
    driver_config={"actions": [
        PageAction(type="fill", placeholder="Search", value="laptop"),
        {"type": "click", "text": "Apply"},
        {"type": "wait_for", "response": "/api/items"},
    ]},
)
node = graph.get_node_by_url("https://shop.example/search")
node.metadata["actions_log"]            # [{"index": 0, "ok": True, ...}, ...]
node.metadata["requests_after_actions"]  # XHR/fetch requests made after the actions
```

Action types: `click`, `fill`, `select`, `press`, `scroll`, `wait_for`, `back`.
A target is found by `selector`, `text`, `placeholder` or `label` (`nth` picks
among visible matches). The first failing action stops the chain and is written
to `node.metadata["actions_error"]`.

## Controlling a running crawl

`async_crawl(..., control_channel=channel)` (and `crawl`, `GraphCrawlerClient.crawl`)
reads commands from the channel between pages:

```python
import graph_crawler as gc
from graph_crawler.domain.interfaces import AsyncQueueControlChannel, force_visit_command, stop_command

channel = AsyncQueueControlChannel()
channel.send(force_visit_command("https://example.com/hidden-page"))
graph = await gc.async_crawl("https://example.com", control_channel=channel)

# from a plugin, a callback or another task while the crawl runs:
channel.send(stop_command("enough data"))   # graph.metadata["stop_reason"] == "enough data"
```

| command        | `data`                  | effect                                         |
| -------------- | ----------------------- | ---------------------------------------------- |
| `STOP`         | `reason` (optional)     | stops the crawl, `stop_reason` = `reason`      |
| `PAUSE`        | —                       | pauses until `RESUME` or `STOP`                |
| `RESUME`       | —                       | continues a paused crawl                       |
| `REPRIORITIZE` | `url`, `priority`       | moves a queued URL in the queue                |
| `FORCE_VISIT`  | `url`                   | adds the URL to the queue with top priority    |

`SPIDER_COMMANDS` is this set. `SKIP_URL`, `CHANGE_STRATEGY`, `SET_DEPTH_LIMIT`,
`REQUEST_STATE` and `UPDATE_GOAL` are declared in `CrawlCommand` but not executed:
the spider logs `WARNING Unsupported control command ...` (once per command type
per crawl) and goes on; they will be removed in the next minor release. A supported
command without its required `data` fields is also skipped with a `WARNING`.

## Plugins and custom nodes

A node plugin runs on every fetched page and writes to `user_data`, `metadata`
or the typed fields of your `Node` subclass:

```python
import graph_crawler as gc
from graph_crawler import BaseNodePlugin, Node, NodePluginType


class ProductNode(Node):
    price: int | None = None


class PriceExtractor(BaseNodePlugin):
    url_pattern = r"/product/"        # run only on matching URLs (or: domains = ["shop.example"])
    expects_nonempty = True           # log a WARNING when nothing was extracted
    extraction_key = "prices"

    @property
    def name(self):
        return "price_extractor"

    @property
    def plugin_type(self):
        return NodePluginType.ON_HTML_PARSED

    def execute(self, ctx):
        el = ctx.html_tree.css_first(".price")
        if el is not None:
            ctx.user_data["prices"] = [el.text()]
            ctx.set_node_field("price", el.text())   # validated against the annotation: "10" → 10
        return ctx


graph = gc.crawl("https://shop.example", node_class=ProductNode, plugins=[PriceExtractor()])
```

- `domains` / `url_pattern` can also be passed per instance:
  `PriceExtractor(config={"domains": ["shop.example"]})`. Several instances of one
  plugin class with different routes run side by side.
- `graph.search("acme", user_data_keys=("prices",))` searches plugin data as well
  as titles and URLs.
- `node.public_user_data` is `user_data` without the reserved internal keys
  (`RESERVED_USER_DATA_KEYS`).
- `gc.load_graph(path)` restores the `Node` subclass when its module is already
  imported; `load_graph(path, node_class=ProductNode)` sets it explicitly.

## Storage

`storage="memory"` (default), `"json"`, `"sqlite"`, `"postgresql"`, `"mongodb"`.
For large crawls use `sqlite` or a server backend: nodes are evicted from RAM
and loaded back on access.

## Environment variables

| variable              | effect                                                        |
|-----------------------|---------------------------------------------------------------|
| `GC_TIER_TTL`         | seconds the working tier is remembered per host (default 1200) |
| `GC_BROWSER_ENGINE`   | default browser engine when the config sets none              |
| `GC_ENGINE_STRICT`    | `1` — raise `ImportError` instead of falling back to another engine |
| `CLOAKBROWSER_BINARY` | path to an existing CloakBrowser binary                       |

## Logging

The library adds a `NullHandler` to the `graph_crawler` logger and never
configures logging itself. At `INFO` it writes one line per fetched page, plus the
start of the crawl and warnings such as `HTTP 404 for <url>`:

```python
import logging
logging.basicConfig(level=logging.INFO)
```
