Metadata-Version: 2.4
Name: hermes-zyte-scraper
Version: 1.3.0
Summary: Production-grade Zyte + Scrapy Cloud integration for Hermes agents
Author-email: DeepSpace <deepspace28@users.noreply.github.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/deepspace28/hermes-zyte-scraper
Project-URL: Issues, https://github.com/deepspace28/hermes-zyte-scraper/issues
Project-URL: Repository, https://github.com/deepspace28/hermes-zyte-scraper
Keywords: hermes,hermes-plugin,zyte,scrapy,web-scraping,scrapy-cloud,ai-agent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: zyte-api>=0.4
Requires-Dist: scrapy>=2.11
Requires-Dist: scrapy-zyte-api>=0.8
Requires-Dist: requests>=2.28
Requires-Dist: shub>=2.12
Requires-Dist: tenacity>=8.5.0
Dynamic: license-file

# hermes-zyte-scraper

Production-grade Zyte + Scrapy Cloud integration for Hermes agents (v1.3.0).

Describe a scraping task in natural language, extract data via Zyte API, generate production Scrapy spiders, deploy to Scrapy Cloud, schedule runs, and retrieve stored results.

## Installation

### Method 1: pip install (recommended)

```bash
pip install hermes-zyte-scraper
```

Hermes will auto-discover the plugin via entry points. No need to manually enable.

### Method 2: Manual install

```bash
git clone https://github.com/deepspace28/hermes-zyte-scraper.git ~/.hermes/plugins/hermes-zyte-scraper
cd ~/.hermes/plugins/hermes-zyte-scraper
pip install -e .
```

Set credentials in `~/.hermes/.env`:

| Variable | Purpose |
|----------|---------|
| `ZYTE_API_KEY` | Zyte API extraction and spider runs |
| `SCRAPY_CLOUD_API_KEY` | Deploy, schedule, list jobs, fetch results |
| `SCRAPY_CLOUD_PROJECT_ID` | Numeric Scrapy Cloud project ID (e.g. `867424`) |

**Scrapy Cloud:** Also set `ZYTE_API_KEY` in your project's environment variables in the [Zyte dashboard](https://app.zyte.com).

## Tools

| Tool | When to use |
|------|-------------|
| `zyte_extract` | Quick one-off or moderate multi-page extraction (auto schema inference) |
| `zyte_build_spider` | Generate a full Scrapy + Zyte project (requires `start_url`) |
| `zyte_deploy` | Deploy generated project to Scrapy Cloud |
| `zyte_schedule` | Run spider once or on a cron schedule (uses `run.json` API) |
| `zyte_list_jobs` | Monitor Scrapy Cloud jobs |
| `zyte_get_results` | Pull items from Scrapy Cloud storage |

Load the workflow skill in Hermes: `skill_view("hermes-zyte-scraper:zyte-scraping")`

## zyte_extract highlights (v1.3.0)

- **Auto schema inference** — Indeed → `jobPostingNavigation`, Zillow/Amazon → `productList`
- **Multi-page pagination** — browserHtml + robust next-page detection
- **Sessions** — optional `session_id` for cookie/IP reuse
- **Custom attributes** — `custom_attributes="address,price,beds,baths"` per Zyte docs

## End-to-end example

```
# 1. Quick extraction (schema inferred from URL)
zyte_extract url="https://books.toscrape.com/" max_pages=5

# 2. Build a production spider
zyte_build_spider \
  description="Scrape book titles, prices, and URLs with pagination" \
  start_url="https://books.toscrape.com/" \
  spider_name="books-demo"

# 3. Deploy to Scrapy Cloud
zyte_deploy project_path="~/.hermes/spiders/books-demo"

# 4. Schedule a one-time run
zyte_schedule project_id="867424" spider="books-demo"

# 5. Monitor and fetch results
zyte_list_jobs project_id="867424"
zyte_get_results job_id="867424/books-demo/1"
```

Use `dry_run=true` on operational tools to validate without API calls.

## Battle testing

Recorded minimums: `tests/fixtures/battle_matrix_expected.json`

Run live matrix locally (requires `ZYTE_API_KEY`):

```bash
python scripts/run_battle_matrix.py
```

## Development

```bash
pip install -r requirements.txt pytest
python -m pytest tests/ -v
```

## Cost notes

Zyte charges per successful API response. Browser rendering, sessions, custom attributes, and large multi-page crawls increase cost. Set spending limits in the Zyte dashboard.
