Metadata-Version: 2.4
Name: hyae-toolkit
Version: 0.1.2
Summary: High Yield Asset Engine: polite Python web scrapers and Claude-powered automation templates
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: pyyaml>=6.0
Requires-Dist: pydantic>=2.5
Requires-Dist: anthropic>=1.0
Provides-Extra: webhook
Requires-Dist: flask>=3.0; extra == "webhook"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: flask>=3.0; extra == "dev"

# hyae-toolkit

Polite Python web scrapers and Claude-powered automation templates, driven by small YAML configs.

- **Scrapers** respect `robots.txt`, rate-limit per host, retry transient failures with backoff, and refuse non-HTTP(S) URLs.
- **AI commands** use Claude to extract structured records, write digests, and repurpose articles. Scraped text is passed to the model as untrusted data.

## Install

```bash
pip install hyae-toolkit
```

Requires Python 3.10+.

## Commands

| Command | What it does | Needs Claude credentials |
|---|---|---|
| `hyae scrape CONFIG [-o out.csv]` | CSS-selector list scraper with pagination | No |
| `hyae prices CONFIG [-o out.csv]` | Track product prices and keep a history | No |
| `hyae rss CONFIG [-o out.csv] [--digest out.md]` | Collect new RSS/Atom entries; optionally write an AI digest | Only with `--digest` |
| `hyae extract URL [--schema product\|job\|event]` | AI-extract structured records from a page | Yes |
| `hyae repurpose URL_OR_FILE [--voice ...]` | AI content pack from an article | Yes |

Output format follows the file extension: `.csv` or `.json`. CSV files are written as UTF-8 with a BOM so Excel opens them correctly.

Global options: `-v/--verbose`, `--model` (default `claude-opus-5-5`), `--effort` (`low`, `medium`, `high`, `xhigh`, `max`).

## Example: scrape a listing

Save as `books.yaml`:

```yaml
fetch:
  user_agent: "HYAE-Toolkit/0.1 (you@example.com)"
  min_delay: 1.0
start_url: https://books.toscrape.com/
item_selector: article.product_pod
fields:
  title: {selector: "h3 a", attr: title}
  price: {selector: ".price_color"}
  url: {selector: "h3 a", attr: href, absolute: true}
next_selector: li.next a
max_pages: 3
```

```bash
hyae scrape books.yaml -o output/books.csv
```

A field is either a selector string (element text) or a mapping with `selector` plus optional `attr`, `absolute: true` (resolve relative URLs) and `many: true` (collect all matches).

`fetch` options: `user_agent`, `min_delay` (seconds between requests to one host), `timeout`, `max_retries`, `respect_robots`, `headers`. Set `user_agent` to something that identifies you.

More example configs (price watch, RSS digest) are in `config/examples/` in the source distribution.

## Claude credentials

AI commands use the Anthropic SDK's standard credential lookup: `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, or an `ant auth login` profile. Never put a key in a config file.

## PayPal webhook: sell and deliver a file

`hyae.webhook` is a Flask listener for PayPal IPN (Instant Payment Notification). When someone pays, it emails them your product file as an attachment.

```bash
pip install "hyae-toolkit[webhook]"
```

For each notification it:

1. Posts it back to PayPal to verify it. If PayPal can't be reached it returns 503, so PayPal retries later.
2. Delivers only if `payment_status` is `Completed`, the payment went to your PayPal address, and the amount and currency match exactly. Sandbox notifications are refused unless sandbox mode is on.
3. Emails the file to the buyer once per `txn_id`, then appends the payment to a JSON-lines log. If sending fails, nothing is logged and it returns 500, so PayPal's retry tries again.

Configure it with environment variables. They are checked at startup.

| Variable | Required | Meaning |
|---|---|---|
| `PAYPAL_RECEIVER_EMAIL` | Yes | Your PayPal account email; payments to anyone else are ignored |
| `HYAE_PRICE` | Yes | Exact price, e.g. `500.00` |
| `HYAE_CURRENCY` | Yes | 3-letter code, e.g. `GBP` |
| `HYAE_PRODUCT_PATH` | Yes | File to email to buyers |
| `SMTP_HOST` | Yes | Mail server |
| `SMTP_FROM` | Yes | Sender address |
| `SMTP_PORT` | No | `587` (STARTTLS, default) or `465` (implicit TLS) |
| `SMTP_USER`, `SMTP_PASSWORD` | No | SMTP login |
| `PAYPAL_SANDBOX` | No | `true` to verify against the PayPal sandbox |
| `HYAE_PAYMENTS_LOG` | No | Payment log path (default `data/payments.jsonl`) |

Run it behind HTTPS with a production WSGI server, as a **single process** (the duplicate-delivery guard is per process):

```bash
pip install waitress
waitress-serve --call hyae.webhook:create_app
```

Then set your PayPal IPN URL to `https://<your-host>/ipn`. Test end to end with `PAYPAL_SANDBOX=true` and PayPal's IPN simulator before taking real payments.

## Scrape responsibly

Only scrape sites you are allowed to, keep `respect_robots` on, and use a sensible `min_delay`. books.toscrape.com is a public sandbox built for scraping practice.
