Metadata-Version: 2.4
Name: scrapingpros
Version: 0.7.5
Summary: Python SDK for the Scraping Pros API — web scraping with browser rendering, proxy rotation, and structured data extraction.
Project-URL: Homepage, https://gitlab.com/7Puentes/scrapingpros-python-sdk
Project-URL: Documentation, https://api.scrapingpros.com/llms-full.txt
Project-URL: Repository, https://gitlab.com/7Puentes/scrapingpros-python-sdk
Project-URL: Issues, https://gitlab.com/7Puentes/scrapingpros-python-sdk/-/issues
Author-email: 7Puentes <dev@7puentes.com>
License-Expression: MIT
License-File: LICENSE
Keywords: api,browser-rendering,data-extraction,proxy,scraping,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.25.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-httpx>=0.30.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# scrapingpros

**Production web scraping at scale, without the plumbing.** Submit a list of URLs, stream results as the workers finish them, and let the server handle the hard parts — browsers, proxies, retries, anti-bot, soft-block detection. Submit 50,000 URLs in one call, walk away, come back to handled results.

[![PyPI](https://img.shields.io/pypi/v/scrapingpros.svg)](https://pypi.org/project/scrapingpros/) [![Python](https://img.shields.io/pypi/pyversions/scrapingpros.svg)](https://pypi.org/project/scrapingpros/) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)

📚 **[Full docs](https://docs.scrapingpros.com/docs/category/python-sdk)** · 📦 **[PyPI](https://pypi.org/project/scrapingpros/)** · 📝 **[Release notes](https://docs.scrapingpros.com/docs/documentation/Python%20SDK/release-notes)** · 🔧 **[API status](https://api.scrapingpros.com)**

---

## Install

```bash
pip install scrapingpros
```

Requires Python 3.10+. No signup needed — the demo token below gives you 5,000 credits/month, 30 req/min, all features enabled.

## The headline use case — `submit_batch()` + `iter_results()`

Streaming async batches are the centerpiece of the SDK. They scale from a handful of URLs up to tens of thousands, with progress, automatic resume, soft-block detection, and per-job traceability via `custom_id`. Benchmarked at N=1000 with `browser=True`, this is **5× faster** than the now-removed `scrape_many` (185 s vs 930 s) and more reliable (998 / 1000 vs 990 / 1000).

```python
import asyncio
from scrapingpros import AsyncClient

async def main():
    async with AsyncClient("demo_6x595maoA6GdOdVb") as client:
        # Submit 50,000 URLs in one call — the API queues them across the worker pool.
        batch = await client.submit_batch("daily-products", [
            {"url": product_url, "custom_id": product_id}
            for product_id, product_url in catalog.items()
        ])

        # Stream results as workers finish them. Progress, ETA, and counters are live.
        async for result in batch.iter_results():
            if result.guidance.success:
                save(product_id=result.custom_id, content=result.content)
            else:
                log_failure(result.custom_id, result.guidance.error_type, result.guidance.next_steps)

            # Live progress — no extra API calls
            print(f"{batch.pct:.1%}  ({batch.success_count}/{batch.total})  ETA {batch.eta_seconds}s")

asyncio.run(main())
```

What you get out of the box:

- **Streaming**, not "submit and poll". Results come back the moment each worker finishes — your loop body runs in real time.
- **`custom_id`** round-trips through every layer (request → job → result → webhook). Map results back to your domain objects without depending on order.
- **`result.guidance.success`** is the *server's* verdict on whether the page produced usable content. It catches soft-blocks (Google CAPTCHA pages with 200 + large body, Amazon "Robot Check") that a naive `status_code` check would miss.
- **Automatic refunds.** Credits are refunded on the spot for any response that doesn't deliver usable content (4xx/5xx, captchas, worker failures, timeouts). You pay for *successful* content only.
- **Per-job retries** with proxy rotation and IP/fingerprint changes. `retry_on_block=True` handles anti-bot sites without you writing retry logic.
- **Worker-restart resilient.** The SDK re-attaches transparently to in-flight runs across transient `ConnectionError` and worker churn.

## Common patterns

### Downloading files (PDFs, images, binaries) — v0.6.0+

```python
resp = client.scrape("https://investors.example.com/charter.pdf", browser=False)
if resp.is_binary:
    resp.save("charter.pdf")              # one-liner: writes to disk
    # or: data = resp.body                 # bytes, mirrors requests.Response.content
else:
    print(resp.content)                   # text response (markdown / html)
```

`resp.content_type` carries the MIME (e.g. `"application/pdf"`, `"image/png"`). Works inside `submit_batch` / `batch_scrape` too — every yielded `ScrapeResponse` has the same accessors.

### Browser-rendered + anti-bot protected sites

```python
result = client.scrape(
    "https://spa-site.com",
    browser=True,            # 5 credits, full JS rendering
    retry_on_block=True,     # auto-retry up to 3x with different IP/fingerprint
)
```

The API picks the right engine internally per target — you don't configure browser engines.

### Structured data extraction (CSS / XPath, no parsing on your side)

```python
result = client.scrape("https://quotes.toscrape.com/", extract={
    "quotes":  {"selector": "css:.text",   "multiple": True},
    "authors": {"selector": "css:.author", "multiple": True},
})
print(result.extracted_data["quotes"])
```

### Form-encoded POST (OAuth2, legacy APIs)

```python
from scrapingpros import MethodPOST

resp = client.scrape(token_url, http_method=MethodPOST(
    payload={"grant_type": "client_credentials"},
    content_type="form",   # since v0.5.0; default is "json"
))
```

### Wait for hidden DOM nodes (`<script>` tags with embedded JSON)

```python
from scrapingpros import WaitForSelectorAction

result = client.scrape(url, browser=True, actions=[
    WaitForSelectorAction(selector="css:script#__NEXT_DATA__", time=8000, state="attached"),
])
```

### Capture response bodies (auth tokens, GraphQL payloads)

```python
from scrapingpros import NetworkCaptureConfig

result = client.scrape(url, browser=True, network_capture=NetworkCaptureConfig(
    resource_types=["xhr", "fetch"],
    url_pattern="*identitytoolkit.googleapis.com*",   # captures matching response bodies (≤64 KB)
))
for entry in result.network_requests or []:
    if "body" in entry:
        token = parse_token(entry["body"])
```

### List return instead of streaming — `batch_scrape()`

Same server-side scaling as `submit_batch()`, simpler signature. Drop-in replacement for `scrape_many()` (removed in v0.7.0):

```python
results = client.batch_scrape([
    {"url": u, "custom_id": product_id, "browser": True}
    for product_id, u in catalog.items()
])
for r in results:
    if r.guidance.success:
        save(r.custom_id, r.content)
```

### Crash-resilient pipelines

For long-running scrapers that can't afford to lose track of a batch:

```python
from scrapingpros import SyncClient, SubmitTimeout

client = SyncClient("...")

# Since v0.5.3, submit_batch automatically generates an Idempotency-Key
# UUID per call, so retrying a SubmitTimeout returns the SAME collection
# (server dedupes within 24h — no duplicate run, no double cost).
try:
    batch = client.submit_batch(name, items, submit_timeout=30.0)
except SubmitTimeout:
    batch = client.submit_batch(name, items)   # safe — server replays
```

For belt-and-braces, use `find_recent_batch` to verify before retrying. It uses server-side `?name=&since=` filters and reattaches to the live run automatically:

```python
import uuid
from datetime import datetime, timezone

batch_name = f"daily-{uuid.uuid4().hex[:8]}"
fired_at = datetime.now(timezone.utc)

try:
    batch = client.submit_batch(batch_name, items)
except SubmitTimeout:
    batch = client.find_recent_batch(name=batch_name, since=fired_at)
    if batch is None:
        batch = client.submit_batch(batch_name, items)

# Reattaching from a persisted (cid, rid) on a separate process:
for r in client.iter_results(saved_cid, saved_rid):
    save(r.custom_id, r.content)
```

If a `get_job_result` raises one of the typed 404 subclasses (`JobResultPending`, `JobResultExpired`, `JobResultLost`, `JobNotFound`), each tells you exactly what to do next — no string parsing required. See [Batch API docs](https://docs.scrapingpros.com/docs/documentation/Python%20SDK/batch-api).

## Pricing — pay only for usable content

| Plan | Price | Credits/mo | Rate | Concurrent |
|---|---|---|---|---|
| Demo (public) | Free | 5,000 | 30/min | 5 |
| Free | $0 | 1,000 | 30/min | 5 |
| Starter | $29 | 25,000 | 30/min | 10 |
| Growth | $69 | 100,000 | 60/min | 20 |
| Pro | $199 | 500,000 | 120/min | 50 |
| Scale | $499 | 2,500,000 | 200/min | 100 |
| Enterprise | Custom | Unlimited | 2,000/min | Custom |

1 simple request = 1 credit, 1 browser request = 5 credits. Credits are refunded automatically on failures (`is_success=false`). Anti-bot, proxy rotation, and per-country routing are included on every plan.

Check your usage at any time:

```python
client.scrape(url)
print(client.quota_remaining, "credits left")
print(client.billing())
```

## Error handling

```python
from scrapingpros import SyncClient, AuthenticationError, RateLimitError, QuotaExceededError

try:
    result = client.scrape(url)
except AuthenticationError:
    print("Invalid token — use demo_6x595maoA6GdOdVb for testing")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s")
except QuotaExceededError:
    print("Monthly quota exceeded — upgrade your plan for more requests")
```

All SDK exceptions inherit from `ScrapingProsError`.

## Configuration

```python
client = AsyncClient(
    "demo_6x595maoA6GdOdVb",            # or your dedicated token / SP_TOKEN env var
    base_url="https://api.scrapingpros.com",   # default
    timeout=120.0,                       # request timeout in seconds
    max_retries=3,                       # auto-retry on 429
)
```

> A blocking `SyncClient` exists with the same surface for REPL sessions, notebooks, and one-off scripts that need a single result. Inside a running event loop it emits a `RuntimeWarning` — use `AsyncClient` there.

## More

- **[Full documentation](https://docs.scrapingpros.com/docs/category/python-sdk)** with end-to-end recipes for batches, JS execution, viability tests, and anti-bot strategies.
- **[Release notes](https://docs.scrapingpros.com/docs/documentation/Python%20SDK/release-notes)** for every version.
- **[API reference for AI agents](https://api.scrapingpros.com/llms-full.txt)** — single-file `llms-full.txt` for Claude / GPT / Cursor.
- **MCP server** for AI agents: [Model Context Protocol](https://modelcontextprotocol.io) integration with 6 tools and anti-injection protection.

## License

MIT
