Metadata-Version: 2.4
Name: sarvcrawl
Version: 1.0.5
Summary: Python SDK for SarvCrawl — crawl sites, parse documents, and run keyword or hybrid semantic search over your knowledge bases
Author: Sarv Webs Pvt. Ltd.
License: Apache-2.0
Project-URL: Homepage, https://crawl.sarv.com
Project-URL: Documentation, https://crawl.sarv.com/docs/python-sdk
Project-URL: Changelog, https://crawl.sarv.com/changelog
Keywords: sarvcrawl,knowledge-base,rag,scrape,crawl,semantic-search,vector-search,sdk
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.32.4
Requires-Dist: aiohttp>=3.14.3
Requires-Dist: pydantic>=2.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
Requires-Dist: pytest-mock>=3.10; extra == "dev"
Requires-Dist: responses>=0.23; extra == "dev"
Requires-Dist: aioresponses>=0.7; extra == "dev"
Requires-Dist: python-dotenv>=1.0; extra == "dev"
Dynamic: license-file

<div align="center">

# sarvcrawl

**Turn any website or document pile into a searchable knowledge base — from Python.**

The official Python SDK for [SarvCrawl](https://crawl.sarv.com). Crawl sites,
parse PDFs and Office files to clean Markdown, then query them over hybrid
keyword + vector search. Sync and async clients, fully typed.

[![PyPI](https://img.shields.io/pypi/v/sarvcrawl.svg?logo=pypi&logoColor=white)](https://pypi.org/project/sarvcrawl/)
[![Python](https://img.shields.io/pypi/pyversions/sarvcrawl.svg?logo=python&logoColor=white)](https://pypi.org/project/sarvcrawl/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)

[Quick start](#-quick-start) · [Async](#-async) · [API](#-api) · [Errors](#-errors) · [Configuration](#-configuration)

</div>

---

## 🚀 Quick start

```bash
pip install sarvcrawl
```

Get an API key from [crawl.sarv.com](https://crawl.sarv.com) — 1 billion free
tokens on sign-up, no card.

```python
import os, time
from sarvcrawl import KBClient

kb = KBClient(api_key=os.environ["KB_API_KEY"])

# 1. Somewhere to put things
knowledge_base = kb.create_kb("Docs", description="Product documentation")

# 2. Fill it — crawl a site, or upload a file
job = kb.crawl(knowledge_base.id, "https://docs.example.com", max_depth=2, limit=25)

# 3. Wait for it, then ask a question
while kb.get_job(job.job_id).status in ("pending", "processing"):
    time.sleep(3)

hits = kb.embed_search_kb(knowledge_base.id, "how does authentication work?")
for hit in hits.hits:
    print(hit.title, "→", hit.url)
```

Ingest is asynchronous: `scrape`, `crawl`, `map_job`, `search_job` and `upload`
return a `JobRef`, and you poll `get_job(job_id)` until it settles. Search is
synchronous.

---

## ⚡ Async

Every method exists on `AsyncKBClient` with the same signature.

```python
import asyncio
from sarvcrawl import AsyncKBClient

async def main():
    # Use it as a context manager — it owns an aiohttp session that must be
    # closed, or Python prints "Unclosed client_session" on exit.
    async with AsyncKBClient(api_key="sarv_sk_…") as kb:
        for k in await kb.list_kbs():
            print(k.id, k.name)

asyncio.run(main())
```

---

## 📚 API

Every method returns a typed pydantic model — `KnowledgeBase`, `Job`,
`SearchResponse`, `PageContent`, `AuditReport` — so your editor completes fields
and a typo fails fast instead of yielding `None`.

### Knowledge bases

| Method | What it does |
|---|---|
| `create_kb(name, description=None, settings=None)` | Create a KB |
| `list_kbs()` | Every KB you own |
| `get_kb(kb_id)` · `get_kb_stats(kb_id)` | One KB, and its counts |
| `delete_kb(kb_id)` | Delete a KB and everything in it — irreversible |

### Ingest

| Method | What it does |
|---|---|
| `scrape(kb_id, url, ...)` | One page |
| `crawl(kb_id, url, max_depth=5, limit=500, exclude_paths=None, ...)` | Follow links from a root |
| `map_job(kb_id, url, limit=1000)` | Discover URLs only — no content, no page cost |
| `search_job(kb_id, query, ...)` | Web search straight into the KB |
| `upload(kb_id, file_path, ocr_language="eng", ...)` | PDF, DOCX, XLSX, PPTX, CSV, EPUB, images, audio |
| `monitor(kb_id, monitor_type, schedule, notify_url, ...)` | Re-crawl on a schedule, with webhooks |

### Search

| Method | What it does |
|---|---|
| `search_kb(kb_id, q, page=1, size=10, ...)` | BM25 keyword search |
| `embed_search_kb(kb_id, q, size=10, ...)` | Hybrid — vector KNN and BM25 fused with Reciprocal Rank Fusion |

`embed_search_kb` is what you want for questions phrased in natural language;
`search_kb` for exact terms, identifiers and error codes.

### Jobs

`get_job(job_id)` · `list_jobs(...)` · `list_kb_jobs(kb_id, ...)` ·
`cancel_job(job_id)` · `delete_job(job_id)` · `get_job_logs(job_id)` ·
`get_job_audit(job_id)` · `get_job_source_md(job_id)` ·
`get_job_source_pdf(job_id)`

`get_job_audit` scores the conversion — ROUGE-1 and Jaccard against the source —
so you can tell a clean parse from a mangled one before you trust the content.

### Pages, files, export

`list_pages(kb_id, ...)` · `get_page(kb_id, page_id, format="json")` ·
`list_job_files(kb_id, job_id)` · `download_file(kb_id, job_id, file_path)` ·
`export_job_zip(kb_id, job_id)` · `export_kb_zip(kb_id)` ·
`export_kb(kb_id, format="jsonl")`

`export_kb` is a generator — it streams records rather than building the whole
export in memory, so a large KB does not have to fit in RAM.

### Health

`health()` · `me()`

---

## ⚠️ Errors

Every failure raises a subclass of `KBError`, so you can branch on the kind
rather than inspecting a status code.

```python
from sarvcrawl import KBClient, NotFoundError, UnauthorizedError

try:
    kb.get_kb("nope")
except NotFoundError:
    ...
except UnauthorizedError:
    ...  # bad or revoked key
```

`KBError` · `BadRequestError` · `UnauthorizedError` · `NotFoundError` ·
`InternalServerError` · `ServiceUnavailableError`

---

## 🔧 Configuration

```python
KBClient(
    api_key="sarv_sk_…",              # or the KB_API_KEY env var
    api_url="https://crawl.sarv.com", # the default — set your own host to self-host
    timeout=60,
)
```

| Argument | Env fallback | Default |
|---|---|---|
| `api_key` | `KB_API_KEY` | — (**required**; the constructor raises without it) |
| `api_url` | `KB_API_URL` | `https://crawl.sarv.com` |
| `timeout` | — | `60` seconds |

> [!IMPORTANT]
> `api_url` is the **origin**, not the API prefix. The REST API lives at
> `/api/…` and this client adds that itself, while `/health` and `/auth/me` sit
> *outside* `/api` — so a value ending in `/api` makes every call 404.

> [!NOTE]
> This package reads those environment variables but **does not call
> `load_dotenv()`** — a library has no business reading a `.env` from your
> working directory and mutating `os.environ`. Call it yourself in your own
> entrypoint if you want one.

The HTTP API underneath is documented at
[crawl.sarv.com/docs](https://crawl.sarv.com/docs) if you would rather call it
directly.

---

## 📄 License

Apache-2.0. The full text ships in the package as `LICENSE`, and is also at
[apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0).

<div align="center">

**[Docs](https://crawl.sarv.com/docs/python-sdk)** ·
**[SarvCrawl](https://crawl.sarv.com)** ·
**[Node SDK](https://www.npmjs.com/package/sarvcrawl)** ·
**[MCP server](https://www.npmjs.com/package/sarvcrawl-mcp)**

</div>
