Metadata-Version: 2.5
Name: scrub.bot
Version: 0.1.7
Summary: Deterministic data cleaning for records, text, documents and more. Same input, same output; every edit logged, anything uncertain flagged.
Project-URL: Homepage, https://www.scrub.bot
Project-URL: Documentation, https://www.scrub.bot/docs
Project-URL: Issues, https://github.com/scrubbot/scrub.bot/issues
License-Expression: LicenseRef-BSL-1.1
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: alembic>=1.18.4
Requires-Dist: arrow>=1.4.0
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: boto3>=1.43.19
Requires-Dist: dateparser>=1.4.0
Requires-Dist: duckdb>=1.5.3
Requires-Dist: faker>=37.0.0
Requires-Dist: fastapi>=0.136.3
Requires-Dist: fastmcp<4,>=3.3.1
Requires-Dist: ftfy>=6.3.1
Requires-Dist: geoip2>=5.2.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: jsonschema>=4.26.0
Requires-Dist: langdetect>=1.0.9
Requires-Dist: manuf>=1.1.5
Requires-Dist: mutagen>=1.47.0
Requires-Dist: num2words>=0.5.14
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: phonenumbers>=9.0.31
Requires-Dist: pillow>=12.2.0
Requires-Dist: presidio-analyzer>=2.2.362
Requires-Dist: presidio-anonymizer>=2.2.362
Requires-Dist: psycopg[binary]>=3.3.4
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: pydantic-settings>=2.14.1
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pypdf>=6.0.0
Requires-Dist: python-docx>=1.2.0
Requires-Dist: python-multipart>=0.0.29
Requires-Dist: python-pptx>=1.0.2
Requires-Dist: python-whois>=0.9
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: redis>=5.2.0
Requires-Dist: rich>=14.0.0
Requires-Dist: rq>=2.9.0
Requires-Dist: schwifty>=2026.3.0
Requires-Dist: sentry-sdk[fastapi]>=2.61.1
Requires-Dist: spacy>=3.8.14
Requires-Dist: sqlalchemy>=2.0.50
Requires-Dist: stripe>=15.2.0
Requires-Dist: tiktoken>=0.13.0
Requires-Dist: timezonefinder>=8.2.4
Requires-Dist: typer>=0.25.1
Requires-Dist: ua-parser>=1.0.0
Requires-Dist: usaddress>=0.5.16
Requires-Dist: uvicorn[standard]>=0.48.0
Requires-Dist: validators>=0.35.0
Provides-Extra: ai
Requires-Dist: detoxify>=0.5.2; extra == 'ai'
Requires-Dist: openai-whisper>=20240930; extra == 'ai'
Requires-Dist: pytesseract>=0.3.13; extra == 'ai'
Requires-Dist: torch>=2.12.0; extra == 'ai'
Requires-Dist: transformers>=5.9.0; extra == 'ai'
Provides-Extra: dev
Requires-Dist: httpx>=0.28.1; extra == 'dev'
Requires-Dist: mypy>=2.1.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.3.0; extra == 'dev'
Requires-Dist: pytest>=9.0.3; extra == 'dev'
Requires-Dist: ruff>=0.15.14; extra == 'dev'
Provides-Extra: grpc
Requires-Dist: grpcio-tools>=1.76.0; extra == 'grpc'
Requires-Dist: grpcio>=1.76.0; extra == 'grpc'
Provides-Extra: server
Description-Content-Type: text/markdown

# scrub.bot

ScrubBot is deterministic data cleaning for records, free text and files: spreadsheets and JSON, pasted text, PDFs and Office documents, images and audio. Its rule-based handlers trim and fix casing, standardize dates and phone numbers, repair broken encoding, strip HTML, mask PII, validate emails and remove duplicates, so the same input always gives the same output. It never invents a value; anything it can't fix with certainty is flagged with a reason. Every run returns the cleaned data, a log of every change, and a list of what it couldn't resolve. Use it in the browser, from the command line, over an API, or from AI agents through MCP; `scrub serve` runs the whole engine on your machine. Model-based handlers (OCR, transcription, classification) are an opt-in extra and are labeled as such.

The package contains the `scrub` command and the engine behind it. `scrub serve` starts the engine with SQLite and no other services, so nothing leaves your computer. The same CLI can also point at a hosted or self-hosted ScrubBot API. AI model handlers are an optional extra: `pip install "scrub.bot[ai]"`.

## Install

```bash
pip install scrub.bot
```

Requires Python 3.12 or newer. If pip reports "No matching distribution found" (older Python, such as the 3.9 that ships with macOS) or "externally-managed-environment" (Homebrew or system Python), install it as an isolated tool instead:

```bash
uv tool install scrub.bot      # or: pipx install scrub.bot
```

Installs two commands that are the same program: `scrub` (short, for daily use) and `scrubbot` (explicit, for docs and scripts).

## Start the engine

```bash
scrub serve                      # local API on http://127.0.0.1:8000, prints a key
```

In another terminal:

```bash
export SCRUBBOT_API_URL=http://127.0.0.1:8000
scrub api doctor                 # check key, endpoint, and version
curl -sO https://www.scrub.bot/demo/contacts.csv
scrub clean/text/whitespace contacts.csv -o contacts.clean.csv
```

The CLI provisions its own key against a local server. To use a hosted or self-hosted ScrubBot API instead, set `SCRUBBOT_API_URL` and `SCRUBBOT_API_KEY`. Hosted accounts are not open yet.

## Run a handler

```bash
scrub handlers pii                              # search the catalog
scrub handlers clean/pii                        # show one handler and its config
scrub clean/pii customers.csv --config strategy=mask
scrub clean/text/whitespace notes.csv
scrub runs                                      # recent run history
```

`scrub clean/pii file.csv` is shorthand for `scrub run clean/pii file.csv`. Input can be CSV, JSON, or inline text. Output goes to stdout or `--output`.

## Run a pipeline

`prep.yaml`:

```yaml
name: customer_prep
handlers:
  - handler: clean/text/whitespace
    config: {}
  - handler: clean/pii
    config: {strategy: mask}
  - handler: validate/format/email
    config: {}
```

```bash
scrub pipeline prep.yaml customers.csv
```

## Python

```python
import scrubbot

c = scrubbot.Client(api_key="sk_...", base_url="https://api.scrub.bot")

result = c.run_handler("clean/pii", data=[{"email": "a@b.com"}], config={"strategy": "mask"})
print(result["records"], result["changes"])

for event in c.stream_pipeline(
    record_chunks=[[{"text": "  hi  "}]],
    handlers=[{"handler": "clean/text/whitespace", "config": {}}],
):
    print(event)
```

Errors raise `scrubbot.ApiError` (HTTP status and payload) or `scrubbot.ApiConnectionError` (unreachable or timed out). The package also ships the generated, fully typed client classes (`scrubbot.HandlersApi`, `scrubbot.PipelineApi`, `scrubbot.JobsApi`, and so on) for callers who want a model per endpoint.

## Environment variables

| Variable | Meaning |
|----------|---------|
| `SCRUBBOT_API_KEY` | API key; overrides the saved login |
| `SCRUBBOT_API_URL` | API base URL; the default today is `http://localhost:8000`, so set this to the hosted API |
| `SCRUBBOT_DEFAULT_FORMAT` | Output format for runs |
| `SCRUBBOT_TIMEOUT` | Request timeout in seconds |

## Self-hosting and license

The server is in the same repository and runs with Docker Compose. It is source-available under the Business Source License 1.1 and converts to Apache 2.0 on 2030-01-01. Internal use is permitted; offering it as a hosted service to third parties needs a commercial license. This client package follows the same license.

Docs: https://www.scrub.bot/docs. Bugs and feature requests: https://github.com/scrubbot/scrub.bot/issues.
