Metadata-Version: 2.4
Name: dirbear
Version: 0.1.2
Summary: Download client for Dir Bear datasets: distilled, token-counted web text for AI training (dirbear.com)
Author-email: Dir Bear <hello@dirbear.com>
License: MIT
Project-URL: Homepage, https://dirbear.com/
Project-URL: Datasets, https://dirbear.com/datasets/
Project-URL: Open Library, https://dirbear.com/library/
Project-URL: Hugging Face, https://huggingface.co/directorybear
Project-URL: For AI agents, https://dirbear.com/llms.txt
Keywords: dataset,ai training data,llm,pretraining,web corpus,text corpus,jsonl,parquet,nlp,dir bear
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31
Requires-Dist: rich>=13.7
Requires-Dist: questionary>=2.0
Provides-Extra: parquet
Requires-Dist: pyarrow>=15; extra == "parquet"
Requires-Dist: zstandard>=0.22; extra == "parquet"

# dirbear — download client for [Dir Bear](https://dirbear.com/)

[Dir Bear](https://dirbear.com/) distils terabytes of public web crawl into clean, full-length,
token-counted documents for AI training — every page scored with Bear Rank, sorted onto topic
shelves, delivered as zstd-compressed JSONL. This is the command-line client customers use to
download what they've bought.

- **Browse the datasets and pricing:** [dirbear.com/datasets](https://dirbear.com/datasets/)
- **Read 10,000 documents free:** [the Open Library](https://dirbear.com/library/)
- **Free Hugging Face datasets:** [huggingface.co/directorybear](https://huggingface.co/directorybear)

One
credential (a customer ID), resumable downloads with checksums, an arrow-key
menu, and local conversion to Parquet.

```
pipx install "dirbear[parquet]"   # or: pip install "dirbear[parquet]" (plain `dirbear` works too)
dirbear
```

```
  { ≡ }  Dir Bear   ·   secure downloads
  ──────────────────────────────────────────────────────────

› Customer ID  DB-7F3A-9C21-XK4Q

  Product                 Files      Size    Build
  Finance & Business         42   61.2 GB    2026.09
  Mixed bundles              18   38.0 GB    2026.09
  Customer DB-7F3A-9C21-XK4Q · Acme AI · access until 2027-09-23 · 60 files, 99.2 GB

› What would you like to do?
  ❯ Browse files and download one
    Download everything
    Download everything and convert to Parquet
    Convert downloaded files to Parquet  (~/DirBear)
    Change download folder  (~/DirBear)
    Use a different customer ID
    Exit
```

Non-interactive:

```
dirbear --customer DB-7F3A-9C21-XK4Q --all --dest /data/dirbear          # everything, JSONL.zst
dirbear --customer DB-7F3A-9C21-XK4Q --all --parquet                      # …and convert each file
dirbear convert /data/dirbear                                             # convert later
dirbear --list
```

## How access works

There is no server. The client reads two static files from the CDN:

- `https://dirbear.com/data/accounts/<CUSTOMER_ID>.json` — `{customer, name, expires, access:["shelf:finance","mixed"]}`
- `https://dirbear.com/data/catalog.json` — every product with its files (path, bytes, sha256) and the
  base URL, which includes a random per-build folder (`/d-RFkt6yjL1cWk/`).

The client expands the access list (wildcards allowed, `["*"]` = everything), downloads
from `base + path` with HTTP Range resume and three retries, verifies SHA-256, and skips
files already complete. When a customer's year ends, `dirbear-admin sweep` deletes their
account file and the client can no longer list or find the files. Someone determined
could keep URLs they'd already seen — that's accepted; the point is that nothing is
linkable or guessable from the website.

Only the customer ID and download folder are stored locally (`~/.dirbear/config.json`).

## Parquet without storing Parquet

Files are delivered as `.jsonl.zst`. "Convert to Parquet" runs on the customer's
machine (`pip install pyarrow zstandard`), streaming each file into a zstd-compressed
Parquet file next to it — the same rows, the industry format, and nothing extra on the CDN.

## Releasing

```
cd cli && python -m pip install build twine && python -m build && twine upload dist/*
```

Until it is on PyPI: `pipx install ./cli`.

## The admin side

`admin/dirbear_admin.py` (never shipped) manages customers and the catalogue — see
its docstring. Emails and notes stay in `admin/customers.json` on the Mac; only id,
name, expiry and access go to the CDN.

Questions or orders: hello@dirbear.com · [dirbear.com](https://dirbear.com/)
