Metadata-Version: 2.4
Name: daggydata
Version: 0.1.0
Summary: Upload datasets to Daggy with an API key: one file, several, or a hive-partitioned folder, up to 25 GB.
Keywords: daggy,daggydata,datasets,upload,parquet,csv
Author: Daggy
License-Expression: MIT
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Requires-Dist: requests>=2.31
Requires-Python: >=3.10
Project-URL: Homepage, https://daggydata.com
Description-Content-Type: text/markdown

# daggydata

Upload datasets to [Daggy](https://daggydata.com) from a script, a notebook or cron, with an API
key: one file, several files, or a hive-partitioned folder, up to 25 GB. Bytes go straight to
storage in parallel parts; only `requests` is needed, no cloud credentials.

```bash
pip install daggydata
```

The package installs the `daggy` command and imports as `daggy`.

Create an API key on the **API keys** page of the Daggy app, then:

```bash
export DAGGY_API_KEY=daggy_prod_...
daggy upload sales ./sales.parquet
daggy upload events ./events/ --overwrite --description "Clickstream"
```

```python
import daggy  # reads DAGGY_API_KEY

ds = daggy.upload("sales", "./sales.parquet")
print(ds.name, ds.num_obs, "rows")
```

## What gets uploaded

- **A file** is uploaded under its name, with anything outside `[A-Za-z0-9_.-]` replaced
  (`Q3 sales.csv` → `Q3_sales.csv`).
- **A folder** becomes one dataset. Hive folders like `region=west/` become columns. Marker
  files (`_SUCCESS`, `.crc`, `_temporary/`) are skipped.
- **Formats:** `.parquet .pq .csv .tsv .txt .json .ndjson .jsonl`, one format per upload,
  uncompressed.
- **`--overwrite`** (`overwrite=True`) replaces an existing dataset; without it, an existing
  name is refused. Appends are not supported.

| Limit | Value |
|---|---|
| Size per upload | 25 GiB |
| Files per upload | 200 |
| Open uploads at once | 5 |
| Time to finish an upload | 20 hours |
| Paths in a folder | `[A-Za-z0-9_.=/-]`, at most 8 levels |

## Resuming

The session id is printed (logged, from Python) before any data is sent. If a run is killed or
a part keeps failing, run it again with the id; only the missing parts are sent:

```bash
daggy upload sales ./sales.parquet --resume <session_id>
```

```python
try:
    daggy.upload("sales", "./sales.parquet")
except daggy.UploadError as e:
    print(e.code, e.session_id, e.final_state)
    # final_state None or "uploading": daggy.upload(..., resume=e.session_id)
```

`UploadError` has `status`, `code`, `message`, `session_id` and `final_state` (`None` for a
refused request or a failed part, otherwise `failed`, `aborted` or `uploading`). From Python,
progress goes to the `daggy` logger: `logging.basicConfig(level=logging.INFO)` shows it.

## Options

| CLI | Python | |
|---|---|---|
| `--overwrite` | `overwrite=True` | replace an existing dataset |
| `--description D` | `description="D"` | dataset description (≤ 10,000 characters) |
| `--resume SID` | `resume="SID"` | finish an earlier upload |
| `--workers N` | `workers=N` | parallel part uploads (default 8) |
| `--api URL` | `api_url="URL"` | API address; default from the key (`DAGGY_API_URL` also works) |
| `DAGGY_API_KEY` | `api_key="..."` | the key |

CLI exit codes: 0 done; 1 the upload ended `failed`/`aborted` or needs `--resume`; 2 bad usage,
a refused request or a part that kept failing; 130 interrupted.

## License

MIT
