Metadata-Version: 2.4
Name: datasluice
Version: 0.2.0
Summary: One Python interface for open-data discovery, extraction, format normalization, and pipeline integration
Project-URL: bugs, https://github.com/nitish-raj/datasluice/issues
Project-URL: changelog, https://github.com/nitish-raj/datasluice/releases
Project-URL: documentation, https://nitish-raj.github.io/datasluice/
Project-URL: homepage, https://github.com/nitish-raj/datasluice
Author-email: Nitish Raj <contact@rajnitish.com>
Maintainer-email: Nitish Raj <contact@rajnitish.com>
License: MIT
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: rich
Requires-Dist: typer
Provides-Extra: all
Requires-Dist: dlt; extra == 'all'
Requires-Dist: duckdb; extra == 'all'
Requires-Dist: fsspec>=2025.1; extra == 'all'
Requires-Dist: httpx>=0.27; extra == 'all'
Requires-Dist: openpyxl; extra == 'all'
Requires-Dist: pandas; extra == 'all'
Requires-Dist: polars; extra == 'all'
Requires-Dist: pyarrow; extra == 'all'
Requires-Dist: zstandard>=0.23; extra == 'all'
Provides-Extra: compression
Requires-Dist: zstandard>=0.23; extra == 'compression'
Provides-Extra: dlt
Requires-Dist: dlt; extra == 'dlt'
Provides-Extra: duckdb
Requires-Dist: duckdb; extra == 'duckdb'
Provides-Extra: http
Requires-Dist: httpx>=0.27; extra == 'http'
Provides-Extra: pandas
Requires-Dist: pandas; extra == 'pandas'
Provides-Extra: parquet
Requires-Dist: pyarrow; extra == 'parquet'
Provides-Extra: polars
Requires-Dist: polars; extra == 'polars'
Provides-Extra: storage
Requires-Dist: fsspec>=2025.1; extra == 'storage'
Provides-Extra: streaming
Requires-Dist: pyarrow; extra == 'streaming'
Provides-Extra: xlsx
Requires-Dist: openpyxl; extra == 'xlsx'
Description-Content-Type: text/markdown

<p align="center">
  <img src="docs/assets/datasluice.png" alt="DataSluice" width="600">
</p>

<p align="center">
  One Python interface for open-data discovery, extraction, format normalization, and pipeline integration
</p>

<p align="center">
  <a href="https://pypi.org/project/datasluice/"><img src="https://img.shields.io/pypi/v/datasluice.svg" alt="PyPI version"></a>
  <a href="https://github.com/nitish-raj/datasluice/actions"><img src="https://github.com/nitish-raj/datasluice/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://nitish-raj.github.io/datasluice/"><img src="https://img.shields.io/badge/docs-online-blue" alt="Documentation"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License"></a>
</p>

---

* [GitHub](https://github.com/nitish-raj/datasluice/) | [PyPI](https://pypi.org/project/datasluice/) | [Documentation](https://nitish-raj.github.io/datasluice/)
* Created by [Nitish Raj](https://rajnitish.com/) | GitHub [@nitish-raj](https://github.com/nitish-raj) | PyPI [@nitish-raj](https://pypi.org/user/nitish-raj/)
* MIT License

## Installation

```bash
pip install datasluice
```

Optional extras for format and integration support:

```bash
pip install "datasluice[pandas,polars,parquet,xlsx]"
pip install "datasluice[all]"          # everything
```

### Apache Airflow

Airflow integration is a separate distribution that imports from the
`airflow.providers.datasluice` namespace:

```bash
pip install apache-airflow-providers-datasluice
```

## Quick Start

```python
from datasluice import DataSluice

# Point at any supported portal — the portal type is auto-detected
ds = DataSluice("https://www.data.gouv.fr")

# Search for datasets
results = ds.search("climate")
for dataset in results:
    print(dataset.title, len(dataset.resources))

# Inspect a dataset and list its resources
dataset = ds.get_dataset("some-dataset-id")
for resource in dataset.resources:
    print(resource.format, resource.url)

# Download a resource
path = ds.download(dataset.resources[0], "data/")
print(path)
```

CLI:

```bash
datasluice search "climate" --portal https://www.data.gouv.fr
datasluice inspect -p https://www.data.gouv.fr <dataset-id>
datasluice detect https://demo.ckan.org
datasluice download -p https://www.data.gouv.fr <dataset-id> --format csv
```

## Features

* **Unified API** — one interface for CKAN, data.gouv.fr, Socrata, and custom portals
* **Auto-detection** — point at a URL and DataSluice figures out the portal type
* **Format normalization** — CSV, JSON, XLSX, Parquet, and GeoJSON readers
* **Integrations** — pandas, Polars, dlt, DuckDB, and Apache Airflow (separate provider)
* **CLI** — search, inspect, download, and detect from the command line
* **Pipeline-ready** — retry, rate-limiting, caching, and checksum verification built in

## Documentation

Documentation is built with [Zensical](https://zensical.org/) and deployed to GitHub Pages.

* **Live site:** https://nitish-raj.github.io/datasluice/
* **Preview locally:** `just docs-serve` (serves at http://localhost:8000)
* **Build:** `just docs-build`

API documentation is auto-generated from docstrings using [mkdocstrings](https://mkdocstrings.github.io/).

Docs deploy automatically on push to `main` via GitHub Actions. To enable this, go to your repo's Settings > Pages and set the source to **GitHub Actions**.

## Development

To set up for local development:

```bash
# Clone your fork
git clone git@github.com:your_username/datasluice.git
cd datasluice

# Install dependencies (including all optional deps for dev)
uv sync --all-extras

# Install just (task runner) — one-time setup
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to .venv/bin

# Install in editable mode with live updates
uv tool install --editable .
```

This installs the CLI globally but with live updates - any changes you make to the source code are immediately available when you run `datasluice`.

Install pre-commit hooks:

```bash
uv run pre-commit install
```

Run tests:

```bash
uv run pytest
```

Run quality checks (format, lint, type check, test):

```bash
just qa
```

## Release Process

Releases are automated with [Release Please](https://github.com/googleapis/release-please). There is no manual version bumping or tagging.

1. Use [**Conventional Commits**](https://www.conventionalcommits.org/) (`feat:`, `fix:`, `docs:`, …) — see [CONTRIBUTING.md](CONTRIBUTING.md) for the full list.
2. Release Please maintains a **release PR** on `main` that bumps the version and updates the changelog.
3. **Merge the release PR** → Release Please creates a Git tag and a **GitHub Release**.
4. The GitHub Release auto-triggers **publishing to [TestPyPI](https://test.pypi.org/project/datasluice/)**, then **waits for approval** before publishing to [PyPI](https://pypi.org/project/datasluice/).

## Contributing

Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, conventions, and the release workflow. Please follow the [Code of Conduct](CODE_OF_CONDUCT.md).
