Metadata-Version: 2.4
Name: pyingestkit
Version: 2.2.0
Summary: Composable ingestion tooling for Python.
Author: Thomas Awounfouet
License-Expression: MIT
Project-URL: Homepage, https://github.com/tawounfouet/pyingestkit
Project-URL: Documentation, https://tawounfouet.github.io/pyingestkit/
Project-URL: Repository, https://github.com/tawounfouet/pyingestkit.git
Project-URL: Changelog, https://github.com/tawounfouet/pyingestkit/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/tawounfouet/pyingestkit/issues
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3,>=2.11
Requires-Dist: PyYAML<7,>=6
Requires-Dist: python-dotenv<2,>=1.0
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=6; extra == "dev"
Requires-Dist: pytest-randomly>=3.15; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: mypy>=1.11; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: bandit>=1.8; extra == "dev"
Requires-Dist: pip-audit>=2.9; extra == "dev"
Requires-Dist: pre-commit>=4; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6; extra == "dev"
Requires-Dist: openpyxl<4,>=3.1; extra == "dev"
Requires-Dist: pyarrow<24,>=16; extra == "dev"
Requires-Dist: psycopg[binary]<4,>=3.2; extra == "dev"
Requires-Dist: boto3<2,>=1.35; extra == "dev"
Requires-Dist: SQLAlchemy<3,>=2.0; extra == "dev"
Requires-Dist: httpx<1,>=0.28.1; extra == "dev"
Requires-Dist: typer<0.28,>=0.27; extra == "dev"
Requires-Dist: rich<16,>=15; extra == "dev"
Requires-Dist: tenacity<10,>=9.1.4; extra == "dev"
Provides-Extra: cli
Requires-Dist: typer<1,>=0.15; extra == "cli"
Requires-Dist: rich<16,>=13; extra == "cli"
Provides-Extra: http
Requires-Dist: httpx<1,>=0.28.1; extra == "http"
Provides-Extra: postgres
Requires-Dist: SQLAlchemy<3,>=2.0; extra == "postgres"
Requires-Dist: psycopg[binary]<4,>=3.2; extra == "postgres"
Provides-Extra: s3
Requires-Dist: boto3<2,>=1.35; extra == "s3"
Provides-Extra: excel
Requires-Dist: openpyxl<4,>=3.1; extra == "excel"
Provides-Extra: parquet
Requires-Dist: pyarrow<24,>=16; extra == "parquet"
Dynamic: license-file

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="docs/assets/brand/pyingestkit-primary-lockup-dark.png">
    <source media="(prefers-color-scheme: light)" srcset="docs/assets/brand/pyingestkit-primary-lockup-light.png">
    <img src="docs/assets/brand/pyingestkit-logo-horizontal.png" alt="PyIngestKit — From any source to reliable datasets" width="480">
  </picture>
</p>

<p align="center">
  <strong>From any source to reliable datasets.</strong><br>
  <code>Acquire</code> &nbsp;→&nbsp; <code>Preserve RAW</code> &nbsp;→&nbsp; <code>Validate</code> &nbsp;→&nbsp; <code>Publish</code>
</p>

<p align="center">
  <a href="https://github.com/tawounfouet/pyingestkit/actions/workflows/ci.yml"><img src="https://github.com/tawounfouet/pyingestkit/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://github.com/tawounfouet/pyingestkit/actions/workflows/security.yml"><img src="https://github.com/tawounfouet/pyingestkit/actions/workflows/security.yml/badge.svg" alt="Security"></a>
  <a href="https://pypi.org/project/pyingestkit/"><img src="https://img.shields.io/pypi/v/pyingestkit.svg" alt="PyPI version"></a>
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.11%E2%80%933.14-blue.svg" alt="Python 3.11–3.14"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
  <a href="https://tawounfouet.github.io/pyingestkit/"><img src="https://img.shields.io/badge/docs-online-blue.svg" alt="Docs"></a>
</p>

---

**PyIngestKit** is a lightweight, composable Python framework for reliable, traceable and replayable batch ingestion.

It focuses strictly on **how to ingest data reliably** — preserving durable RAW acquisition evidence, producing immutable content-addressed dataset versions, and enforcing atomic compare-and-swap (CAS) publications with zero accidental overwrites.

> External orchestrators (Airflow, Dagster, Prefect) decide **when to run**. PyIngestKit guarantees **how it runs safely**.

---

## ⚡ Highlights

- 🛡️ **Governed CAS Publication** — Atomic compare-and-swap with durable ledgers (PostgreSQL / Memory), retention holds, guarded GC, and governed rollback.
- 📦 **Zero-Bloat Core** — Pure standard library core by default. Heavy connectors (`psycopg`, `boto3`, `httpx`, `pyarrow`) are strictly opt-in extras.
- 🔄 **Strict Deterministic Replay** — Replay runs from verified historical RAW artifacts; never silently re-fetches live external sources.
- 🩺 **Built-in CLI & Self-Diagnostics** — Standardized `pik` command line with instant environment health check via `pik doctor`.

---

## 🚀 Quickstart (30 seconds)

### 1. Installation

```bash
# Base framework (zero heavy dependencies)
pip install pyingestkit

# Or with specific providers
pip install "pyingestkit[postgres,s3,http]"
```

### 2. Run an Ingestion Pipeline

```python
from pathlib import Path
from pyingestkit import IngestionDefinition, IngestionRuntime, Source
from pyingestkit.adapters.filesystem import FileArtifactStore, FileSourceConnector
from pyingestkit.application.decoders import DecoderRegistry
from pyingestkit.application.sources import SourceRegistry
from pyingestkit.decoders import CsvDecoder
from pyingestkit.stores import FileDatasetVersionStore

root = Path("./.pyingest")

runtime = IngestionRuntime(
    sources=SourceRegistry([FileSourceConnector()]),
    decoders=DecoderRegistry([CsvDecoder()]),
    artifacts=FileArtifactStore(root=root / "artifacts"),
    versions=FileDatasetVersionStore(root=root / "datasets"),
)

result = runtime.run(
    IngestionDefinition(
        name="customers_pipeline",
        source=Source.file(path="./data/customers.csv"),
        decoder="csv",
        dataset="customers",
    )
)

print(f"Status: {result.status.value} | Version: {result.version_id}")
```

---

## 🛠️ CLI: `pik`

PyIngestKit includes a fast CLI with zero runtime overhead on startup:

```bash
# Run system and environment diagnostics
pik doctor

# Inspect environment in JSON format
pik doctor --json
```

---

## 🧩 Providers & Optional Extras

| Extra | Capabilities | Drivers Installed |
| :--- | :--- | :--- |
| `pyingestkit` *(base)* | Core runtime, Filesystem, Memory, CSV, JSON(L) | Standard library |
| `pyingestkit[http]` | Robust HTTP(S) acquisition with retry policies | `httpx` |
| `pyingestkit[postgres]`| PostgreSQL targets, staging & publication ledger | `psycopg`, `sqlalchemy` |
| `pyingestkit[s3]` | S3 / MinIO / Ceph / R2 object storage artifacts | `boto3` |
| `pyingestkit[parquet]` | High-performance Parquet format decoder | `pyarrow` |
| `pyingestkit[excel]` | Excel `.xlsx` / `.xls` spreadsheets decoding | `openpyxl` |

---

## 📚 Documentation & Resources

- 📖 **Official Documentation :** [https://tawounfouet.github.io/pyingestkit/](https://tawounfouet.github.io/pyingestkit/)
- 🏛️ **Architecture & RFC-001 :** [RFC-001 Publication & Lifecycle Governance](docs/rfc/RFC-001-publication-lifecycle-governance.md)
- 🚀 **Migration Guide (V1 → V2) :** [Migration Guide](docs/guides/migrate-v1-to-v2.md)
- 📝 **Changelog & Releases :** [Release Notes](docs/releases/v2.1.0.md)

---

## 📄 License

Distributed under the [MIT License](LICENSE).
