Metadata-Version: 2.4
Name: datacoolie
Version: 0.2.0
Summary: Metadata-driven ETL framework that unifies engines, stays cloud-agnostic, and is currently batch-first with a roadmap to micro-batch and streaming.
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Keywords: etl,etl-framework,data-engineering,metadata-driven,polars,pyspark,lakehouse
Author: DataCoolie Team
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Provides-Extra: all
Provides-Extra: aws
Provides-Extra: cli
Provides-Extra: databricks-external
Provides-Extra: fabric-external
Provides-Extra: metadata-db
Provides-Extra: metadata-excel
Provides-Extra: metadata-yaml
Provides-Extra: polars
Provides-Extra: polars-delta
Provides-Extra: polars-hash
Provides-Extra: polars-iceberg
Provides-Extra: polars-sql
Provides-Extra: source-api
Provides-Extra: source-db-mssql-odbc-polars
Provides-Extra: source-db-native-polars
Provides-Extra: source-db-oracle-polars
Provides-Extra: source-db-polars
Provides-Extra: source-excel-polars
Provides-Extra: spark
Provides-Extra: spark-delta
Requires-Dist: azure-identity (>=1.17,<2.0) ; extra == "all"
Requires-Dist: azure-identity (>=1.17,<2.0) ; extra == "fabric-external"
Requires-Dist: azure-keyvault-secrets (>=4.8,<5.0) ; extra == "all"
Requires-Dist: azure-keyvault-secrets (>=4.8,<5.0) ; extra == "fabric-external"
Requires-Dist: azure-storage-file-datalake (>=12.14,<13.0) ; extra == "all"
Requires-Dist: azure-storage-file-datalake (>=12.14,<13.0) ; extra == "fabric-external"
Requires-Dist: boto3 (>=1.43.2) ; extra == "all"
Requires-Dist: boto3 (>=1.43.2) ; extra == "aws"
Requires-Dist: build (>=1.2,<2.0) ; extra == "all"
Requires-Dist: build (>=1.2,<2.0) ; extra == "cli"
Requires-Dist: connectorx (>=0.4,<0.5) ; extra == "all"
Requires-Dist: connectorx (>=0.4,<0.5) ; extra == "source-db-polars"
Requires-Dist: databricks-sdk (>=0.121,<0.122) ; extra == "all"
Requires-Dist: databricks-sdk (>=0.121,<0.122) ; extra == "databricks-external"
Requires-Dist: delta-spark (>=3.0) ; extra == "all"
Requires-Dist: delta-spark (>=3.0) ; extra == "spark-delta"
Requires-Dist: deltalake (>=0.15) ; extra == "all"
Requires-Dist: deltalake (>=0.15) ; extra == "polars-delta"
Requires-Dist: fastexcel (>=0.9) ; extra == "all"
Requires-Dist: fastexcel (>=0.9) ; extra == "source-excel-polars"
Requires-Dist: httpx (>=0.24) ; extra == "all"
Requires-Dist: httpx (>=0.24) ; extra == "source-api"
Requires-Dist: jsonschema (>=4.0,<5.0) ; extra == "all"
Requires-Dist: jsonschema (>=4.0,<5.0) ; extra == "cli"
Requires-Dist: openpyxl (>=3.1) ; extra == "all"
Requires-Dist: openpyxl (>=3.1) ; extra == "metadata-excel"
Requires-Dist: openpyxl (>=3.1) ; extra == "source-excel-polars"
Requires-Dist: oracledb (>=2.0,<5.0) ; extra == "all"
Requires-Dist: oracledb (>=2.0,<5.0) ; extra == "source-db-oracle-polars"
Requires-Dist: packaging (>=24.0,<27.0) ; extra == "all"
Requires-Dist: packaging (>=24.0,<27.0) ; extra == "cli"
Requires-Dist: polars (>=1.0) ; extra == "all"
Requires-Dist: polars (>=1.0) ; extra == "polars"
Requires-Dist: polars (>=1.0) ; extra == "polars-delta"
Requires-Dist: polars (>=1.0) ; extra == "polars-iceberg"
Requires-Dist: polars (>=1.0) ; extra == "polars-sql"
Requires-Dist: polars (>=1.0) ; extra == "source-db-mssql-odbc-polars"
Requires-Dist: polars (>=1.0) ; extra == "source-db-native-polars"
Requires-Dist: polars (>=1.0) ; extra == "source-db-oracle-polars"
Requires-Dist: polars (>=1.0) ; extra == "source-db-polars"
Requires-Dist: polars (>=1.0) ; extra == "source-excel-polars"
Requires-Dist: polars-hash (>=0.6) ; extra == "all"
Requires-Dist: polars-hash (>=0.6) ; extra == "polars-hash"
Requires-Dist: pyiceberg (>=0.12.0,<0.13.0) ; extra == "all"
Requires-Dist: pyiceberg (>=0.12.0,<0.13.0) ; extra == "polars-iceberg"
Requires-Dist: pymssql (>=2.3,<3.0) ; extra == "all"
Requires-Dist: pymssql (>=2.3,<3.0) ; extra == "source-db-native-polars"
Requires-Dist: pymysql (>=1.1,<3.0) ; extra == "all"
Requires-Dist: pymysql (>=1.1,<3.0) ; extra == "source-db-native-polars"
Requires-Dist: pyodbc (>=5.0,<6.0) ; extra == "all"
Requires-Dist: pyodbc (>=5.0,<6.0) ; extra == "source-db-mssql-odbc-polars"
Requires-Dist: pyspark (>=3.5) ; extra == "all"
Requires-Dist: pyspark (>=3.5) ; extra == "spark"
Requires-Dist: pyspark (>=3.5) ; extra == "spark-delta"
Requires-Dist: python-dateutil (>=2.9,<3.0)
Requires-Dist: pyyaml (>=6.0,<7.0) ; extra == "all"
Requires-Dist: pyyaml (>=6.0,<7.0) ; extra == "cli"
Requires-Dist: pyyaml (>=6.0,<7.0) ; extra == "metadata-yaml"
Requires-Dist: sqlalchemy (>=2.0,<3.0) ; extra == "all"
Requires-Dist: sqlalchemy (>=2.0,<3.0) ; extra == "metadata-db"
Requires-Dist: sqlalchemy (>=2.0,<3.0) ; extra == "source-db-mssql-odbc-polars"
Requires-Dist: sqlglot (>=30,<31) ; extra == "all"
Requires-Dist: sqlglot (>=30,<31) ; extra == "polars-sql"
Requires-Dist: typing-extensions (>=4.4) ; extra == "all"
Requires-Dist: typing-extensions (>=4.4) ; extra == "cli"
Requires-Dist: tzdata (>=2024.1) ; sys_platform == "win32"
Project-URL: Documentation, https://datacoolie.github.io/datacoolie/
Project-URL: Homepage, https://github.com/datacoolie/datacoolie
Project-URL: Repository, https://github.com/datacoolie/datacoolie
Description-Content-Type: text/markdown

<p align="center">
    <img src="https://raw.githubusercontent.com/datacoolie/datacoolie/main/docs/images/banners/datacoolie-banner-dark.png" alt="DataCoolie banner" width="100%">
</p>

<p align="center">
  <a href="https://pypi.org/project/datacoolie/"><img src="https://img.shields.io/pypi/v/datacoolie" alt="PyPI version"></a>
  <a href="https://pypi.org/project/datacoolie/"><img src="https://img.shields.io/pypi/pyversions/datacoolie" alt="Python versions"></a>
  <a href="https://pypi.org/project/datacoolie/"><img src="https://img.shields.io/pypi/dm/datacoolie" alt="Downloads"></a>
  <a href="https://github.com/datacoolie/datacoolie/actions/workflows/ci.yml"><img src="https://github.com/datacoolie/datacoolie/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://datacoolie.github.io/datacoolie/"><img src="https://img.shields.io/badge/docs-datacoolie.github.io-blue" alt="Docs"></a>
  <a href="LICENSE"><img src="https://img.shields.io/github/license/datacoolie/datacoolie" alt="License"></a>
</p>

# DataCoolie — Multi-engine, Multi-platform Data Pipeline Framework

DataCoolie is a metadata-driven Python data pipeline framework. It supports
SQL and custom Python sources, unifies compatible execution across Polars and
Spark, and runs on Local, Microsoft Fabric, Databricks and AWS environments.
It is batch-first and scales stages through independently launched Driver jobs.

## What problem does it solve?

Data teams often prototype pipelines locally, then rewrite the same pipeline
for Spark and again for each cloud runtime. That duplicates ETL code and makes
operational behavior such as watermarks, schema hints, partitions, load
strategies, and maintenance drift across environments.

DataCoolie solves this by separating pipeline intent from execution details.
You define connections, dataflows, transforms, and operational controls as
metadata, then run the same intent on Polars or Spark and on local, Fabric,
Databricks, or AWS platforms.

A source can combine multiple operations into one engine-compatible DataFrame
through SQL or a Python function. Built-in transformers operate on that current
DataFrame; the project runner remains responsible for engine setup, custom
table registration, credentials and host-specific configuration.

## Why it helps

- **Metadata-driven** — pipeline behavior lives in metadata instead of being re-implemented in each job.
- **Right-sized compute** — small and medium jobs can stay on lighter runtimes like Polars or local execution instead of paying Spark or cluster overhead too early.
- **Portable** — reuse one canonical metadata model across environments, with overlays and runners for target-specific paths, catalogs, engines, and runtimes.
- **Engine-unified** — compatible pipeline intent runs on Spark and Polars through engine-specific runners.
- **Cloud-agnostic** — `local`, `aws`, `fabric`, `databricks` platforms abstract file I/O and secrets.
- **Lakehouse-native** — first-class Delta Lake and Apache Iceberg via the shared `fmt="delta"` / `fmt="iceberg"` API; concrete addressing and optional dependencies vary by engine.
- **Operationally complete** — watermarks, schema hints, partitions, load strategies, logging, and maintenance are built in.
- **Extensible components** — engines, platforms, sources, destinations,
  transformers, and secret resolvers use registries with Python entry-point
  discovery; built-ins are also registered in-process.

## See DataCoolie in action

Watch the short demos to see the metadata-driven workflow from setup through
pipeline execution:

| English demo | Vietnamese demo |
| --- | --- |
| [![Watch the English DataCoolie demo](https://i.ytimg.com/vi/L9ejQf9tYAE/hqdefault.jpg)](https://youtu.be/L9ejQf9tYAE) | [![Xem demo DataCoolie bằng tiếng Việt](https://i.ytimg.com/vi/0lnxfS6WYew/hqdefault.jpg)](https://youtu.be/0lnxfS6WYew) |

## Start here

If you are evaluating DataCoolie for the first time, use this order:

1. Install the smallest useful runtime: `pip install "datacoolie[polars-delta]"`
2. Run the quick start below
3. Then move to the docs for using your own input and building a multi-stage flow

If you already know your runtime will be Spark, swap the install to
`pip install "datacoolie[spark-delta]"` and keep the same metadata pattern.

## Installation

```bash
# Most common first install
pip install "datacoolie[polars-delta]"

# Spark-first local validation
pip install "datacoolie[spark-delta]"

# Add stable hash_columns support to a Polars runtime
pip install "datacoolie[polars-delta,polars-hash]"

# Core only (mainly useful for extension work)
pip install datacoolie

# Project tooling (init, validate, inspect, build, metadata conversion)
pip install "datacoolie[cli]"

# All engines
pip install "datacoolie[all]"

# External platform SDKs (native Fabric/Databricks runtimes use base install)
pip install "datacoolie[fabric-external]"
pip install "datacoolie[databricks-external]"
pip install "datacoolie[aws]"  # AWS, MinIO, or LocalStack
```

The project CLI is intentionally preparation-only; run dataflows from a
project-owned Python script or notebook so custom engine setup remains under
user control. The `dc` and `datacoolie` executables are equivalent aliases.
See the [DataCoolie user guide](docs/guide/index.md) and
[CLI reference](docs/guide/cli/index.md).

Extras are composable by use case rather than by a platform × engine matrix.
For example, `polars-delta,source-db-oracle-polars,aws` covers a Polars Delta
pipeline that reads Oracle and writes to S3. Use
`source-db-native-polars` when a MySQL or MSSQL source must retain unsigned
integer and high-precision decimal values before schema-hint casting.

## Quick Start

Install, then run two short scripts:

1. `prepare_quickstart.py` creates a sample CSV and `metadata.json`.
2. `run_quickstart.py` loads that metadata and runs the pipeline.

```bash
pip install "datacoolie[polars]"
```

### Part 1 — Prepare sample data and metadata

```python
# prepare_quickstart.py
import json
from pathlib import Path

root = Path("dc_quickstart")
(root / "input" / "orders").mkdir(parents=True, exist_ok=True)
(root / "output").mkdir(parents=True, exist_ok=True)

(root / "input/orders/orders.csv").write_text(
    "order_id,customer_id,amount\n1,100,19.99\n2,100,42.50\n3,101,7.25\n"
)

metadata = {
    "connections": [
        {"name": "csv_in", "connection_type": "file", "format": "csv",
         "configure": {"base_path": str(root / "input"),
                "read_options": {"header": "true", "inferSchema": "true"}}},
        {"name": "parquet_out", "connection_type": "file", "format": "parquet",
         "configure": {"base_path": str(root / "output")}},
    ],
    "dataflows": [
        {"name": "orders_csv_to_parquet", "stage": "bronze2silver",
         "processing_mode": "batch",
         "source": {"connection_name": "csv_in", "table": "orders"},
         "destination": {"connection_name": "parquet_out", "table": "orders",
                         "load_type": "full_load"},
         "transform": {}},
    ],
}
metadata_path = root / "metadata.json"
metadata_path.write_text(json.dumps(metadata, indent=2))
print(f"Created {metadata_path}")
```

```bash
python prepare_quickstart.py
```

### Part 2 — Run the pipeline

```python
# run_quickstart.py
from pathlib import Path

from datacoolie.engines.polars_engine import PolarsEngine
from datacoolie.platforms.local_platform import LocalPlatform
from datacoolie.metadata.file_provider import FileProvider
from datacoolie.orchestration.driver import DataCoolieDriver

root = Path("dc_quickstart")
metadata_path = root / "metadata.json"

platform = LocalPlatform()
engine = PolarsEngine(platform=platform)
provider = FileProvider(config_path=str(metadata_path), platform=platform)

with DataCoolieDriver(engine=engine, metadata_provider=provider) as driver:
    result = driver.run(stage="bronze2silver")
    print(f"Completed: {result.succeeded}/{result.total}")
```

```bash
python run_quickstart.py
```

Reuse the same dataflow intent with `SparkEngine` or a cloud platform by adding
the target runtime dependencies, runner, and environment-specific paths or
catalog settings. Compatibility is per selected engine, address mode, and
installed optional dependency.

## What to do next

- Understand the framework and ecosystem boundary: <https://datacoolie.github.io/datacoolie/introduction/>
- Use your own files while keeping the same runner pattern: <https://datacoolie.github.io/datacoolie/guide/getting-started/use-your-own-data/>
- Configure Driver paths, replay and run attributes: <https://datacoolie.github.io/datacoolie/guide/operations/runtime-configuration/>
- Build a multi-stage bronze→silver tutorial flow: <https://datacoolie.github.io/datacoolie/guide/getting-started/multi-stage-dataflow/>
- Learn the metadata model field by field: <https://datacoolie.github.io/datacoolie/guide/metadata/>
- Install the official DataCoolie Skills workflow: <https://datacoolie.github.io/datacoolie/introduction/ai-skills/>
- Watch the WWI multi-cloud Medallion walkthrough: <https://datacoolie.github.io/datacoolie/examples/wwi-medallion-multicloud/>

## AI-assisted project workflow

DataCoolie Skills are an official public feature. Install the five lifecycle
Skills with
`npx skills add datacoolie/datacoolie`. The framework-facing project source of
truth remains `datacoolie.yml`, its configured metadata/components, and
project-owned runners. Generated builds and `.runtime` state stay separate
from source; agent approvals and evidence are workflow concerns documented by
the Skills guide and do not change the Driver runtime contract.

The shared project and framework contract lives in the [public project workflow](https://datacoolie.github.io/datacoolie/guide/cli/project/)
and [runtime configuration guide](https://datacoolie.github.io/datacoolie/guide/operations/runtime-configuration/).
[`ai/AGENTS.md`](ai/AGENTS.md) is the agent routing and safety layer: it routes
work by required outcome, while discovery, material design, provisioning, and
explicit release retain their separate exact-scope gates.

See the [public DataCoolie Skills guide](https://datacoolie.github.io/datacoolie/introduction/ai-skills/)
for prerequisites, installation, routing, project state, and approval boundaries.

## Testbed & scenarios

See [usecase-sim/README.md](usecase-sim/README.md) for a ready-made integration
testbed that exercises the local `{polars,spark} × {file,database,api}` matrix,
selected AWS-platform file scenarios, lakehouse maintenance, and a Docker-compose
backend stack. It is a representative scenario set, not a full platform cross-product.

## License

[AGPL-3.0-or-later](LICENSE) — free and open source.

See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution terms.

