Metadata-Version: 2.4
Name: flint-core
Version: 0.6.0
Summary: A minimalist, agnostic Python framework to standardize data engineering pipelines.
License: MIT
License-File: LICENSE
Keywords: data-engineering,pyspark,pandas,data-catalog
Author: idperez720
Author-email: ivandavidperez4@gmail.com
Requires-Python: >=3.11,<4.0.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Provides-Extra: aws
Provides-Extra: azure
Provides-Extra: bigquery
Provides-Extra: databricks
Provides-Extra: delta
Provides-Extra: gcp
Provides-Extra: iceberg
Provides-Extra: jdbc
Provides-Extra: mysql
Provides-Extra: odbc
Provides-Extra: pandas
Provides-Extra: postgres
Provides-Extra: snowflake
Provides-Extra: spark
Provides-Extra: spark-delta
Provides-Extra: sqlite
Requires-Dist: adlfs (>=2024.10.0) ; extra == "azure"
Requires-Dist: click (>=8.4.1,<9.0.0)
Requires-Dist: databricks-sql-connector (>=4.3.0,<5.0.0) ; extra == "databricks"
Requires-Dist: delta-spark (>=4.4.1,<4.5.0) ; extra == "spark-delta"
Requires-Dist: deltalake[pyarrow] (>=1.6.1,<2.0.0) ; extra == "delta"
Requires-Dist: gcsfs (>=2024.10.0) ; extra == "gcp"
Requires-Dist: jaydebeapi (>=1.2.3,<2.0.0) ; extra == "jdbc"
Requires-Dist: pandas (>=3.0.3,<4.0.0) ; extra == "pandas"
Requires-Dist: pandas-gbq (>=0.35.0,<0.36.0) ; extra == "bigquery"
Requires-Dist: psycopg2-binary (>=2.9.12,<3.0.0) ; extra == "postgres"
Requires-Dist: pyiceberg (>=0.11.1,<0.12.0) ; extra == "iceberg"
Requires-Dist: pyodbc (>=5.3.0,<6.0.0) ; extra == "odbc"
Requires-Dist: pyspark (>=4.1.2,<4.2.0) ; extra == "spark-delta"
Requires-Dist: pyspark (>=4.1.2,<5.0.0) ; extra == "spark"
Requires-Dist: pyyaml (>=6.0.3,<7.0.0)
Requires-Dist: s3fs (>=2024.10.0) ; extra == "aws"
Requires-Dist: snowflake-connector-python (>=4.6.0,<5.0.0) ; extra == "snowflake"
Requires-Dist: sqlalchemy (>=2.0.51,<3.0.0) ; extra == "databricks"
Requires-Dist: sqlalchemy (>=2.0.51,<3.0.0) ; extra == "mysql"
Requires-Dist: sqlalchemy (>=2.0.51,<3.0.0) ; extra == "postgres"
Requires-Dist: sqlalchemy (>=2.0.51,<3.0.0) ; extra == "sqlite"
Project-URL: Repository, https://github.com/idperez720/flint-core
Description-Content-Type: text/markdown

# Flint Core Framework 🚀

`flint` is a lightweight, extensible framework for structuring, describing, and
executing data engineering pipelines in Python. Its opt-in v1 workflow currently
supports local Spark/Delta and Pandas CSV/Delta; it does not manage clusters.

Flint focuses on two pillars: Runtime for pipeline execution and Catalog for
declarative dataset metadata. Its I/O adapters support Pandas and PySpark;
transformations remain ordinary user-owned Python functions.

---

## ✨ Core Features

* **Executable Project Scaffolds**: Generate a local Spark or Pandas example
  without installing dependencies or starting an engine.
* **Versioned Catalog**: Separate logical dataset definitions from physical
  bindings in strict YAML manifests with explicit environment selection.
* **Explicit Runtime**: Compile the complete graph before execution; invoke the
  same sequential runner from Python or CLI without a scheduler.
* **First-Class Spark**: Pass a session explicitly or request local creation;
  Spark pipelines do not convert through Pandas or silently adopt active sessions.
* **Fail-Closed Persistence**: Validate complete datasets and publish explicit,
  atomic create-only outputs on supported local filesystems.

These are opt-in v1 contracts, not promises about the separate legacy APIs.
The public v1 API and wider Architecture RFC remain Draft. The next agreed
package version is **0.6.0**, not a stable 1.0 API; see the
[release review](docs/release-0.6.0.md) for migration and merge gates.

---

## Utilities Removed

The next release removes the legacy SCD2 and deduplication APIs and their Pandas
and Spark implementations. Utilities are deferred, not part of the current core.
This is an intentional breaking change; there are no compatibility shims.
See the [migration notes](docs/migration.md) before upgrading existing pipelines.

---

## 📖 Complete Documentation

The opt-in [Spark-first Delta workflow](docs/spark-delta-v1.md) supports versioned
Catalog manifests, explicit engines/resources, full-plan compilation, and shared
Python/CLI execution. Generate its executable example with
`flint init --format v1 --engine spark`; run with an explicit borrowed session or
`flint run --spark-local`. Spark runs without Pandas. The
[Pandas workflow](docs/project-workflow.md) supports local CSV and Delta too.
All v1 writers support explicit atomic `create` only. Remote/cloud execution,
weaker write guarantees, and a scheduler are not implied. Existing APIs remain separate.

For comprehensive guides, testing architecture deep-dives, and complete API references, visit our documentation site:
👉 **[Flint Core Docs](https://flint-core.readthedocs.io/en/latest/)**

---

## ⚖️ License

Distributed under the **MIT License**. Any modification or distribution (including forks) must include the original copyright notice and liability waiver. See `LICENSE` for more information.

