Metadata-Version: 2.1
Name: pyspark-delta-scd2
Version: 0.3.1
Summary: This project utilizes faker-pyspark to generate random schema and dataframes to mimic data table snapshots. Using these snapshots to process and apply SCD2 pattern into delta table as the destination.
Home-page: https://github.com/spsoni/pyspark-delta-scd2
License: MIT
Keywords: Faker, PySpark,DeltaTable
Author: Sury Soni
Author-email: github@suryasoni.info
Requires-Python: >=3.8.1,<4.0.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: delta-spark (>=2.4.0,<3.0.0)
Requires-Dist: faker-pyspark (>=0.3.0,<0.4.0)
Project-URL: Repository, https://github.com/spsoni/pyspark-delta-scd2
Description-Content-Type: text/markdown


# Demo PySpark Delta Table SCD2 implementation

This project utilizes `faker-pyspark` to generate random schema and dataframes to mimic data table snapshots.

Using these snapshots to process and apply SCD2 pattern into delta table as the destination. 

## Installation

Install with pip:

``` bash
pip install pyspark-delta-scd2

```

### Generate incremental updates to dataframe and apply scd2

``` python
>>> from pyspark_delta_scd2 import get_spark, PySparkDeltaScd2
>>> spark = get_spark()
>>> demo  = PySparkDeltaScd2(spark=spark)
>>> # initial load
>>> df1   = demo.process()
>>> # incremental update
>>> df2   = demo.process()
>>> # df2 should have some deletes, updates and inserts

```

