Metadata-Version: 2.4
Name: streamwright-spark
Version: 0.1.0
Summary: Spark jobs for StreamWright (kind: SparkJob), attached as `sw spark ...`.
Author-email: Karthick Jaganathan <kjaganathan.sde@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://streamwright.web.app
Project-URL: Documentation, https://streamwright.web.app/docs/
Project-URL: Repository, https://github.com/karthick-jaganathan/streamwright-spark
Project-URL: Issue Tracker, https://github.com/karthick-jaganathan/streamwright-spark/issues
Keywords: data,pipeline,etl,spark,pyspark,deltalake,kafka,sqs,yaml,streamwright
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml==6.0.2
Requires-Dist: streamwright~=0.1.9
Provides-Extra: spark
Requires-Dist: pyspark<3.6,>=3.5; extra == "spark"
Provides-Extra: delta
Requires-Dist: delta-spark<3.3,>=3.2; extra == "delta"
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == "postgres"
Provides-Extra: kafka
Requires-Dist: confluent-kafka>=2.4; extra == "kafka"
Requires-Dist: aws-msk-iam-sasl-signer-python>=1.0; extra == "kafka"
Provides-Extra: sqs
Requires-Dist: boto3>=1.34; extra == "sqs"
Provides-Extra: emr
Requires-Dist: boto3>=1.34; extra == "emr"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: jsonschema>=4; extra == "test"
Provides-Extra: all
Requires-Dist: pyspark<3.6,>=3.5; extra == "all"
Requires-Dist: delta-spark<3.3,>=3.2; extra == "all"
Requires-Dist: psycopg[binary]>=3.1; extra == "all"
Requires-Dist: confluent-kafka>=2.4; extra == "all"
Requires-Dist: aws-msk-iam-sasl-signer-python>=1.0; extra == "all"
Requires-Dist: boto3>=1.34; extra == "all"
Dynamic: license-file

# StreamWright Spark

[![PyPI](https://img.shields.io/pypi/v/streamwright-spark.svg)](https://pypi.org/project/streamwright-spark/)
[![Python](https://img.shields.io/pypi/pyversions/streamwright-spark.svg)](https://pypi.org/project/streamwright-spark/)
[![License](https://img.shields.io/pypi/l/streamwright-spark.svg)](https://pypi.org/project/streamwright-spark/)
[![Docs](https://img.shields.io/badge/docs-streamwright.web.app-brightgreen.svg)](https://streamwright.web.app/docs/spark/overview/)

`kind: SparkJob` for [StreamWright](https://pypi.org/project/streamwright/): declarative batch pipelines on Apache Spark.

- **Read and write** object storage, Delta Lake, databases, Kafka and SQS with StreamWright connectors.
- **Exactly-once batches**: batch state, leases and output receipts; replay-safe writes.
- **Any source connector** (REST APIs, ads platforms) through driver-side call steps.
- **Runs** locally, on Spark standalone and on AWS EMR on EC2.

## Install

```bash
pip install "streamwright-spark[spark]"    # needs Java 11+
sw connectors -i files
```

## Example

```yaml
# orders.yaml
kind: SparkJob
version: 1
name: daily_orders
connections:
  lake: {provider: files, roots: ["/data/lake/"]}
steps:
  - name: orders
    connection: lake
    service: object
    method: read
    arguments: {path: "raw/orders/*.csv", format: csv, options: {header: true}}
  - name: daily_totals
    sql: SELECT CAST(ordered_at AS DATE) AS day, COUNT(*) AS orders FROM orders GROUP BY 1
outputs:
  - name: save_totals
    connection: lake
    service: object
    method: write
    arguments: {path: "/data/lake/curated/daily_totals/", format: parquet, mode: overwrite}
    sql: SELECT * FROM daily_totals
runtime:
  state: {sqlite: /data/state/daily_orders.db}
  platform: {type: local}
```

```bash
sw spark validate orders.yaml
sw spark run orders.yaml --once
```

## Links

[Documentation](https://streamwright.web.app/docs/spark/overview/) ·
[StreamWright](https://github.com/karthick-jaganathan/streamwright)
