Metadata-Version: 2.3
Name: FlowerPower
Version: 0.3.4
Summary: A simple workflow framework. Hamilton + APScheduler = FlowerPower
Author-email: "Volker L." <ligno.blades@gmail.com>
Keywords: apscheduler,dask,hamilton,pipeline,ray,scheduler,workflow
Requires-Python: >=3.11
Requires-Dist: fsspec>=2024.5.0
Requires-Dist: munch>=4.0.0
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.7.1
Requires-Dist: sf-hamilton-sdk>=0.5.2
Requires-Dist: sf-hamilton[visualization]>=1.69.0
Requires-Dist: typer>=0.12.3
Provides-Extra: dask
Requires-Dist: dask[complete]>=2024.7.1; extra == 'dask'
Provides-Extra: mongodb
Requires-Dist: pymongo>=4.7.2; extra == 'mongodb'
Provides-Extra: mqtt
Requires-Dist: paho-mqtt>=2.1.0; extra == 'mqtt'
Provides-Extra: ray
Requires-Dist: ray>=2.34.0; extra == 'ray'
Provides-Extra: redis
Requires-Dist: redis>=5.0.4; extra == 'redis'
Provides-Extra: scheduler
Requires-Dist: aiosqlite>=0.20.0; extra == 'scheduler'
Requires-Dist: apscheduler>=4.0.0a5; extra == 'scheduler'
Requires-Dist: asyncpg>=0.29.0; extra == 'scheduler'
Requires-Dist: greenlet>=3.0.3; extra == 'scheduler'
Requires-Dist: sqlalchemy>=2.0.30; extra == 'scheduler'
Provides-Extra: ui
Requires-Dist: sf-hamilton-ui>=0.0.11; extra == 'ui'
Description-Content-Type: text/markdown

# FlowerPower

![Bild](./image.png)

FlowerPower is a simple workflow framework based on the fantastic [Hamilton](https://github.com/DAGWorks-Inc/hamilton) and [Advanced Python Scheduler - APScheduler](https://github.com/agronholm/apscheduler)

## Installation

```shell
pip install "flowerpower" 
# with scheduler
pip install "flowerpower[scheduler]" 
# with mqtt event broker
pip install "flowerpower[scheduler,mqtt]" 
# with redis event broker
pip install "flowerpower[scheduler,redis]" 
# with mongodb data store
pip install "flowerpower[scheduler,mongodb]"
# with ray distributed computing
pip install "flowerpower[scheduler,ray]"
# with dask distributed computing
pip install "flowerpower[scheduler,dask]"
```

## Usage

### 0) Optional: Dev Services
```shell
curl -O https://raw.githubusercontent.com/legout/flowerpower/main/docker/Dockerfile
curl -O https://raw.githubusercontent.com/legout/flowerpower/main/docker/docker-compose.yml

# Hamilton UI, which allows you to track and visualize your pipelines
docker-compose up hamilton_ui -d 
# jupyterlab and code-server
docker-compose up jupytercode -d 
# s3 compatible object storage
docker-compose up minio -d 
#  mosquitto mqtt broker if you want to use mqtt as the event broker
docker-compose up mqtt -d 
# valkey (OSS redis) if you want to use redis as the event broker
docker-compose up redis -d 
# mongodb if you want to use mongodb as the data store
docker-compose up mongodb -d 
 # postgres db if you want to use postgres as data store and/or event broker. This db is also used for hamilton ui
docker-compose up postgres -d

```


### a) Initialze a new flowerpower project
```shell
flowerpower init new-project
cd new-project
```
This adds basic config files `conf/pipelines.yml`, `conf/scheduler.yml` and `conf/tracker.yml`

### b) Add a new pipeline
```shell
flowerpoweradd-pipeline my_flow
```
A new file `pipelines/my_flow.py` is created and the relevant entries are added to the config files.

### c) Setup the new pipeline
Edit `pipelines/my_flow.py` and add the pipeline functions.

FlowerPower uses [Hamilton](https://github.com/DAGWorks-Inc/hamilton) that converts your pipeline functions into nodes and then creates a [Directed Acyclic Graph (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph).

It is therefore mandatory to write your pipeline files according to the Hamilton paradigm. You can read more about this in the Hamilton documentaion chapter [Function, Nodes and DataFlow](https://hamilton.dagworks.io/en/latest/concepts/node/)

Optinally edit the config files `conf/pipelines.yml`, `conf/scheduler.yml` and `conf/tracker.yml`

### d) Run or Scheduler the new pipeline
```shell
flowerpower run-pipeline my_flow
# or schedule with a 30 seconds interval
flowerpower schedule-pipeline my_flow interval --interval-params seconds=30 --auto-start
```





