Metadata-Version: 2.4
Name: mr-piply
Version: 0.3.4
Summary: Lightweight Python workflow orchestration platform
Author: Medrcm Piply Team
License: MIT
Project-URL: Homepage, https://github.com/gurudevkumar51/piply
Project-URL: Repository, https://github.com/gurudevkumar51/piply
Project-URL: Issues, https://github.com/gurudevkumar51/piply/issues
Keywords: workflow,scheduler,orchestration,automation,pipeline
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: fastapi<1.0,>=0.110
Requires-Dist: uvicorn<1.0,>=0.29
Requires-Dist: pydantic<3,>=2
Requires-Dist: jinja2<4,>=3.1
Requires-Dist: pyyaml<7,>=6
Requires-Dist: typer<1,>=0.12
Requires-Dist: httpx<1,>=0.27
Requires-Dist: python-dotenv<2,>=1
Requires-Dist: tzdata>=2024.1; sys_platform == "win32"
Provides-Extra: postgres
Requires-Dist: psycopg[binary]<4,>=3.1; extra == "postgres"
Provides-Extra: dev
Requires-Dist: ruff<1,>=0.6; extra == "dev"
Requires-Dist: pre-commit<4,>=3; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest<10,>=9.0.3; extra == "test"
Requires-Dist: pytest-cov<6,>=5; extra == "test"
Requires-Dist: tomli>=2; python_version < "3.11" and extra == "test"
Provides-Extra: types
Requires-Dist: mypy<2,>=1; extra == "types"
Provides-Extra: build
Requires-Dist: build<2,>=1; extra == "build"
Requires-Dist: twine<7,>=6.2; extra == "build"
Requires-Dist: pkginfo>=1.12; extra == "build"

# Piply

Piply is a lightweight Python pipeline framework for teams that want YAML-defined workflows, schedules, retries, logs, sensors, and an operations UI without running a heavy orchestration stack.

The checkout is a **0.3.4 release candidate, not an approved release**. See the
[release assessment](docs/RELEASE_CHECK.md) for test results and remaining
blockers. Published installation instructions below install the latest
approved version, not this candidate.

Windows Python 3.10-3.14 and native PostgreSQL parity checks now pass locally.
Linux Python 3.10-3.14 review CI also passed. CI is restored to the existing
Linux runner and Python 3.13; the corrected workflow passed in review run 670.
No Windows CI runner is required.
`codex/` review-branch pushes validate without publishing. Do not push staging
merely to test the candidate.

It stays small on purpose:

- local dependency-aware DAG execution
- SQLite by default for runs, logs, task outputs, queue state, and sensors,
  with optional PostgreSQL when you'd rather keep state in a managed database
- FastAPI plus server-rendered UI
- no Redis, Celery, Airflow, Prefect, or external queue required

## Features

**Authoring**

- Multi-task pipelines with `depends_on`
- Python script, Python callable, CLI, API, webhook, email, and SSH tasks
- Split the config across files with `include:` — deployments in the master,
  pipelines and alerts in their own
- Reusable YAML `variables` with `{name}` interpolation
- `.env`, environment variables, explicit secrets, and reusable SQL connections
- Metadata-driven `entities` expansion for reusable task templates
- Optional `pipeline_templates` and `pipeline_deployments` for tenant reuse
- Lightweight conditional execution: `run_if: "{report} == 'payment'"`
- Conditional variable values: `active_browser: true if env == "dev" else false`
- Declared artifacts: `artifacts: ["out/*.csv"]`

**Execution**

- Task priority, via `priority: high`, `priority: "***"`, or an `extract***:` id
- Entity priority, via a `payment*` / `adjustment**` suffix on entity values
- Task and pipeline timeouts with a configurable kill grace period
- Per-task upstream failure behavior: `skip`, `fail`, or `continue`
- Task output passing through `context["task_id"]`
- Downstream pipelines inherit upstream variables, env, and outputs
- Every run stores its full runtime configuration, so a downstream run can be
  retried or replayed without re-running the upstream chain

**Operations**

- Schedules, sensors, retries, cancellation, reruns, and searchable logs
- Dry-run preview: `piply plan`, plus an in-UI execution preview
- Manual runs prompt for `{placeholder}` values an upstream would normally supply
- Live log streaming: `piply logs --follow` with pipeline/run/task filters
- Retention and cleanup: `piply prune` with automatic SQLite `VACUUM`
- Backfill a single run or a whole schedule window
- Graceful shutdown and startup recovery — no orphaned RUNNING records
- Prometheus metrics at `GET /metrics` and a runtime Diagnostics page
- Airflow-style pipeline listing with template grouping, sorting, and filtering
- Last-five-runs status dots on every pipeline row, each linking to its run
- Guided first-run setup: choose SQLite or PostgreSQL, and create the first
  admin, before anything is written
- Optional PostgreSQL metadata store: `PIPLY_DATABASE=postgresql://...`, movable
  later from Settings with the existing history copied across
- Accounts, roles, and per-pipeline view/edit/run permissions
- Runs record who started them; pauses and manual runs are attributed by name
- Central SMTP configured once, reused by email tasks and run notifications
- Microsoft Teams alerts to channels and group chats, with reusable destination
  groups and webhooks kept in the environment rather than YAML
- Runs page with filters, sorting, and full multi-level trigger lineage
- Run-level pause/resume: active tasks finish, then further task starts wait
- Upcoming schedule slots within 15 minutes on the Runs page
- Admin audit history in the UI, API, and CLI, with credential-free summaries

## Quick Start

```bash
python -m pip install --upgrade pip
python -m pip install mr-piply
piply init my-piply-project
piply validate --config my-piply-project/piply.yaml
piply start --config my-piply-project/piply.yaml
```

Open [Piply](http://127.0.0.1:8000). For a source checkout, install with
`python -m pip install -e ".[dev,test]"` from the repository root instead.

### Conda and Python 3.13

```bash
conda create -n piply python=3.13 pip
conda activate piply
python -m pip install --upgrade pip
python -m pip install mr-piply
piply init my-piply-project
piply start --config my-piply-project/piply.yaml
```

Activate the same environment whenever starting Piply. Use `python -m pip`
so installation and execution use the same interpreter. After upgrading,
restart the server and refresh the browser. Candidate checks cover explicit
JavaScript MIME types and locally bundled graph libraries on fresh Windows
Conda installations; see the release assessment for the exact coverage.
Windows installs also include timezone data for cron schedules. Minimal Linux
images must provide an IANA database or install `tzdata` explicitly.

Run on a different port when `8000` is already in use:

```bash
piply start --config piply-demo/piply.yaml --port 8080
piply start --config piply-demo/piply.yaml --host 0.0.0.0 --port 8080
```

Create a starter workspace:

```bash
piply init my-piply-project
piply run extract_flow --config my-piply-project/piply.yaml --wait
```

## Minimal YAML

```yaml
version: "1"
title: Piply Workspace
workspace: .

variables:
  scripts_dir: pipelines
  batch_id: demo-batch

connections:
  app_db: sqlite:///sensor_demo.db

pipelines:
  extract_flow:
    schedule:
      every: 15m
    retry:
      attempts: 2
      mode: resume
      delay_seconds: 10
    triggers_on_success:
      - report_flow
    tasks:
      extract:
        type: python
        path: "{scripts_dir}/extract.py"
        function: extract_data

      transform:
        type: python
        path: "{scripts_dir}/extract.py"
        function: transform_data
        depends_on: [extract]

      validate:
        type: cli
        command: python {scripts_dir}/validate_cli.py {batch_id}
        cwd: .
        depends_on: [transform]

  report_flow:
    tasks:
      build_report:
        type: python
        path: "{scripts_dir}/report.py"
        function: build_report
```

Python callable tasks can consume upstream outputs:

```python
def transform_data(context):
    extracted = context["extract"]
    return {"records": extracted["records"] + 1}
```

For plain commands, omit `shell` so Piply uses the platform default shell. Set
`shell: bash` only for Bash-specific syntax and only on machines where Bash is
installed and available on `PATH`:

```yaml
tasks:
  load_env_and_run:
    type: cli
    shell: bash
    command: set -a && source .env && set +a && conda run -n py312_extract python {scripts_dir}/job.py
    cwd: .
```

## Dynamic Entity Mapping

Use `entities` when one task template should run once per business value. Piply expands templates at runtime into normal DAG tasks, so existing retries, logs, outputs, and parallel execution continue to work.

```yaml
pipelines:
  extract_flow:
    entities:
      report:
        - payment
        - adjustment
        - refund
    max_parallel_tasks: 3
    tasks:
      extract:
        type: python
        path: pipelines/extract.py
        function: extract_data
        kwargs:
          report: "{report}"

      validate:
        type: cli
        command: python validate.py --report {report}
        depends_on: [extract]
```

Runtime tasks are generated as `payment.extract -> payment.validate`, `adjustment.extract -> adjustment.validate`, and `refund.extract -> refund.validate`.

## Advanced Deployments

Simple `pipelines:` YAML remains the default. For repeated tenant or environment rollouts, define a reusable template and deployment-specific schedules or variables:

```yaml
pipeline_templates:
  report_pipeline:
    tasks:
      extract:
        type: python
        path: pipelines/extract.py
        function: extract_data
        kwargs:
          tenant: "{tenant}"

pipeline_deployments:
  client_a_reporting:
    template: report_pipeline
    schedule:
      every: 15m
    variables:
      tenant: client_a

  client_b_reporting:
    template: report_pipeline
    schedule:
      cron: "0 * * * *"
    tenant: client_b
```

Each deployment becomes a normal runnable pipeline id, so the scheduler, UI, CLI, and API keep working without a second execution model.

### Deployment Variables In Downstream Pipelines

Variables from a deployment are automatically passed to a downstream pipeline started through `triggers_on_success`. This is useful when several deployments share one downstream workflow. Parent deployment variables take precedence for the triggered run.

A **manual** run has no parent to inherit from, so it uses the downstream pipeline's own variables or top-level defaults. When that leaves a `{placeholder}` with no value, the UI prompts for it. In the CLI, supply `--var` or use `--prompt`; without those flags the CLI warns and can continue with literal placeholders. See [Missing runtime values](docs/UI_GUIDE.md#missing-runtime-values).

```yaml
pipelines:
  Bronze_to_Silver:
    tasks:
      dbt:
        type: cli
        env:
          DBT_CLIENT: "{practice}"
        command: dbt run --selector appointment_silver

pipeline_templates:
  ECW_Extract_test:
    tasks:
      extract:
        type: cli
        command: echo extract

pipeline_deployments:
  BENNETT_ETL_Flow:
    template: ECW_Extract_test
    variables:
      practice: BENNETT
    triggers_on_success: [Bronze_to_Silver]
```

When `BENNETT_ETL_Flow` succeeds, Piply runs `Bronze_to_Silver` with `DBT_CLIENT=BENNETT`. A deployment for PALOS would use the same target and set `practice: PALOS`.

## Common CLI

```bash
# Project
piply --version
piply init my-piply-project
piply validate --config piply-demo/piply.yaml
piply list --config piply-demo/piply.yaml
piply plan extract_flow --config piply-demo/piply.yaml     # dry run, nothing executes

# Running
piply run extract_flow --config piply-demo/piply.yaml --wait
piply run extract_flow --tenant acme --param batch=2026-05-26
piply run Bronze_to_Silver --var practice=BENNETT       # fill a {placeholder}
piply run Bronze_to_Silver --prompt                     # or be asked for them
piply tasks list extract_flow
piply tasks run extract_flow validate --tenant acme --param region=west
piply tasks retry <run_id> <task_id> --mode resume

# Inspecting
piply runs --limit 20
piply logs <run_id>
piply logs --follow --pipeline extract_flow                # live, colored
piply artifacts <run_id>
piply diagnostics
piply audit --since 7d

# Maintaining
piply backfill <run_id>                                    # replay a run's config
piply backfill nightly --from 2026-07-01 --to 2026-07-08   # fill a schedule window
piply prune --dry-run
piply prune --run-days 14 --max-runs 100
piply backup /backups                                      # safe while running
piply restore /backups/piply-20260804T074211Z.db           # stop the server first
piply migrate-db --to postgresql://piply@db:5432/piply     # SQLite -> PostgreSQL

# Serving
piply users create admin --role admin                       # switches auth on
piply users grant alice nightly=view,run
piply pause extract_flow
piply resume extract_flow
piply pause-run <run_id>                                    # active tasks finish
piply resume-run <run_id>
piply start --config piply-demo/piply.yaml --port 8080
piply stop --config piply-demo/piply.yaml
```

## Docs

**[Documentation index](docs/README.md)** — start here; everything below, organised.

**Writing config**

- [Pipelines](docs/PIPELINES.md): schedules, concurrency, retries, chaining, tags
- [Tasks](docs/TASKS.md): the six task types, dependencies, failure behaviour, artifacts
- [Variables](docs/VARIABLES.md): `{name}`, the four scopes, precedence, conditionals
- [Environment Variables](docs/ENVIRONMENT.md): `${NAME}`, `.env`, every `PIPLY_*` setting
- [Templates and Deployments](docs/TEMPLATES_AND_DEPLOYMENTS.md): one pipeline shape, many tenants
- [Sensors](docs/SENSORS.md): file, SQL, and API triggers, and how polling actually behaves
- [Notifications](docs/NOTIFICATIONS.md): Microsoft Teams and email alerts, and what happens when delivery fails
- [YAML Specification](docs/YAML_SPECIFICATION.md): the reference — every config key, with defaults
- [Execution Examples](docs/EXAMPLES.md): runnable patterns for each feature

**Operating Piply**

- [CLI Reference](docs/CLI.md): every command and flag
- [UI Guide](docs/UI_GUIDE.md): every page and what it answers
- [HTTP API Reference](docs/API.md): every route, with the permission it requires
- [Authentication](docs/AUTHENTICATION.md): accounts, roles, and pipeline permissions
- [Audit History](docs/AUDIT.md): recorded actions, attribution, filters, and retention
- [Runbook](docs/RUNBOOK.md): setup, upgrade, recovery, and incident procedures
- [Release Assessment](docs/RELEASE_CHECK.md): v0.3.4 evidence and publishing gates
- [Backlog](docs/BACKLOG.md): issues, priorities, and acceptance criteria
- [Metadata Store](docs/DATABASE.md): SQLite, PostgreSQL, migration, and the full schema
- [FAQ](docs/FAQ.md): the "why is it doing that" answers, and an error-message index
- [Migration Guide](docs/MIGRATION.md): moving onto pipeline templates and deployments
- [Changelog](CHANGELOG.md): what changed per release, and what to check before upgrading

**Understanding Piply**

- [Security](docs/SECURITY.md): trust model, what is protected, deployment checklist
- [Runtime Lifecycles](docs/LIFECYCLES.md): scheduler, pipeline, task, retry, recovery, retention
- [Technical Architecture](docs/architecture/technical_architecture.md): maintainer guide to the whole system
- [Roadmap](docs/ROADMAP.md): what is planned for the next releases
- [Future Features](docs/FUTURE_FEATURES.md): proposed ideas, ranked by value vs cost

## Metadata Store

Piply keeps its own runtime state in SQLite by default, with nothing to
configure. Point it at PostgreSQL when you would rather that state lived in a
managed database, for example so a container redeploy cannot lose it:

```bash
pip install "mr-piply[postgres]"
export PIPLY_DATABASE="postgresql://piply:secret@db.internal:5432/piply"
piply start
```

The schema is created and migrated automatically, and no pipeline configuration
changes. `piply backup` / `piply restore` remain SQLite-only; use `pg_dump` and
`pg_restore` for PostgreSQL. Run one Piply instance per database either way.

Already running on SQLite and want to keep the history? Stop the server and copy
it across:

```bash
piply migrate-db --to "postgresql://piply:secret@db.internal:5432/piply"
```

Run ids are preserved, so retry chains, downstream lineage, and accounts all
survive. Full details, the Docker volume setup for the SQLite default, and a
column-by-column schema reference are in [Metadata Store](docs/DATABASE.md).

## Monitoring

```bash
curl -s http://127.0.0.1:8000/metrics
curl -s http://127.0.0.1:8000/api/diagnostics
```

`/metrics` exposes run and task counts by status, running tasks, queue depth and
age, scheduler health, run durations, and per-sensor health in the Prometheus
text format. It accepts the API bearer token, so a scraper does not need UI
credentials.

## Roadmap

Proposed features, ranked by value against cost, with the reasoning for
what is deliberately *not* planned: [Future Features](docs/FUTURE_FEATURES.md).
