Metadata-Version: 2.4
Name: cereyan
Version: 3.3.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
License-File: LICENSE
Summary: Minimal, local-first orchestrator for Python data pipelines with a Rust core
Keywords: orchestration,workflow,data-pipeline,etl,scheduler,dag,local-first,rust
Author: the cereyan authors
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://sercanatalik.github.io/cereyan/changelog/
Project-URL: Documentation, https://sercanatalik.github.io/cereyan/
Project-URL: Homepage, https://sercanatalik.github.io/cereyan/
Project-URL: Issues, https://github.com/sercanatalik/cereyan/issues
Project-URL: Repository, https://github.com/sercanatalik/cereyan

# Cereyan

[![PyPI](https://img.shields.io/pypi/v/cereyan.svg)](https://pypi.org/project/cereyan/)
[![CI](https://github.com/sercanatalik/cereyan/actions/workflows/ci.yml/badge.svg)](https://github.com/sercanatalik/cereyan/actions/workflows/ci.yml)
[![Python](https://img.shields.io/pypi/pyversions/cereyan.svg)](https://pypi.org/project/cereyan/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/sercanatalik/cereyan/blob/main/LICENSE)
[![Status](https://img.shields.io/badge/status-production%20ready-brightgreen.svg)](#status)

Cereyan is a minimal, local-first orchestrator for Python data pipelines. A Rust core (SQLite store, state machine, scheduler, HTTP server) sits behind a thin layer of Python decorators, and the whole thing ships as one wheel with no runtime dependencies. Decorate functions as flows and tasks, run them as plain scripts, and every run is recorded locally; `cereyan serve` adds live monitoring, one queue with processors you size while it runs, schedules (including continuous loops), retries, backfills, rules, and a built-in MCP server for agents on top of the same store. When one machine is not enough, `cereyan worker` on another machine adds its processors to the same queue.

```bash
pip install cereyan
```

```python
# pipeline.py
from datetime import date
from cereyan import flow, task, get_run_logger

@task
def extract(day: date) -> list[int]:
    get_run_logger().info("extracting %s", day)
    return [1, 2, 3]

@task
def load(rows: list[int]) -> int:
    return sum(rows)

@flow(run_name="etl-{day}")
def etl(day: date = date(2026, 9, 6)) -> int:
    return load(extract(day))

if __name__ == "__main__":
    print(etl())
```

```bash
python pipeline.py                                    # records a run in ~/.cereyan/db.sqlite
cereyan run pipeline.py:etl --param day=2026-01-02    # same, with parameters and a summary
cereyan serve . --token $TOKEN                        # API, UI, scheduler, engines at http://127.0.0.1:4200
cereyan worker . --host https://cereyan.internal      # on another machine: its processors join the queue
```

[![The cereyan dashboard beside the grouped sidebar: counts by state over a histogram, the Flows card with each flow's last ten runs and a failed run's logs open, then Upcoming, Needs attention, and Running now](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/dashboard.png)](https://sercanatalik.github.io/cereyan/get-started/tour/#dashboard)

<p align="center"><sub>The dashboard after <code>cereyan serve</code> &mdash; <a href="https://sercanatalik.github.io/cereyan/get-started/tour/">take the full tour</a></sub></p>

- **Offline first.** A script records runs into a local SQLite file; nothing else needs to run.
- **One process to serve.** The API, the web UI, the scheduler, one queue, a warm pool of engine processes, rules, and MCP, in one `cereyan serve`. The Queue page shows every run in line and why it waits, and lets you add or remove processors while the server runs.
- **Workers when you need them.** `cereyan worker` runs on another machine with its own checkout and connects out to the server: when the server's processors are full, runs spill over to workers whose code matches. The store, results and secrets stay on the server, every hand-off carries a lease, the Workers tab shows each machine's status, host and schedule, and each worker serves a status page of its own.
- **Data-pipeline semantics.** Targets make reruns idempotent, backfills cover date ranges or listed values, resources are named semaphores that can be keyed by parameter, flows chain and fan in by key, unique keys stop duplicate runs, continuous schedules run a flow again a set time after each run ends, and rules react to events or to their absence.
- **Durable runs.** Every completed task is checkpointed, so a crash rerun or a retry from failure resumes after the last finished task; a run can sleep, wait for an event, a target, or a message sent through the API, or snooze without holding an engine; and a task keeps notes that survive its retries.

## Screenshots

Every page below is described in the [tour](https://sercanatalik.github.io/cereyan/get-started/tour/); click a shot to jump to it.

| | |
|:--:|:--:|
| [![Runs list beside the sidebar and its scope picker, in collapsible sections whose headers roll up their runs' states, over rows with popover filters, the host that ran each run, and a task-state bar per run](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/runs.png)](https://sercanatalik.github.io/cereyan/get-started/tour/#runs) | [![Run detail: a summary sentence, the facts grid, the tasks rail with states in words, and the log lines with level chips, search and Follow](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/run-detail.png)](https://sercanatalik.github.io/cereyan/get-started/tour/#run-detail) |
| **[Runs](https://sercanatalik.github.io/cereyan/get-started/tour/#runs)** &mdash; scope by project and group, filter, act in bulk | **[Run detail](https://sercanatalik.github.io/cereyan/get-started/tour/#run-detail)** &mdash; logs, tasks and retries |
| [![Flows beside the sidebar, banded by group under the Failing, Scheduled, Waiting for input and Never run chips, a scheduled flow's row menu open on Skip next run, Skip runs and Reschedule, over rows with schedules in words and the last ten runs as squares](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/flows.png)](https://sercanatalik.github.io/cereyan/get-started/tour/#flows) | [![Task timeline: a run's task runs as bars on a time axis, with a dependency view](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/run-timeline.png)](https://sercanatalik.github.io/cereyan/get-started/tour/#run-detail) |
| **[Flows](https://sercanatalik.github.io/cereyan/get-started/tour/#flows)** &mdash; groups, schedules, skips, dependencies, history | **[Timeline](https://sercanatalik.github.io/cereyan/get-started/tour/#run-detail)** &mdash; the task graph of a run |
| [![Events page with the live feed and a JSON payload viewer](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/events.png)](https://sercanatalik.github.io/cereyan/get-started/tour/#events) | [![Rules page listing when/do rules and their firing counts](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/rules.png)](https://sercanatalik.github.io/cereyan/get-started/tour/#rules) |
| **[Events](https://sercanatalik.github.io/cereyan/get-started/tour/#events)** &mdash; what happened, as it happens | **[Rules](https://sercanatalik.github.io/cereyan/get-started/tour/#rules)** &mdash; react to events, or to their absence |
| [![The Queue page: the Capacity card with processor slots and add and remove buttons, then Up next with the runs ready to start and starting later](https://raw.githubusercontent.com/sercanatalik/cereyan/main/docs/images/queue.png)](https://sercanatalik.github.io/cereyan/concepts/engines-and-home/#processors-and-the-queue) | |
| **[Queue](https://sercanatalik.github.io/cereyan/concepts/engines-and-home/#processors-and-the-queue)** &mdash; one line, processors, and workers | |

The UI follows your system theme; the documentation shows the [dark variants](https://sercanatalik.github.io/cereyan/get-started/tour/) too.

## Documentation

The site is at https://sercanatalik.github.io/cereyan/ (built from `docs/` with `just docs`):

- [Quickstart](https://sercanatalik.github.io/cereyan/get-started/quickstart/): from install to a scheduled, retried, backfilled pipeline in ten minutes.
- [Concepts](https://sercanatalik.github.io/cereyan/concepts/app-and-projects/): the model behind flows, runs, states, schedules, targets, resources, backfills, dependencies, events, rules, artifacts, and variables.
- [Guides](https://sercanatalik.github.io/cereyan/guides/retries-timeouts-crashes/): one goal per page, from retries to running the server as a service, [running across machines](https://sercanatalik.github.io/cereyan/guides/run-across-machines/), and using cereyan with an AI agent.
- [Reference](https://sercanatalik.github.io/cereyan/reference/python-api/): the Python API, CLI, HTTP API, MCP tools, events, states, and configuration.
- [Design and limitations](https://sercanatalik.github.io/cereyan/design/limitations/): what cereyan does not do, and why.
- For agents: [llms.txt](https://sercanatalik.github.io/cereyan/llms.txt) and [llms-full.txt](https://sercanatalik.github.io/cereyan/llms-full.txt).
- [Changelog](https://sercanatalik.github.io/cereyan/changelog/).

## Status

Production ready, and published to [PyPI](https://pypi.org/project/cereyan/). The pieces
described here work and are tested on macOS, Linux and Windows, and the interfaces —
names, signatures, defaults, routes, event payloads and the database schema — are settled.
Windows lacks the Unix socket and engine niceness, and cannot interrupt a flow blocked in
most I/O; [Install](https://sercanatalik.github.io/cereyan/get-started/install/#windows)
lists what that changes.
Read the [changelog](https://sercanatalik.github.io/cereyan/changelog/) before upgrading: 3.0 changes
the engine protocol (servers and workers of 2.x and 3.0 do not mix), migrates the store,
and starts a server with one processor instead of one per CPU.
Issues and questions are welcome.

## Development

Rust stable, Python 3.11 or newer with uv, Node 22, and just. `just ui` builds the UI, `just dev` builds the extension in place, `just test` and `just lint` run everything, `just demo` serves the examples. Details, the repository layout, and the release checklist are in the [contributing guide](https://sercanatalik.github.io/cereyan/contributing/). Design decisions are on the [Design](https://sercanatalik.github.io/cereyan/design/architecture/) pages, and changes not yet released are under Unreleased in `CHANGELOG.md`.

## Credits

Cereyan owes its shape to two projects that came first.

[**Luigi**](https://github.com/spotify/luigi) contributed the idea the data-pipeline semantics rest on: a task declares the target it produces, and work is skipped when that target already exists. `Target` and `LocalTarget` keep Luigi's names because they are Luigi's concept, and idempotent reruns and backfills follow from it.

[**Prefect**](https://github.com/PrefectHQ/prefect) contributed the authoring model — flows and tasks as decorated functions, runs carrying explicit states, a server watching them — and the look of the UI. Six UI components were adapted from Prefect's, rewritten in React from the Vue originals


Neither is a dependency, and cereyan deliberately does far less than either: one queue, one process, one wheel, no database to run, and workers that are plain processes on machines you already have rather than a platform of pools and images. If you need what they do, use them. [Migrating from Prefect or Luigi](https://sercanatalik.github.io/cereyan/guides/migrate/) says what carries over and what does not.

## License

MIT, see `LICENSE`.

