Metadata-Version: 2.4
Name: xileh
Version: 0.3.2
Summary: A lightweight, composable pipeline abstraction built around a hierarchical data container.
Author-email: Matthias Dold <matthias.dold@gmx.net>
License: MIT
Project-URL: Homepage, https://github.com/matthiasdold/xileh
Project-URL: Documentation, https://matthiasdold.github.io/xileh/
Project-URL: Repository, https://github.com/matthiasdold/xileh
Project-URL: Issues, https://github.com/matthiasdold/xileh/issues
Project-URL: Changelog, https://github.com/matthiasdold/xileh/releases
Keywords: pipeline,data-container,workflow
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pyyaml
Requires-Dist: tqdm
Requires-Dist: rich
Requires-Dist: tomli-w
Provides-Extra: lite
Provides-Extra: pandas
Requires-Dist: pandas; extra == "pandas"
Requires-Dist: pyarrow; extra == "pandas"
Provides-Extra: polars
Requires-Dist: polars; extra == "polars"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: ipython; extra == "dev"
Provides-Extra: docs
Requires-Dist: quartodoc; extra == "docs"
Requires-Dist: nbformat; extra == "docs"
Requires-Dist: nbclient; extra == "docs"
Requires-Dist: jupyter-cache; extra == "docs"
Requires-Dist: ipykernel; extra == "docs"
Requires-Dist: griffe<2.0.0; extra == "docs"
Provides-Extra: build
Requires-Dist: build; extra == "build"
Requires-Dist: twine; extra == "build"
Provides-Extra: examples
Requires-Dist: fire; extra == "examples"
Provides-Extra: all
Requires-Dist: xileh[build,dev,docs,examples,pandas,polars]; extra == "all"
Dynamic: license-file

# xileh

![CI](https://github.com/matthiasdold/xileh/workflows/CI%20Testing/badge.svg?branch=main)
[![codecov](https://codecov.io/gh/matthiasdold/xileh/branch/main/graph/badge.svg)](https://codecov.io/gh/matthiasdold/xileh)
![Tests](https://img.shields.io/badge/tests-46%20passed-brightgreen)
![Python](https://img.shields.io/badge/python-3.11+-blue)

A lightweight, composable pipeline abstraction built around a hierarchical
data container.

![xileh_gif](./assets/xileh_2022-05-02_16-35_small.gif)

## Overview

`xileh` abstracts complex processing pipelines, motivated by `sklearn.Pipeline`
but with data capacities going beyond a feature matrix `X` and target `y`. It
is built from two components:

- **`xData`** — a hierarchical container holding *data*, a *header*
  (description / control flags), and *meta* information. It can nest other
  `xData` objects to pass multiple data entities through a pipeline.
- **`xPipeline`** — an ordered set of plain Python functions, each taking and
  returning an `xData` object.

![schematic](assets/schematic.png)

### Why build something new?

Placing libraries on two axes — *built-in functionality* and
*specificity / restrictions* — `xileh` deliberately aims for minimal built-in
functionality and maximal freedom of customization.

![func_spec_landscape](./assets/func_spec_landscape.svg)

`sklearn.Pipeline` is great for ML pipelines but specific about data shape and
the `fit`/`transform` contract. Config-driven tools like `hydra` provide a
single source of truth but impose a format that can hinder rapid prototyping.
`xileh` instead:

- imposes as few restrictions on your workflow as possible (arbitrary data
  objects and plain Python functions),
- integrates easily with a function-based workflow during development,
- provides a single source of truth for the processing,
- enables reuse of whole pipelines by strongly motivating composition.

## Installation

```bash
pip install xileh
```

The bare install is **lite** (numpy only). Optional data backends are
available as extras:

```bash
pip install xileh[pandas]   # pandas DataFrame/Series save & load (parquet)
pip install xileh[polars]   # polars DataFrame/Series save & load (parquet)
```

Saving or loading a container that holds a backend-specific type without the
corresponding extra installed raises an error pointing at the extra to install.

Or from source:

```bash
git clone git@github.com:matthiasdold/xileh.git
cd xileh
pip install -e .
```

`xileh` targets Python 3.11+.

## Quick example

```python
from xileh import xData, xPipeline


def add_ones(pdata, name="ones", size=3):
    pdata.add([1] * size, name=name)
    return pdata


pl = xPipeline("demo")
pl.add_steps(("add_ones", add_ones))

root = xData([], name="root")
pl.eval(root)
```

> The container was renamed from `xPData` to `xData`; `xPData` remains
> available as a backwards-compatible alias.

## Documentation

Full documentation, including a quick-start guide, worked examples, and the API
reference, is available at
**[matthiasdold.github.io/xileh](https://matthiasdold.github.io/xileh/)**.

## License

[MIT](LICENSE)
