Metadata-Version: 2.4
Name: rypipe
Version: 0.4.0
Classifier: Development Status :: 3 - Alpha
Requires-Dist: pyarrow>=15
Requires-Dist: rypipe[pandas,polars] ; extra == 'all'
Requires-Dist: pytest>=7 ; extra == 'dev'
Requires-Dist: maturin>=1.4,<2.0 ; extra == 'dev'
Requires-Dist: rypipe[all] ; extra == 'dev'
Requires-Dist: zensical ; extra == 'docs'
Requires-Dist: seoslug>=2.0.1 ; extra == 'docs'
Requires-Dist: pandas>=1.5 ; extra == 'pandas'
Requires-Dist: polars>=0.20 ; extra == 'polars'
Provides-Extra: all
Provides-Extra: dev
Provides-Extra: docs
Provides-Extra: pandas
Provides-Extra: polars
License-File: LICENSE
Summary: Format-agnostic data ingestion framework: Rust core with Python bindings. Adapters for XML, JSON, CSV, etc. are separate packages.
Author-email: Emiliano Gandini <emiliano.gandini@protonmail.com>
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/emiliano-go/rypipe
Project-URL: Repository, https://github.com/emiliano-go/rypipe

<p align="center">
  <img src="https://raw.githubusercontent.com/emiliano-go/rypipe/refs/heads/master/docs/overrides/icon.png" alt="rypipe" width="128"/>
</p>
<p align="center">
  <strong style="font-size: 2.5em;">rypipe</strong>
</p>

<p align="center">
  <strong>Format-agnostic columnar ingestion engine. Rust core, Python bindings.</strong>
</p>

<p align="center">
  Parse row-oriented byte streams into Apache Arrow tables with parallel
  scheduling, memory-bounded execution, query pushdown, and a chainable
  pipeline API. Format adapters live in separate packages.
</p>

<p align="center">
  <a href="https://www.python.org/downloads/">
    <img src="https://img.shields.io/badge/Python-3.10%2B-3776AB?logo=python&logoColor=white&style=for-the-badge" alt="Python">
  </a>
  <a href="https://www.rust-lang.org/">
    <img src="https://img.shields.io/badge/Rust-1.78%2B-000000?logo=rust&logoColor=white&style=for-the-badge" alt="Rust">
  </a>
  <a href="LICENSE">
    <img src="https://img.shields.io/badge/License-MIT-10AC84?style=for-the-badge" alt="License">
  </a>
  <a href="https://github.com/emiliano-go/rypipe/actions/workflows/test.yml">
    <img src="https://img.shields.io/github/actions/workflow/status/emiliano-go/rypipe/test.yml?branch=master&style=for-the-badge&logo=github&label=Tests" alt="Tests">
  </a>
  <a href="https://rypipe.emiliano-go.com/">
    <img src="https://img.shields.io/badge/Docs-rypipe.emiliano--go.com-8A2BE2?style=for-the-badge&logo=readthedocs" alt="Docs">
  </a>
  <a href="https://pypi.org/project/rypipe/">
    <img src="https://img.shields.io/badge/PyPI-rypipe-006DAD?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI">
  </a>
</p>

---

## What is rypipe

`rypipe` is a format- and source-agnostic ingestion framework that provides a
common execution runtime for turning arbitrary record-oriented data sources
into typed columnar data. It separates format-specific parsing from
format-agnostic execution, so the same engine can parse XML, JSON, CSV, HTML,
or any other row-oriented format once you provide a small adapter.

Add a new format by implementing two small traits: `Splitter` and
`RecordParser`.

> rypipe was originally developed as the ingestion engine for
> [crxml](https://github.com/emiliano-go/crxml), a Crystal Reports XML parser,
> and was later extracted and abstracted. The engine's design, performance
> characteristics, and API were
> shaped by real-world production use with crxml. We use crxml as the primary
> example throughout the documentation because it demonstrates the full power
> of the framework: complex nested schemas, large files, parallel processing,
> and advanced filtering.

## Quick start

Any parser you build on rypipe (or any adapter package you install)
gets this same API: a small Rust crate behind a clean, chainable Python
interface with fused filtering, parallel parsing, and bounded-memory
streaming. The quick start below shows crxml as a concrete example;
your format's adapter would look and behave the same.

```bash
pip install crxml
```

```python
import crxml
from crxml import CrystalXMLSource, CastTypes, FilterRows, col

source = CrystalXMLSource("report.xml", row_tag="Details")

# Simple read
table = source.to_arrow()

# Pipeline with stages
result = (
    source
    | CastTypes({"Amount": float})
    | FilterRows(field="Status", op="==", value="Active")
    | crxml.to_arrow()
)

# Expression predicates fuse into the Rust parse loop too
result = (
    source
    | FilterRows((col("Amount") > 100) & (col("Status") == "Active"))
    | crxml.to_arrow()
)

# Convert to DataFrame
df = source.to_pandas()

# Constant-memory streaming
for batch in source.iter_record_batches(memory="64MiB"):
    writer.write_batch(batch)
```

## Why rypipe

- **One runtime, many formats.** XML, JSON, CSV, HTML, TSV, and any future
  format share the same parallel scheduler, memory-bounded executor, and pushdown
  infrastructure. An adapter is two small traits, not a full engine.
  crxml (Crystal Reports XML) is the reference adapter that proved this model.

- **Measured performance.** On the documented 533 MB crxml workload, ~4.2 GB/s
  parallel and ~950 MB/s single-threaded (Ryzen 7 5800X). Arrow export moves
  string/dictionary buffers without copying; primitive arrays are copied.
  Predicate-first evaluation. Layout prediction via memcmp.

- **Correctness by construction.** Differential testing, fuzz targets, property
  tests, and a tier-ladder profiler.

- **Python-native ergonomics.** Chainable pipeline API with automatic fusion of
  rename/drop/cast/filter into the Rust parse loop. Streaming with bounded
  memory. Schema discovery. DataFrame and Parquet sinks.

## What rypipe is not

- **Not a query engine.** No joins, aggregations, window functions, or SQL.
- **Not a parser.** Each format needs an adapter package.
- **Not a data warehouse.** It ingests into Arrow; it does not store or serve.
- **Not pure Python.** Adapters are written in Rust for performance. Python
  users consume data through adapter APIs; they do not need to write Rust
  unless creating a new adapter.

## Features

- **Zero-copy friendly**: decoders emit borrowed strings; the engine copies only
  when necessary.
- **GIL-free parsing**: heavy work runs outside Python's GIL.
- **Parallel by default**: chunked parsing with `rayon` scales to many cores.
- **Memory bounded**: stream files larger than RAM with `iter_record_batches`
  and a configurable budget.
- **Pushdown filters**: rename, drop, type, and filter rows while parsing.
- **Expression API**: polars-style `col(...)` predicates that fuse into the
  parse loop, composable with `&`, `|`, `~`.
- **Observer hooks**: per-row callbacks from the engine, in Rust or Python.
- **Transparent decompression**: gzip, zstd, and lz4 inputs detected by magic
  bytes and decompressed automatically.
- **Pipeline API**: chainable rename/drop/cast/filter stages with automatic fusion.
- **Arrow native**: produces `RecordBatch` and exports via the C Data Interface.

## Crates

| Crate | Purpose |
|-------|---------|
| `rypipe-core` | Pure Rust engine: `Value`, `ExecutionPlan`, `TableBuilder`, `Pipeline`, parallel/bounded drivers, Arrow export |
| `rypipe-python` | PyO3 bindings for adapter packages; exposes the `rypipe` package |
| `rypipe-test` | Property-based testing helpers and fixtures for adapter development |

## Documentation

- [Tutorial](docs/tutorial/index.md): install, first read, pipeline, stages, sinks, streaming
- [Writing Adapters](docs/building-adapters/index.md): add a new format
- [Architecture](docs/architecture/index.md): how the engine works internally
- [Advanced](docs/advanced/index.md): fusion, execution modes, memory, parallelism
- [Python API](docs/reference/python-api.md): full API reference
- [Rust API](docs/reference/rust-api.md): full API reference

## Building

```bash
# Rust only
cargo build --workspace --release

# Python extension
maturin develop --release
```

## Testing

```bash
# Rust
cargo test --workspace --all-features

# Python
pip install -e ".[dev]"
pytest crates/rypipe-python/tests/
```

## License

MIT

---

<p align="center">
  <img src="https://raw.githubusercontent.com/emiliano-go/rypipe/refs/heads/master/docs/overrides/badge.png" alt="rypipe badge" width="400"/>
</p>

