Metadata-Version: 2.5
Name: rowfuse
Version: 0.1.1
Summary: Stream CSV merges with deterministic row hashes and disk-backed deduplication.
Project-URL: Homepage, https://github.com/pyaidev/rowfuse
Project-URL: Documentation, https://github.com/pyaidev/rowfuse#readme
Project-URL: Repository, https://github.com/pyaidev/rowfuse
Project-URL: Issues, https://github.com/pyaidev/rowfuse/issues
Author: Nurmuhammad Mashrapov
Keywords: csv,deduplication,hashing,merge,parquet,streaming
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.11
Provides-Extra: parquet
Requires-Dist: pyarrow>=21.0.0; extra == 'parquet'
Description-Content-Type: text/markdown

# RowFuse

Stream large CSV merges, generate deterministic row hashes, and remove duplicates with a disk-backed index.

Version **0.1.1**. [PyPI](https://pypi.org/project/rowfuse/) · [Source code](https://github.com/pyaidev/rowfuse) · [Issues and feature requests](https://github.com/pyaidev/rowfuse/issues)

## Installation

Requires Python 3.11 or newer:

```bash
pip install rowfuse

# Include Parquet output support:
pip install 'rowfuse[parquet]'
```

CSV support uses the Python standard library. PyArrow is imported only when Parquet output is requested.

## Quick start

```python
from rowfuse import merge

result = merge(
    ["data/january.csv", "data/february.csv"],
    output="merged.parquet",
    hash_algorithm="sha256",
    hash_by="all",
    hash_column="_row_hash",
    deduplicate=True,
    keep="first",
)

print(result.files_merged)
print(result.rows_written)
print(result.duplicates_removed)
print(result.file_hash)
result.write_report("report.json")
```

Use `output="merged.csv"` for CSV output. Sources can be an ordered list of paths, a glob such as `"data/*.csv"`, or a list containing multiple glob patterns.

## Merge behavior

- Records are concatenated in input order. Each glob expands in sorted path order.
- Each physical input file is read once, even if it appears more than once or is referenced through a symlink or hardlink.
- The output contains one header. Columns follow the first file's order, and subsequent files are aligned by column name.
- `schema_mode="strict"` is the default. Every input must contain the same set of column names; their order may differ.
- `schema_mode="union"` appends new columns in first-seen order. Missing values become empty fields in CSV and `null` in Parquet.
- CSV values remain strings, including in Parquet output. Values such as `00123`, `NA`, and `1.00` retain their original text.
- UTF-8 and UTF-8 BOM are supported by default. Set `encoding` for another input encoding and `delimiter` for another separator. Output CSV files use UTF-8.
- Empty physical records are skipped and counted in `blank_rows_skipped`. A quoted empty string is retained as a record.
- Duplicate or empty column names, incorrect field counts, malformed quoted records, and decoding errors raise an error with input-file context. Quarantining invalid records is planned for a future version.

```python
result = merge(
    "data/*.csv",
    output="merged.csv",
    schema_mode="union",
    delimiter=";",
    output_delimiter=",",
)
```

## Hashing and deduplication

Set `hash_column="_row_hash"` to include a row digest in the output. Omit this option to preserve the data columns alone. The checksum of the complete output file is always returned as `result.file_hash`.

```python
from rowfuse import hash_file, hash_row, merge

digest = hash_row({"id": "001", "name": "Alice"})
checksum = hash_file("merged.csv")

result = merge(
    "data/*.csv",
    output="users.csv",
    hash_by=["user_id"],
    hash_column="_row_hash",
    deduplicate=True,
)
```

- SHA-256 is the default algorithm. SHA-512 and BLAKE2b are also supported.
- `hash_by="all"` includes every data column. Explicitly selected hash columns must exist in every input, including in `union` mode.
- The hash format is `rowfuse-row-v1`. Column-name/value pairs are sorted by name, encoded as compact UTF-8 JSON, and prefixed with `rowfuse:row:v1` followed by a NUL byte.
- Values are hashed exactly as read, without trimming, case folding, numeric conversion, or Unicode normalization. Column order does not affect the hash; column names and values do.
- A missing value (`None`) and an explicit empty string (`""`) have different hashes. Both appear empty in CSV output; Parquet preserves the distinction. Hashes using `hash_by="all"` depend on the complete merged schema.
- `deduplicate=False` is the default. Enable it to retain the first record for each distinct combination of selected values. Only `keep="first"` is supported.
- The disk index compares complete selected values as well as their digests. A hash collision does not cause distinct records to be removed.

## Large files and output handling

CSV output is written one record at a time. Parquet buffers are flushed after `batch_size` rows (50,000 by default) or `batch_bytes` of UTF-8 cell data (8 MiB by default). These settings control buffers rather than total process memory: wide records, Python objects, and Arrow allocations contribute additional overhead.

Deduplication uses a temporary SQLite file with an approximately 8 MiB cache target. The index stores complete selected values, so its disk usage can be substantial. Choose its location with `temp_dir`. Temporary index files are cleaned up after normal completion or a handled Python exception.

```python
merge(
    "data/*.csv",
    output="merged.parquet",
    deduplicate=True,
    batch_size=10_000,
    batch_bytes=4 * 1024 * 1024,
    temp_dir="/path/to/scratch",
)
```

Output is first written to a temporary file in the destination directory and published atomically after completion. Existing files are protected by the default `overwrite=False`. Even with `overwrite=True`, a failed merge preserves the previous output. An output file matched by an input glob is excluded; explicitly listing it as an input is rejected.

Input files must remain unchanged during a merge. File metadata changes are checked, but RowFuse does not lock files against external writers. A forced termination or power loss may leave temporary files behind. This release supports local CSV inputs and does not resume interrupted jobs. CSV field lengths follow Python's `csv.field_size_limit()` setting.

## Command line

```bash
rowfuse merge 'data/*.csv' \
    --output merged.parquet \
    --hash-column _row_hash \
    --deduplicate \
    --report report.json

# Deduplicate using selected columns:
python -m rowfuse merge a.csv b.csv \
    -o merged.csv --hash-by user_id --hash-by order_id --deduplicate
```

The CLI prints a JSON result. `--report` also writes a new JSON report file. A report-writing error does not undo the merge output that was completed before it.

## Development

```bash
git clone https://github.com/pyaidev/rowfuse.git
cd rowfuse
uv sync --extra parquet
uv run --extra parquet pytest
uv run ruff check .
uv run ruff format --check .
uv build
```

CSV tests also run without PyArrow installed. Parquet tests are skipped in that environment.

GitHub Actions tests Python 3.11 through 3.14. Pushing a version tag starting with `v` triggers the publishing workflow. See the [publishing guide](https://github.com/pyaidev/rowfuse/blob/main/PUBLISHING.md) for trusted publisher configuration and release steps.

## Measured example

A local macOS run using Python 3.12.10 processed two synthetic CSV files containing 300,000 records and 83.7 MB of input data, with hashing and disk-backed deduplication enabled. Both output formats retained 225,000 records and removed 75,000 duplicates.

| Output format | Merge time | Peak process RSS |
| --- | --- | --- |
| CSV | 8.048 seconds | 32.2 MB |
| Parquet | 6.681 seconds | 79.4 MB |

These are single-run measurements with repetitive 256-byte text payloads. Results depend on hardware, storage, column widths, and values. RSS covers the whole process; elapsed time covers merging and the output checksum after input generation.

```bash
uv run --extra parquet python benchmarks/merge_memory.py --rows-per-file 150000 --format csv
uv run --extra parquet python benchmarks/merge_memory.py --rows-per-file 150000 --format parquet
```

The exact measurements are recorded in [benchmarks/results.json](https://github.com/pyaidev/rowfuse/blob/main/benchmarks/results.json).

Planned additions include `inspect()`, `validate()`, rejected-record output, and checkpoints for completed files. These features are not part of the current API.
