Metadata-Version: 2.4
Name: farsflow
Version: 0.1.2
Summary: Deterministic, semantic-safe Persian text preprocessing for AI, NLP, and search pipelines.
Author-email: Mahdi Hosseini <mhossza@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/mhhoss/farsflow
Project-URL: Documentation, https://github.com/mhhoss/farsflow#readme
Project-URL: Source, https://github.com/mhhoss/farsflow
Project-URL: Issues, https://github.com/mhhoss/farsflow/issues
Keywords: farsi,persian,preprocessing,nlp,text-cleaning,normalization,ai-pipelines,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# farsflow

farsflow is a lightweight **Persian text preprocessing** library focused on deterministic, semantic-safe normalization for modern AI and search pipelines.

[![PyPI version](https://img.shields.io/pypi/v/farsflow.svg)](https://pypi.org/project/farsflow/)
[![Python Versions](https://img.shields.io/pypi/pyversions/farsflow.svg)](https://pypi.org/project/farsflow/)

## ❓ WHY farsflow??

Persian text often contains inconsistent spacing, Arabic/Persian character variants, invisible Unicode formatting characters, Arabic diacritics, and broken ZWNJ usage that negatively affect search, embeddings, NLP pipelines, and LLM applications.

farsflow provides a minimal and deterministic preprocessing layer designed to clean text without aggressive or semantic-destructive transformations.

---

## 🚀 Features

- Deterministic and semantic-safe normalization
- Safe ZWNJ (Joiner) correction
- Whitespace and punctuation cleanup
- Line and paragraph structure preserved by default
- Unicode cleanup (Arabic/Persian variants, Bidi controls, invisible formatting characters)
- Configurable normalization options (digits, diacritics, hamza folding, line structure)
- Modular processors
- Zero dependencies

---

## 📦 Installation

```bash
pip install farsflow
```

## ✨ Quick Start

```python
import farsflow as ff

text = "سلام  دنیا!  این یك   تست است  که می نویسم  ۴۵۶"
cleaned = ff.clean(text)
print(cleaned)
```
Expected output:
```text
سلام دنیا! این یک تست است که می‌نویسم 456
```

---

## 🧩 Pipeline Components

farsflow ships with a set of modular, composable components:

- **Normalizer** — character normalization, Unicode cleanup, and optional diacritics removal
- **JoinerFixer** — fixes ZWNJ usage without over-correction
- **SpaceCleaner** — trims redundant whitespace and punctuation spacing
- **Pipeline** — orchestrates components in a deterministic order

You can customize the pipeline:

```python
from farsflow import (
    Pipeline,
    Normalizer,
    SpaceCleaner,
)

pipeline = Pipeline([
    Normalizer(),
    SpaceCleaner(),
    # JoinerFixer skipped to demonstrate modular behavior
])

text = "می  نويسم  که این   يك   متن  تستي است"
cleaned = pipeline(text)
print(cleaned)
```
Expected output:
```text
می نویسم که این یک متن تستی است
```

## 📄 Line and paragraph structure

By default farsflow **preserves line breaks**. Within each line, runs of spaces
and tabs collapse and the edges are trimmed; a run of blank lines becomes a
single blank line. This is what you want for RAG chunking, sentence splitting,
and anything downstream that relies on paragraph boundaries.

```python
import farsflow as ff

ff.clean("تيتر خبر\n\n\nبراساس گزارش ها، نرخ تورم كاهش يافته است.")
# 'تیتر خبر\n\nبراساس گزارش‌ها، نرخ تورم کاهش یافته است.'
```

To fold everything onto one line instead — the behavior of 0.1.1 and earlier —
set the policy on the `Normalizer`:

```python
from farsflow import Pipeline, Normalizer, JoinerFixer, SpaceCleaner

pipeline = Pipeline([
    Normalizer(newline_policy="collapse"),
    JoinerFixer(),
    SpaceCleaner(),
])
pipeline("تيتر خبر\n\nمتن خبر")   # 'تیتر خبر متن خبر'
```

The policy lives on `Normalizer` — the first stage — because line structure has
to be settled before any later rule runs. `JoinerFixer` deliberately never
matches across a line break, so if a later stage turned newlines into spaces, a
half-space rule would miss on the first pass and fire on the second, and the
output would not be stable.

---

## ⚠️ Lossy operations are opt-in

Two `Normalizer` options delete or merge information instead of just
reshaping it. Both default to `False` — `ff.clean()` never does this unless
you ask for it:

- **`remove_diacritics`** (default `False`) — strips Arabic diacritics
  (harakat, tanvin, shadda, sukun): `"کاملاً"` → `"کاملا"`. Diacritics are rare
  in everyday Persian text, but appear in poetry, Quranic quotations, and some
  literary or TTS text, where deleting them is a real loss. Turn this on for
  search/embedding pipelines that want maximum recall regardless.
- **`fold_hamza_carriers`** (default `False`) — folds `أ`/`إ` to `ا` and `ؤ`
  to `و`: `"مؤلف"` → `"مولف"`. These letters carry a real distinction in some
  Arabic loanwords, so folding them is a lossy, fuzzy-matching choice, not a
  safe default.

```python
from farsflow import Pipeline, Normalizer, JoinerFixer, SpaceCleaner

pipeline = Pipeline([
    Normalizer(remove_diacritics=True, fold_hamza_carriers=True),
    JoinerFixer(),
    SpaceCleaner(),
])
```

The `ي`/`ى`→`ی`, `ك`→`ک`, and `ة`/`ۀ`→`ه` substitutions stay on unconditionally
— those forms are not used distinctly in Persian orthography, so folding them
loses nothing.

---

## Design Principles

farsflow follows a few core principles:

- deterministic output
- semantic-safe transformations
- opt-in for potentially lossy operations
- zero dependencies
- modular architecture

---

🧪 Testing

```bash
pytest
# or:
pytest path/to/test_file.py
```

---

## 🗺 Scope

farsflow intentionally stays small: three processors, a pipeline, and a handful of documented options. (v0.1.2)

---

📄 License

MIT License — see [LICENSE](LICENSE).

---

🤝 Contributing

Contributions are welcome.  
Please open an issue or submit a pull request on GitHub.

📝 Changelog

See [CHANGELOG.md](CHANGELOG.md) for version history.
