Metadata-Version: 2.1
Name: urduflow
Version: 0.2.0
Summary: Urdu-aware primitives for LLM/RAG pipelines: script normalization, morphology-aware tokenization, Urdu-punctuation chunking, and code-switch routing.
Author-email: Aabideen <aabideenbofficial@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/GurusGeek/urduflow
Project-URL: Repository, https://github.com/GurusGeek/urduflow
Project-URL: Issues, https://github.com/GurusGeek/urduflow/issues
Keywords: urdu,nlp,rag,llm,tokenization,code-switching,pakistan
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Urdu
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev
License-File: LICENSE

# urduflow

**Urdu-aware primitives for LLM/RAG pipelines.**

Most LLM infrastructure assumes English punctuation, Latin script, and monolingual queries. Pakistani users often mix Urdu script, Roman Urdu, English technical terms, and numbers in the same message.

`urduflow` helps with:

1. **Script normalization** — Arabic/Urdu Unicode variants collapse to stable canonical forms before embedding.
2. **Morphology-aware tokenization** — conservative splitting of common Urdu suffixes and postpositions.
3. **Urdu smart chunking** — respects Urdu punctuation (`۔ ؟ ؛`), poetry/verse lines, and closing punctuation.
4. **Code-switch routing** — labels Urdu, Roman Urdu, English, numbers, and mixed spans.
5. **Framework adapters** — optional LangChain and LlamaIndex-compatible splitters.

Zero runtime dependencies. Python 3.9+.

```bash
pip install urduflow
```

Or install from GitHub:

```bash
pip install git+https://github.com/GurusGeek/urduflow.git
```

## Why this exists

Common RAG pipeline failures with Urdu text:

- `كتاب` and `کتاب` can embed differently even though they represent the same word in different Unicode conventions.
- Western splitters often ignore Urdu sentence endings (`۔`, `؟`, `؛`).
- Poetry and verse can get flattened into prose chunks.
- Queries like `mera code run nahi ho raha can you fix the Python error` are code-switched, not purely English or Urdu script.

`urduflow` is a small, auditable toolkit for those gaps. It is **not** a full Urdu NLP stack. For lemmatization, POS tagging, or broader NLP tasks, see projects like [`urduhack`](https://pypi.org/project/urduhack/), [`LughaatNLP`](https://pypi.org/project/LughaatNLP/), and [`urdu-nlp`](https://pypi.org/project/urdu-nlp/). This package focuses on **Urdu-aware middleware for LLM/RAG pipelines**: normalization, chunking, code-switch routing, optional framework adapters, and seed benchmarks.

## Quick start

```python
from urduflow import UrduNormalizer, UrduSmartSplitter, CodeSwitchRouter

normalizer = UrduNormalizer()
print(normalizer.normalize("كتاب"))
# کتاب

splitter = UrduSmartSplitter(chunk_size=80, chunk_overlap=10)
chunks = splitter.split_text("یہ پہلا جملہ ہے۔ یہ دوسرا جملہ ہے۔")
print(chunks)

router = CodeSwitchRouter()
print(router.route("mera code run nahi ho raha can you fix the Python error"))
# [{'span': 'roman_urdu', 'text': 'mera'},
#  {'span': 'english', 'text': 'code run'},
#  {'span': 'roman_urdu', 'text': 'nahi ho raha'},
#  {'span': 'english', 'text': 'can you fix the Python error'}]
```

## Modules

### `UrduNormalizer`

```python
from urduflow import normalize, canonical, similarity_key

normalize("كتاب")       # کتاب
canonical("كَتَابٍ")    # کتاب
similarity_key("  كتاب ") # کتاب
```

Normalizes:

- Arabic kaf `ك` → Urdu kaf `ک`
- Arabic yeh `ي` / alif maqsura `ى` → Urdu yeh `ی`
- Arabic digits `١٢٣` → Urdu digits `۱۲۳`
- zero-width characters removed
- canonical form strips harakat/diacritics

### `UrduMorphTokenizer`

```python
from urduflow import UrduMorphTokenizer

tok = UrduMorphTokenizer()
print(tok.tokenize("کتابوں"))
# ['کتاب', 'وں']
```

Conservative suffix handling. It is not a full morphological analyzer; it provides safer building blocks for LLM/RAG preprocessing.

### `UrduSmartSplitter`

```python
from urduflow import UrduSmartSplitter

s = UrduSmartSplitter(chunk_size=60, chunk_overlap=10, respect_verse=True)
print(s.split_text("پہلی سطر\nدوسری سطر\nتیسری سطر"))
# ['پہلی سطر', 'دوسری سطر', 'تیسری سطر']
```

Features:

- Urdu sentence endings: `۔`, `؟`, `؛`
- ASCII endings: `.`, `!`, `?`
- decimal safety (`3.14` not split)
- whole-word overlap
- no orphaned closing punctuation
- verse/newline preservation
- metadata helper: `split_with_metadata()`

### `CodeSwitchRouter`

```python
from urduflow import CodeSwitchRouter

r = CodeSwitchRouter()
r.dominant("mera code nahi chal raha")
# roman_urdu

r.needs_bilingual_model("mera code run nahi ho raha can you fix")
# True
```

Uses a lexicon for Roman Urdu detection. Unknown Latin tokens are treated as English.

### LangChain-compatible adapter

```python
from urduflow.integrations import UrduFlowTextSplitter

splitter = UrduFlowTextSplitter(chunk_size=500, chunk_overlap=50)
docs = splitter.create_documents(["یہ پہلا جملہ ہے۔ یہ دوسرا جملہ ہے۔"])
```

LangChain is optional. If it is installed, returned documents use LangChain's `Document`; otherwise `create_documents()` returns plain dictionaries with `page_content` and `metadata`.

### `RomanUrduTransliterator`

```python
from urduflow import RomanUrduTransliterator

RomanUrduTransliterator().transliterate("mera kitab")
# میرا کتاب
```

Lexicon-based transliteration. Unknown words pass through unchanged.

## Benchmarks

Seed benchmarks live in `benchmarks/`. They are small, reproducible checks — not large-scale or embedding-model evaluations.

```bash
python benchmarks/eval_codeswitch.py benchmarks/code_switch_200.jsonl
python benchmarks/eval_chunking.py
python benchmarks/eval_pdf_rag.py
python benchmarks/eval_normalization_retrieval.py
```

Local results on the current seed datasets:

- Code-switch route sequence: **302/326 (92.64%)** on `code_switch_200.jsonl`
- Urdu PDF-RAG seed retrieval: **urduflow 20/20 vs naive ASCII splitter 17/20**
- Normalization retrieval seed: **normalized 8/8 vs raw 7/8**
- Urdu boundary detection: urduflow passes both seed cases; naive ASCII punctuation splitter fails both

More research notes and raw logs are in [`research/`](research/).

## CLI

```bash
urduflow normalize "كتاب"
urduflow canonical "كَتَابٍ"
urduflow split --chunk-size 80 "یہ پہلا جملہ ہے۔ یہ دوسرا جملہ ہے۔"
urduflow route "mera code nahi chal raha can you fix"
urduflow transliterate "mera masla yeh hai"
urduflow benchmark codeswitch --dataset benchmarks/code_switch_200.jsonl
```

## Examples and notebooks

See `examples/` for executable demos and `notebooks/` for minimal LangChain/LlamaIndex walkthroughs.

## Development

```bash
pip install -e ".[dev]"
python -m pytest tests/ -q
```

## License

MIT
