Metadata-Version: 2.4
Name: indic-itn
Version: 0.2.5
Summary: Inverse Text Normalization (ITN) for Indic languages
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: mypy>=1.9.0; extra == "dev"
Requires-Dist: pre-commit>=3.6.0; extra == "dev"
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: ruff>=0.3.0; extra == "dev"
Dynamic: license-file

# indic-itn (v0.2.4)

Inverse Text Normalization (ITN) for Multilingual Indian Languages (Hindi, Tamil, Telugu, Kannada, etc.).

`indic-itn` converts spoken-form ASR (Automatic Speech Recognition) transcriptions into normalized written representations across **phone numbers, numbers, dates, times, currency, decimals, percentages, ordinals, and OTPs** while **strictly preserving all surrounding context words and punctuation**.

---

## Architectural Highlights & Guarantees

* **Zero Context Deletion**: Operating on span-based substitutions (`text[:start] + normalized_span + text[end:]`), ensuring surrounding text before and after numeric expressions is never lost or corrupted.
* **Language Plugin Architecture**: Decouples core normalization engine logic from language-specific vocabulary. Adding a new language (e.g. Malayalam `ml`) requires creating a language plugin directory without touching the core engine.
* **Specialized Phone Number Entity**: Phone numbers are processed as unformatted digit sequences (`9876543210`), supporting spoken native/English digits, repeated digit phrases (`double`/`triple`), optional country codes (`+91`), `plus`, and `zero`.
* **Deterministic Entity Priority Resolution**: Candidate spans are classified and resolved in strict priority order (URL > Email > Phone > OTP > Date > Time > Currency > Decimal > Percentage > Ordinal > Number) to prevent overlapping span corruption.
* **Code-Switching Support**: Seamlessly handles mixed Indic script and English spoken digit expressions (e.g. `"mera phone number hai nine eight seven six..."`).
* **100% Backward Compatibility**: Full support for legacy entry points (`HindiITN`, `TeluguITN`, `KannadaITN`, `TamilITN`, `IndicITN`, `Token`, `Entity`, `load_resource`).

---

## Architecture Overview

```
Raw ASR Spoken Text
      │
      ▼
indic_itn.core.tokenizer (IndicTokenizer)
      │
      ▼
indic_itn.core.entity_detector (Span candidate detection & priority resolution)
      ├── Phone Entity Handler
      ├── Date Entity Handler
      ├── Time Entity Handler
      ├── Currency Entity Handler
      ├── Decimal Entity Handler
      ├── Percentage Entity Handler
      ├── Ordinal Entity Handler
      ├── OTP Entity Handler
      └── General Number Entity Handler
      │
      ▼
indic_itn.languages.<lang> (Language Lexical Parser & Semantic Mapper)
      │
      ▼
indic_itn.entities.<entity> (Canonical Renderer)
      │
      ▼
indic_itn.core.span_replacer (Right-to-Left Safe Substring Replacer)
      │
      ▼
indic_itn.normalization.postprocess (Whitespace & Punctuation Cleanup)
      │
      ▼
Final Normalized Written Text
```

---

## Installation

```bash
pip install indic-itn
```

Or install locally in editable mode for development:

```bash
pip install -e ".[dev]"
```

---

## Usage

### 1. Functional API (`normalize_text`)

```python
from indic_itn import normalize_text

# Phone number with surrounding words
print(normalize_text("call me on nine eight seven six five four three two one zero tomorrow", language="hi"))
# Output: "call me on 9876543210 tomorrow"

# Currency
print(normalize_text("I have five hundred rupees in my account", language="hi"))
# Output: "I have 500 rupees in my account"

# Time & Date
print(normalize_text("meeting is at five thirty pm", language="hi"))
# Output: "meeting is at 5:30 pm"
```

### 2. Object-Oriented Orchestrators

```python
from indic_itn import HindiITN, TamilITN, TeluguITN, KannadaITN

# Hindi
hi = HindiITN()
print(hi.normalize("मेरा नंबर नौ आठ सात छह पाँच चार तीन दो एक शून्य है"))
# Output: "मेरा नंबर 9876543210 है"

# Tamil
ta = TamilITN()
print(ta.normalize("என் போன் நம்பர் ஒன்பது எட்டு ஏழு ஆறு ஐந்து நான்கு மூன்று இரண்டு ஒன்று பூஜ்யம்"))
# Output: "என் போன் நம்பர் 9876543210"

# Telugu
te = TeluguITN()
print(te.normalize("నా ఫోన్ నంబర్ తొమ్మిది ఎనిమిది ఏడు ఆరు ఐదు నాలుగు మూడు రెండు ఒకటి సున్నా ఉంది"))
# Output: "నా ఫోన్ నంబర్ 9876543210 ఉంది"

# Kannada
kn = KannadaITN()
print(kn.normalize("ನನ್ನ ಬಳಿ ಐದು ನೂರು ರೂಪಾಯಿ ಇದೆ"))
# Output: "ನನ್ನ ಬಳಿ ₹500 ಇದೆ"
```

### 3. Debug & Entity Metadata Mode

```python
from indic_itn import IndicITNEngine

engine = IndicITNEngine(lang="hi")
debug_info = engine.normalize("call nine eight seven six five four three two one zero at five pm", return_entities=True)

print(debug_info)
# Output:
# {
#   "original_text": "call nine eight seven six five four three two one zero at five pm",
#   "normalized_text": "call 9876543210 at 5:00 pm",
#   "detected_spans": [
#     {"start": 5, "end": 53, "original": "nine eight...", "normalized": "9876543210", "entity_type": "phone"},
#     {"start": 57, "end": 64, "original": "five pm", "normalized": "5:00 pm", "entity_type": "time"}
#   ]
# }
```

---

## How to Add a New Language

Adding support for a 5th Indian language (e.g. Malayalam `ml`) requires **zero modifications to the core engine**:

### Step 1: Create Resource Directory
Add JSON files in `src/indic_itn/resources/ml/`:
* `numbers.json` (digits, numbers, tens, hundreds, multipliers)
* `keywords.json` (script_range, script_digits, currency, time, phone, otp, decimal, percentage)
* `temporal.json` (months, date_words, weekdays)
* `ordinals.json` (ordinal words mapping)

### Step 2: Implement Language Plugin Class
Create `src/indic_itn/languages/malayalam/normalizer.py`:

```python
from indic_itn.languages.base import BaseLanguage

class MalayalamLanguage(BaseLanguage):
    def __init__(self) -> None:
        super().__init__(lang_code="ml")
```

### Step 3: Register Language
Register the language plugin dynamically or in default registry:

```python
from indic_itn import register_language, normalize_text
from indic_itn.languages.malayalam.normalizer import MalayalamLanguage

register_language("ml", MalayalamLanguage)

# Use immediately
print(normalize_text("spoken text in malayalam", language="ml"))
```

---

## How to Add a New Entity Type

1. Create a handler class in `src/indic_itn/entities/my_entity.py` inheriting from `BaseEntityHandler`.
2. Implement `entity_type` property and `parse(span, lang)` method.
3. Register handler in `IndicITNEngine.entity_handlers`.

---

## Quality Metrics & Benchmark Dataset

`indic-itn` includes an extensive 400-example benchmark test dataset (`tests/fixtures/benchmark_dataset.json`) covering 100 test samples each for Hindi, Tamil, Telugu, and Kannada across numbers, phone numbers, dates, times, currency, decimals, percentages, and mixed language contexts.

| Metric | Target | Benchmark Score |
| :--- | :--- | :--- |
| **Final Normalization Accuracy** | >= 98.0% | **100.00%** (400/400) |
| **Context Preservation Accuracy** | 100.0% | **100.00%** (400/400) |
| **Entity Detection Accuracy** | >= 98.0% | **100.00%** (400/400) |

### Running Benchmark Suite

```bash
pytest tests/benchmark/test_benchmark.py -s
```

---

## Testing & Quality Assurance

### Running Full Test Suite

```bash
pytest --cov=indic_itn --cov-report=term-missing
```

### Running Static Type Checker & Linter

```bash
mypy src
ruff check src tests
```
