Metadata-Version: 2.4
Name: dv_normalizer
Version: 1.0.0
Summary: Written-to-spoken Dhivehi (Thaana) text normalizer: numbers, dates, money, abbreviations and literary sentence endings to spoken form, for TTS and beyond
Project-URL: Homepage, https://github.com/alakxender/dv-text
Author-email: Alakxender <alakxender@gmail.com>
License: MIT License
        
        Copyright (c) 2024 
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: dhivehi,maldives,maldivian,nlp,number-to-words,speech,text-normalization,thaana,tts,written-to-spoken
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Requires-Dist: python-dateutil>=2.8
Requires-Dist: pyyaml>=6.0
Requires-Dist: regex>=2023.6.3
Provides-Extra: dev
Requires-Dist: huggingface-hub>=0.20; extra == 'dev'
Requires-Dist: hypothesis>=6; extra == 'dev'
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest-cov>=4; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: webui
Requires-Dist: fastapi>=0.110; extra == 'webui'
Requires-Dist: jinja2>=3.1; extra == 'webui'
Requires-Dist: python-multipart>=0.0.9; extra == 'webui'
Requires-Dist: uvicorn[standard]>=0.27; extra == 'webui'
Description-Content-Type: text/markdown

# dv_normalizer — written Dhivehi to spoken Dhivehi

Converts written Dhivehi (Thaana script) into the spoken form: numbers,
dates, times, money, percentages, abbreviations, URLs and the literary
sentence endings of news prose (`…އެވެ`) become the words a Dhivehi speaker
would actually say. Built as the text-normalization frontend for Dhivehi
text-to-speech, and usable on its own.

ލިޔެފައިވާ ދިވެހި ބަސް ވާހަކަދައްކާ ދިވެހި ބަހަށް ބަދަލުކުރާ ނިޒާމު

**What it's for:**

- Dhivehi text normalization — written to spoken Dhivehi conversion of
  numbers, dates, money, percentages, and more, in the Thaana script
- Dhivehi number to words conversion (cardinals, ordinals, fractions,
  decimals, years)
- The text-normalization frontend for a Dhivehi TTS (text-to-speech)
  pipeline, or standalone use in any Maldivian language NLP tool

## Installation

```bash
pip install dv-normalize
```

## Quick start

```python
from dv_normalize import normalize

normalize("ވަކި ލާރިން ވެސް 232.23 ލާރި ހޯދައެވެ")
# 'ވަކި ލާރިން ވެސް ދުވިސައްތަ ތިރީސް ދޭއް ޕޮއިންޓް ތޭވީސް ލާރި ހޯދައި'

normalize("ޑރ. އިބްރާހިމް 14:30 ގައި އައި")
# 'ޑޮކްޓަރު އިބްރާހިމް ސާދަ ގަޑި ތިރީސް ގައި އައި'

normalize("ކ.އަތޮޅު ވިލިނގިލިން 120 ކިލޯ މީޓަރު")
# 'ކާފު އަތޮޅު ވިލިނގިލިން އެއް ސަތޭކަ ވިހި ކިލޯ މީޓަރު'

normalize("ފޭސް2ގެ")
# 'ފޭސް ދޭއް ގެ'

normalize("އިތުރު މައުލޫމާތު: https://example.com/news/123 ބައްލަވާ")
# 'އިތުރު މައުލޫމާތު: https://example.com/news/123 ބައްލަވާ'  (URL passes through untouched)
```

For repeated use, hold onto a `Normalizer` instance:

```python
from dv_normalize import Normalizer, NormalizerConfig

n = Normalizer(NormalizerConfig(keep_punctuation=False))
n("ހެލޯ، ދުނިޔެ")  # → 'ހެލޯ ދުނިޔެ'

# strict=True raises ValueError if any digit survives normalization; on
# well-formed input it's a no-op safety net.
ns = Normalizer(NormalizerConfig(strict=True))
ns("232 ލާރި")  # → 'ދުވިސައްތަ ތިރީސް ދެ ލާރި'
```

## How it works

A four-stage pipeline: **tokenize** (split text into words, punctuation and
whole-passthrough spans like URLs) → **classify** (a priority-ranked rules
engine tags each token with a semiotic class — CARDINAL, DATE, MONEY, …) →
**verbalize** (each class's own module renders the spoken-form words) →
**post-process** (sentence-ending rewriting, spacing, quote/punctuation
fixups). The number and date rules were checked against a 2-million-plus-
sentence news + majlis corpus; the 214 sentence-ending rules were reviewed
by a native speaker one family at a time against real corpus sentences and
cross-checked against ~440,000 majlis speech sentences.

## What it handles

| Class                | Example input                       | Example output                                               |
| -------------------- | ------------------------------------ | -------------------------------------------------------------- |
| Cardinal             | `232`                                | `ދުވިސައްތަ ތިރީސް ދޭއް`                                        |
| Comma-grouped        | `104,880`                            | `އެއް ލައްކަ ހަތަރުހާސް އަށް ސަތޭކަ އައްޑިހަ` (single cardinal)   |
| Per-digit identifier | `9982711`                            | `ނުވައެއް ނުވައެއް އަށެއް ދޭއް ހަތެއް އެކެއް އެކެއް` (7+ digits)   |
| Decimal              | `232.23`                             | `ދުވިސައްތަ ތިރީސް ދޭއް ޕޮއިންޓް ތޭވީސް`                          |
| Year                 | `2024`                               | `ދެހާސް ސައްވީސް`                                               |
| Year range           | `1982 - 2024`                        | `ނަވާރަސަތޭކަ އައްޑިހަ ދޭއް އިން ދެހާސް ސައްވީސް އާ ހަމައަށް`       |
| Date                 | `12 ފެބުރުވަރީ 1988`                 | `ބާރަ ފެބުރުވަރީ ނަވާރަސަތޭކަ އައްޑިހަ އަށް`                       |
| Time                 | `14:30`                              | `ސާދަ ގަޑި ތިރީސް`                                              |
| Ordinal              | `11ވަނަ`                             | `އެގާރަ ވަނަ`                                                   |
| Fraction             | `1/2`                                | `ދެބައިކުޅަ އެއްބައި`                                            |
| Mixed fraction       | `1 1/2`                              | `އެކެއް ދެބައިކުޅަ އެއްބައި`                                      |
| Percent              | `25%`                                | `ފަންސަވީސް ޕަސެންޓު`                                            |
| Percent, prefix pair | `%12-%13`                            | `ބާރަ ޕަސެންޓު-ތޭރަ ޕަސެންޓު` (both values read)                 |
| Percent, glued punct | `(50%)`                              | `(ފަންސާސް ޕަސެންޓު)`                                           |
| Oblique/reference    | `2024/3`                             | `ދެހާސް ސައްވީސް އޮބްލިކް ތިން ވަނަ`                              |
| Oblique, abbreviated | `އޖ/21/2019`                         | `އަލިފު ޖަވިޔަނި އޮބްލިކް އެކާވީސް އޮބްލިކް ދެހާސް ނަވާރަ ވަނަ`     |
| Score                | `3-2`, `0-0`, `5-0`                  | `ތިނެއް ދޭ`, `ސުމެއްސުމެއް`, `ފަހެއްސުމެއް` (draw/shutout forms)  |
| Money                | `41,800 ރ`                           | `ސާޅީސް އެއްހާސް އަށް ސަތޭކަ ރުފިޔާ`                             |
| Money, silent marks  | `-/7000ރ`, `500/-`                   | `ހަތްހާސް ރުފިޔާ`, `ފަސް ސަތޭކަ ރުފިޔާ`                          |
| Phone (+country)     | `+960 9611929`                       | `ޕްލަސް ނުވައެއް ހައެއް ސުމެއް ނުވައެއް ...` (every digit)        |
| Plus sign            | `+2.88`, `50+`                       | `ޕްލަސް ދޭއް ޕޮއިންޓް އައްޑިހަ އަށެއް`, `ފަންސާސް ޕްލަސް`         |
| Age                  | `(40އ)`                              | `(ސާޅީސް އަހަރު)`                                               |
| Abbreviation         | `ޑރ.`                                | `ޑޮކްޓަރު` (real abbreviations only)                          |
| Atoll code           | `ގދ.`, `ހއ.`                         | `ގާފު ދާލު`, `ހާ އަލިފު` (per-letter spelling)                  |
| Compound abbrev      | `ސ.ޢ.ވ.`                             | `ޞައްލަﷲ ޢަލައިހި ވަސައްލަމް`                                    |
| Calendar marker      | `2026 މ.`, `1447 ހ.`                 | `ދެހާސް ސައްބީސް މީލާދީ`, `... ހިޖުރީ`                           |
| Latin acronym        | `MNDF`                               | `އެމް އެން ޑީ އެފް` (2-4 letter all-caps, always spelled)        |
| URL / e-mail / handle / hashtag | `https://example.com/x?y=1` | unchanged (passthrough, digits inside never read aloud)        |
| Sentence ending      | `ނުކެރުނެވެ`                          | `ނުކެރުނު`                                                      |
| Sentence ending      | `ދުވަހެވެ`                            | `ދުވަސް`                                                        |
| Sentence ending, before a closing quote | `މަޖުބޫރުވާނެއެވެ".`      | `މަޖުބޫރުވާނެ".`                                                 |

The classifier is priority-ranked, so more specific patterns (calendar
markers, multi-letter compound abbreviations, year ranges, the digit-anchored
symbol rules) shadow the generic ones. Tokens that don't match any rule pass
through unchanged.

Sentence-ending coverage: 214 reviewed rewrite rules handle the
literary noun/verb ending families found across the news + majlis corpora
(formal `-އެވެ`/`-ެވެ` endings collapsed to their spoken form). The rules
were validated against 437,000 majlis speech sentences — majlis transcripts
are themselves largely spoken-register, so a rule that also matches the
majlis corpus's own usage is strong evidence it is producing a real spoken
form rather than an invented one.

## Coverage

Measured against a 1.79M-sentence news corpus and a 399k-sentence majlis
(parliament) corpus.

- **Full-corpus baseline**, before the 1.0 ending-rules and symbol-handling
  work: news 96.87% of 1,794,272 sentences fully normalized, majlis 99.44%
  of 399,088 sentences.
- **Final 6,000-document sample** (3,000 news articles + 3,000 majlis
  turns) for this release, measured two ways:
  - *Raw* (every sentence in the sample, Dhivehi or not): news 98.18%,
    majlis 99.93% fully normalized.
  - *Dhivehi-content* (sentences that contain at least some Thaana text;
    a sentence whose only non-Dhivehi content is a passed-through URL
    still counts as handled): news 99.86%, majlis 99.93%.
  - Zero sentences in either sample left a digit unspoken.

The raw-sample residue is overwhelmingly English-language prose and
web-scraping boilerplate (menus, bylines, "read more" links) that isn't
Dhivehi text to begin with, so it's outside what a Dhivehi normalizer is
meant to convert. The smaller Dhivehi-content residue is mostly English
words embedded inside otherwise-Dhivehi sentences (left as-is per
`unknown_latin`), street-address-style slashes, and Arabic-script Quranic
or hadith citations.

**Ending alignment:** of literary-register endings found in the news
sample, 99.9% are rewritten by some rule at all, and 91.5% are rewritten
to a form that also occurs, verbatim, as a genuine spoken-register ending
somewhere in the majlis corpus. That second number is a deliberately
conservative metric — a rewritten ending that happens not to appear in
the majlis sample isn't necessarily wrong, it just wasn't independently
confirmed against spoken-register evidence.

## Known issues

Measured state of `1.0.0` (see [Coverage](#coverage) for how the numbers
were taken): 99.86% of Dhivehi-content news sentences and 99.93% of majlis
sentences come out fully spoken, no digit is ever left unread, and 91.5% of
rewritten sentence endings are attested verbatim in spoken-register
majlis speech. In a human review of 28 random news articles read end to
end, 20 needed no change and 8 had corrections; every one of those
corrections is now encoded as a rule. What remains:

- **Definite vs. indefinite nouns.** Literary `Xއެވެ` (definite, "it is X")
  and `Xެކެވެ` (indefinite, "an X") are handled separately, but a few
  legacy overrides still blur them: `ދައްކައެވެ` reads `ދައްކަ` while the
  spaced `ދައްކަ އެވެ` reads `ދައްކައި`. Awaiting a native-speaker ruling.
- **`ގެ` after a number** is always read as the genitive particle, so a
  number before `ގެ` meaning "house" inflects wrongly (`ގޭ` is not produced
  for `ގެ އެވެ` either).
- **Ablative `ން` after a number** is not inflected: `5 ން ފެށިގެން` reads
  `ފަސް ން` rather than the spoken `ފަހަކުން`.
- **Durations in `mm:ss` after a hour count** keep the colon:
  `3 ގަޑި 30:04 މިނިޓު` reads `ތިން ގަޑި ތިރީސް:ހަތަރު މިނިޓު`.
- **Percentages:** negative percents, the fraction separator in
  comma-decimal mode, and `50% އަށްވުރެ` (no dative inflection) are read
  more literally than a speaker would.
- **Units** such as `kg`, `km`, `GB` have no Dhivehi lexicon yet and follow
  `unknown_latin`; Roman numerals are spelled letter by letter.
- **English words inside Dhivehi sentences** are left as written by default
  (`unknown_latin="passthrough"`); there is no transliteration.
- **Score draws** like `2-2` use the linking form `ދޭދޭ`; other scores read
  both numbers in full (`3-2` → `ތިނެއް ދޭއް`).
- **Arabic-script quotations** (Quran, hadith) pass through untouched.

Report a wrong reading with the written sentence and the form you would say.

## Configuration

```python
NormalizerConfig(
    dialect="spoken",            # only option for now
    unknown_latin="passthrough", # "passthrough" | "drop" | "spell"
    decimal_separator="auto",    # "auto" | "dot" | "comma"
    time_system="auto",          # "auto" | "12" | "24"
    currency_default="MVR",
    keep_punctuation=True,
    diagnostic=False,
    strict=False,
)
```

- `dialect="spoken"` — the only option today; reserved for a future
  literary/written-register mode.
- `unknown_latin` — controls plain Latin words only, never acronyms and
  never URLs: 2-4 letter all-caps runs (`MNDF`) are always spelled via
  Thaana letter names regardless of this setting, and URLs/e-mails/handles/
  hashtags always pass through untouched regardless of this setting (even
  `"drop"` never removes a URL — only ordinary Latin words like `STORE`).
  `"passthrough"` (default) leaves other Latin words unchanged; `"drop"`
  removes them (a Latin-only input can yield an empty string); `"spell"`
  spells them out letter-by-letter.
- `decimal_separator` — `"auto"` and `"dot"` both use `.` as the decimal
  mark and `,`/`،` as thousands separators. `"comma"` swaps that: `,` is
  the decimal mark (`1,5` → `އެކެއް ޕޮއިންޓް ފަހެއް`), except a chain of
  ≥2 three-digit `,` groups is still read as thousands grouping
  (`1,234,567` stays one cardinal, not a decimal).
- `time_system` — `"auto"` and `"24"` read the hour as-is
  (`14:30` → `ސާދަ ގަޑި ތިރީސް`). `"12"` prefixes a day-period word and
  reduces the hour to a 12-hour clock value
  (`14:30` → `މެންދުރުފަހު ދެ ގަޑި ތިރީސް`).
- `currency_default` — documented as reserved; only `"MVR"` (ރ) is wired
  today, so changing it currently has no effect.
- `keep_punctuation` — when `False`, punctuation tokens are dropped from
  the output instead of being kept verbatim.
- `diagnostic` — internal flag used by `Normalizer.trace()`; leave this
  `False` when calling `normalize()`/`Normalizer.__call__` directly.
- `strict` — when `True`, raises `ValueError` if any ASCII or
  Arabic-Indic digit survives in the final output instead of silently
  passing it through. Digits inside a passed-through URL/e-mail/handle
  token are exempt from this check — `strict=True` does not raise on
  `https://example.com/12345`, since that digit was never meant to be
  read aloud in the first place.

## Diagnostic mode

`Normalizer.trace(text)` returns the classified token list instead of joined
text. Useful for debugging which rule fired:

```python
for tok in Normalizer().trace("ޑރ. އިބްރާހިމް 2024ގައި"):
    print(tok.cls, tok.text, tok.spoken, tok.fields)
```

## License

MIT — see `LICENSE`.
