Metadata-Version: 2.4
Name: turkic-translit
Version: 0.5.10
Summary: Deterministic Latin and IPA transliteration for Kazakh, Kyrgyz, Uzbek, Turkish, Azerbaijani, Uyghur, Finnish, plus tokenizer/glue scripts.
Author-email: Austin Wagner <austinwagner@msn.com>
License-Expression: MIT
Keywords: kazakh,kyrgyz,transliteration,ipa
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: fasttext-wheel==0.9.2
Requires-Dist: gradio<7,>=6.0
Requires-Dist: numpy>=1.26
Requires-Dist: pandas
Requires-Dist: pyicu-wheels==2.15.2
Requires-Dist: pycountry>=23.12
Requires-Dist: python-json-logger>=3.1
Requires-Dist: PyYAML>=6.0
Requires-Dist: rapidfuzz>=3.5
Requires-Dist: rich>=13.7
Requires-Dist: sentencepiece>=0.2.0
Requires-Dist: tabulate>=0.9
Requires-Dist: torch>=2.2
Requires-Dist: tqdm>=4.66
Requires-Dist: transformers<6,>=5.0
Requires-Dist: accelerate>=0.28
Requires-Dist: zstandard>=0.23
Provides-Extra: examples
Requires-Dist: flask; extra == "examples"
Requires-Dist: streamlit; extra == "examples"
Requires-Dist: jupyterlab; extra == "examples"
Provides-Extra: sentry
Requires-Dist: sentry-sdk>=2.0; extra == "sentry"
Provides-Extra: dev
Requires-Dist: ruff>=0.2.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: pytest-xdist>=3.6; extra == "dev"
Requires-Dist: pytest-timeout>=2.3; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4.0; extra == "dev"
Requires-Dist: make>=0.1.6; extra == "dev"
Requires-Dist: types-PyYAML>=6.0; extra == "dev"
Requires-Dist: pandas-stubs>=2.0; extra == "dev"
Requires-Dist: types-tqdm>=4.66; extra == "dev"
Requires-Dist: packaging>=23.0; extra == "dev"
Requires-Dist: panphon<0.23,>=0.22; extra == "dev"
Requires-Dist: sentry-sdk>=2.0; extra == "dev"
Requires-Dist: protobuf>=4.0; extra == "dev"
Dynamic: license-file

# Turkic Transliteration

**Try it:** <https://huggingface.co/spaces/AustinWagner/turkic-transliteration-demo>

*(This repo is the deployment source for that demo. The Space's card lives at
`.github/hf-space/README.md`; a push to `main` writes it, `app.py` and the
pinned requirement onto the Space. This README used to carry a second copy of
that card, which is how the Space came to install a Gradio the package no
longer accepted.)*

Deterministic Latin and IPA transliteration for Turkic languages, plus helper
utilities for corpus building, tokenizer/LM training, and Russian-token
filtering.

The design decision worth noting: **languages are discovered dynamically from
rule files** in `src/turkic_translit/rules/`, so adding a language means adding a
`.rules` file — no code change, and the CLI, the library API (`to_ipa`/`to_latin`)
and the web demo all pick it up at once.

**Supported languages** (verified by the test suite): Azerbaijani, Finnish,
Kazakh, Kyrgyz, Turkish, Uyghur, and Uzbek (both Cyrillic and Latin input).
IPA output is available for all of them; Latin output covers Kazakh and
Kyrgyz (Cyrillic → Latin, with optional embedded Arabic-script handling)
and Turkish (diacritic folding to ASCII).

## Install

Python **3.10–3.14**, whichever platform: `pyicu-wheels` publishes the ICU
extension for all of them.

```bash
pip install turkic-translit
turkic-translit web          # or any of the commands below
```

**The PyPI release can lag the rules on `main`.** A rule fix reaches PyPI
only with the next release, and a corpus transliterated with an older
release carries the older rules; 0.5.9, for example, still writes Kazakh,
Kyrgyz and Cyrillic Uzbek ‹щ› as `ɕː`, which `main` corrected to `ʃː`. To
build a corpus with the current rules, install from the repository:

```bash
pip install "git+https://github.com/wagner-austin/turkic_transliteration.git"
```

The install brings PyTorch with it (for the language-model commands), so
expect several gigabytes.

Nothing else: ICU arrives with it. That was not true before 0.5.6 — PyICU
publishes no wheels on any platform, so an install had to compile against ICU
headers, and this project shipped a Windows wheel fetcher to work around it.
The dependency is `pyicu-wheels`, which publishes the same extension prebuilt
for Linux, macOS and Windows.

To work on the project rather than use it:

```bash
git clone https://github.com/wagner-austin/turkic_transliteration.git
cd turkic_transliteration
pip install -e .[dev]        # or: python scripts/setup_dev.py
turkic-translit web
```

Optional extras: `dev` (ruff, mypy, pytest) · `examples` (Flask, Streamlit,
JupyterLab) · `sentry` (error reporting).

### Package names

- Import path: `turkic_translit`
- PyPI distributable: `turkic-translit`
- Primary CLI entry point: `turkic-translit`

## Development

The Makefile wraps the common tasks (all via Poetry):

```bash
make check   # lint + test (the full gate)
make lint    # guards, then ruff check --fix + ruff format, then strict mypy
make guard   # just the guard rules (scripts/guards/)
make test    # pytest with 100% statement+branch coverage enforced
make web     # launch the Gradio web UI
```

On Windows without GNU Make: `./scripts/run.ps1 <target>`.

Linting, type-checking (strict mypy) and the guard rules all cover `src`,
`tests` and `scripts`. The guards ban weak typing (`Any`, casts, ignores),
silent exception handling, and weak or fake tests — including
transliteration tests that never compare output against an expected value.

## Validation

- Every IPA rule file declares its source in machine-readable `# Source-*`
  header fields, and a test module that pins expected output must name the
  source it inherits (`INHERITS_SOURCE`); both are guard-enforced, as is the
  ban on expectations computed by the code under test.
- Expected values are transcriptions printed in the cited descriptions
  (journal Illustrations, reference-grammar chapters). Each language is
  covered letter-exhaustively plus word sets from a second, independent
  description; Uyghur is additionally checked against the MFA dictionary's
  22,630 pronunciations.
- Departures from a source's printed notation are declared in two tables —
  `NOTATION` (different glyph, same phoneme) and `DECLARED_SIMPLIFICATIONS`
  (a real distinction the rules drop) — each entry carrying a reason. A
  central test fails any entry no printed datum exercises, and a
  cross-language test requires shared letters to agree across languages
  after corpus harmonization unless a contrast is declared with a citation.

## Command-line tools

Console entry points defined in `pyproject.toml`:

| Command | Purpose |
|---------|---------|
| `turkic-translit` | Transliterate text to Latin and/or IPA |
| `turkic-translit web` | Launch the Gradio web demo |
| `turkic-filter-russian` | Drop or mask Russian tokens from a stream |
| `turkic-download-corpus` | Download/prepare OSCAR corpora |
| `turkic-clean-corpus` | Clean and harmonize downloaded corpora |
| `turkic-build-spm` / `turkic-train-spm` | Train a SentencePiece tokenizer |
| `turkic-train-lm` / `turkic-eval-lm` | Train / evaluate a language model |
| `turkic-leven` | Levenshtein-based comparison utility |

### `turkic-translit translit` usage

```bash
turkic-translit translit --lang kk --in text.txt \
    --out-latin kk_lat.txt --out-ipa kk_ipa.txt --arabic
```

- `--lang` — any code advertised by the rules directory; IPA is available
  for `az`, `fi`, `kk`, `ky`, `tr`, `ug`, `uz` and `uzc` (Uzbek Cyrillic)
- `--in` — input file, or `-` for stdin (default: `-`)
- `--out-latin` — Latin output path, or `-` for stdout. Omit to skip.
  Only meaningful for languages with `<lang>_lat.rules` (`kk`, `ky`, `tr`).
- `--out-ipa` — IPA output path, or `-` for stdout. Omit to skip.
- `--arabic` — also transliterate embedded Arabic script (Latin mode only)
- `--benchmark` — log throughput statistics on completion

At least one of `--out-latin` / `--out-ipa` must be specified.
`--log-level` is set on the parent group: `turkic-translit --log-level debug translit ...`.

### `turkic-clean-corpus` usage

The rules transliterate their own script and pass everything else through
unchanged, so English, URLs and code quoted in web text reach the IPA file
in their original spelling. The cleaner removes them: each language's set
of emittable characters is derived from its own rules, and a token
carrying any letter outside that set is dropped whole.

```bash
turkic-translit translit --lang kk --in kk.txt --out-ipa raw/kk_ipa.txt
turkic-clean-corpus --input-dir raw --output-dir clean
```

- `--input-dir` / `--output-dir` — raw IPA corpora in, cleaned corpora and
  `cleaning_manifest.json` out. The language is read from each file name
  (`kk_ipa.txt`, `oscar_kk_ipa.txt`), which must carry exactly one
  two-letter code.
- The stages run in this order: the symbol map harmonises notation across
  languages; lines whose share of transcription characters is below
  `--min-ipa-ratio` (default 0.95) are dropped; foreign tokens are
  dropped; punctuation, digits and other characters the rules cannot emit
  become spaces; lines shorter than `--min-line-chars` (default 30) are
  dropped; repeated lines are dropped, keeping the first. The order and
  the reason for each stage are in `turkic_translit/corpus/clean.py`.
- Last, every corpus in one run is cut to the size of the smallest, so a
  cross-language comparison measures languages rather than data volume.
  Clean one language on its own to keep all of it.
- The manifest counts every drop and replacement, and fingerprints the
  rule files and symbol map the run used.
- Clean only text transliterated by the same version of the rules: the
  emittable set comes from the installed rules, so a corpus made under
  older rules can carry letters the current ones no longer emit, and the
  cleaner drops those tokens as foreign.

### Tokenizer training

```bash
turkic-build-spm --input corpora/kk_lat.txt,corpora/ky_lat.txt \
    --model_prefix spm/turkic12k --vocab_size 12000
```

### Filtering Russian tokens

```bash
cat uz_raw.txt | turkic-filter-russian --mode drop > uz_clean.txt
```


## Project Organization

- `src/turkic_translit/` — core package (`core.py`, `transliterate.py`,
  `rules/`, `cli/`, `web/`, `lm/`, language-ID and filtering modules)
- `data/` — sample data and language resources
- `docs/` — documentation and guides
- `scripts/` — dev + release utilities (`setup_dev.py`, `run.ps1`, `release/`)
- `tests/` — test suite (per-language IPA coverage for all supported languages)
- `cronjob/` — scheduled-task assets for the hosted demo
- `app.py` — Hugging Face Space entry point for the web UI

## FastText language-identification model

Russian-token filtering and language detection use FastText's `lid.176.bin`.
The model is **not** committed (too large); it is downloaded automatically from
the official Facebook AI link on first use and cached in the package directory —
no manual step needed for pip installs, Hugging Face Spaces, or CI. Manual
source if needed:
https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin

## Logging & error reporting

Central logging supports structured JSON with correlation IDs. Control with
`TURKIC_LOG_LEVEL` (DEBUG…ERROR) and `TURKIC_LOG_FORMAT=json|rich` (default
json); libraries can call `turkic_translit.logging_config.setup()`. Optional
Sentry integration via `TURKIC_SENTRY_DSN` (+ `TURKIC_ENV`,
`TURKIC_SENTRY_TRACES`); install with `pip install turkic-translit[sentry]`.

All I/O is UTF-8; on Windows a BOM is written when piping to files to avoid
encoding issues.

## Relationship to the LSTM project

This repo is the upstream data pipeline for the Turkic mutual-intelligibility
LSTM experiments (github.com/wagner-austin/LSTM): `turkic-download-corpus` plus
the IPA transliteration rules produce the per-language IPA corpora that project
trains on.

## License

Apache-2.0
