Metadata-Version: 2.5
Name: thalovant-languages
Version: 0.3.0
Summary: Every word a Thalovant component's rule turns on, for every supported language, in one place.
Project-URL: Homepage, https://github.com/thalovant/thalovant-languages
Project-URL: Repository, https://github.com/thalovant/thalovant-languages
Project-URL: Issues, https://github.com/thalovant/thalovant-languages/issues
Author: Thalovant
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: language,locale,openvoiceos,thalovant,voice-assistant
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: ovos-spec-tools[langcodes]>=1.10.5a1
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: babel>=2.14; extra == 'dev'
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# thalovant-languages

Every word a Thalovant component's rule turns on, for every language the
sources know, derived from those sources and kept in one place, out of
every codebase.

A voice satellite waits on "turn off the" because "the" is a continuation
word. An intent listing closes "quelle heure est-il" with a question mark
because "quelle" opens a question, and reads "volume {level} percent" aloud
as "volume cinquante pour cent" in French. A French synthesiser is told to
say "onze minutes" where it would swallow a consonant. Each of those is a
fact about a language, not about the program that needs it. Here they are
data: one directory per language, 270 of them, read by the SDK and by the
satellite, and none of it written by hand.

## Where the words come from

| What | Source | How |
| --- | --- | --- |
| `continuation_words` | Universal Dependencies treebanks (150 languages) | a word that is a determiner, preposition, conjunction, auxiliary or possessive nearly every time it appears, and almost never the last word of a sentence |
| `trailing_words` | Universal Dependencies | the continuation words that are neither a possessive nor an auxiliary: "coupe le son" and "qué hora es" are whole sentences, not prefixes |
| `question_openers` | Universal Dependencies | an interrogative word (`PronType=Int`), or a word a sentence usually is a question when it starts with it |
| `question_words_anywhere` | Universal Dependencies | an interrogative word a sentence almost always is a question when it holds it, wherever it sits |
| `written_forms` | Universal Dependencies | a closed-class word written capitalised mid-sentence nine times in ten (`i: I`) |
| `plural` | Unicode CLDR (190 locales) | the language's plural rules, verbatim, evaluated at runtime by `_plural.py` |
| `lowercase_map` | Unicode `SpecialCasing.txt` | a language's own unconditional lower-case mappings: Turkish and Azeri's dotted and dotless i |
| `scripts.yaml` | Unicode `PropList.txt`, `Scripts.txt` | scripts written without spaces, scripts where a character is a syllable, the marks that close a sentence (`Sentence_Terminal`) or break a clause (`Terminal_Punctuation`), spaced or unspaced |
| `slot_examples`, `speech_substitutions`, `question_patterns` | `overrides/<tag>.yaml` | what only a person knows: how a slot reads aloud in a house, what a Piper voice mispronounces, an inversion no treebank lists |

Every generated file starts with a header naming its sources, their
versions, the treebanks it was read from, and the thresholds that decided
what got in. The thresholds are numbers a reader can argue with; the
argument then has its facts. Two of them were settled by measurement on the
voice satellite: English "is" ends three sentences in a hundred and is worth
waiting on, "on" ends six ("turn it on") and is not.

## Layout

```text
src/thalovant_languages/languages/
  scripts.yaml            what a writing system does that no word list can
  en/language.yaml        English, every region
  fr/language.yaml        French
  pt/language.yaml        Portuguese ...
  pt-PT/language.yaml     ... and what European Portuguese does differently
overrides/
  en.yaml, fr.yaml        what only a person knows, laid on top
  scripts.yaml            the two marks the synthesiser splits on that Unicode does not list
scripts/derive.py         the generator
```

A directory is named by a BCP-47 tag: bare when what it says holds for
every region, regional when a source says the region differs (CLDR gives
`pt-PT` its own plural rule). A regional file carries only the difference;
the loader lays it over the language's file.

## Use

```python
import thalovant_languages as languages

languages.language("fr-CA")["continuation_words"]   # the French file
languages.words("en", "trailing_words")             # lower-cased, as a set
languages.plural_category("ru", 21)                 # "one", by CLDR's rules
languages.asks("on mange à quelle heure ce soir", "fr")  # True: it holds a question word
languages.language("tlh")                           # {} -- nothing describes it
languages.script_pattern("unspaced").search("今天") # a character of an unspaced script
languages.marks("sentence_ends", "spaced")          # every Sentence_Terminal mark, and "…;"
```

A language is found by the matcher the rest of OVOS uses
(`ovos_spec_tools.language`): `fr-CA` reads `fr`, `en-GB` reads `en`, and a
language nothing describes gets an empty mapping rather than another
language's rules. The keys a component reads are `thalovant_languages.KEYS`.

`THALOVANT_LANGUAGES_DIR=/path/to/checkout/src/thalovant_languages/languages`
reads a checkout instead of the installed data, for trying a change before
it is released; `languages.refresh()` forgets what was read after changing it.

## Regenerating

```bash
pip install -e ".[dev]"
curl -LO https://www.unicode.org/Public/UCD/latest/ucd/PropList.txt      # and Scripts.txt, SpecialCasing.txt
curl -LO https://raw.githubusercontent.com/unicode-org/cldr-json/main/cldr-json/cldr-core/supplemental/plurals.json
# the treebanks: https://universaldependencies.org/ -> the current release archive (~700 MB)
python scripts/derive.py --ud ud-treebanks-v2.18 --cldr plurals.json --unicode .
thalovant-languages check
git diff --stat
```

The generator reads about four gigabytes of treebanks in a minute or two
and rewrites every file; review the diff, commit. To change a word, do not
edit the file: change the generator's thresholds, or put what you know in
`overrides/`, and rerun. `thalovant-languages check` refuses a key no
component reads, a pattern that does not compile, a plural rule the
evaluator cannot read, and a list holding a boolean.

## Who reads it

- `thalovant` (the Python SDK): `thalovant.listing` sets a registered pattern
  the way a person reads it, ranks phrases, and reads slots aloud.
- `thalovant-voice` (the satellite): the semantic endpoint's continuation
  words, the lower-casing of an all-capitals transcript, the plural forms of
  its own counted text, the synthesiser's pronunciation repairs.
- the fleet's fallback skills (weather, date and time): a keyword net claims
  a sentence only when `asks()` says it is a question.

Nothing here parses dates, numbers or colours; the OVOS parsers do that for
skills on the hub. Contractions come from `ovos-utterance-normalizer`, which
already carries them.
