Metadata-Version: 2.5
Name: termflex
Version: 1.0.0
Summary: Morphology-Aware Term Checking for Translation QA and LLM Output Evaluation
Project-URL: Homepage, https://github.com/oksanatkach/termflex
Project-URL: Changelog, https://github.com/<you>/termflex/blob/main/CHANGELOG.md
License-Expression: MIT
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: langcodes
Requires-Dist: lxml
Requires-Dist: polib
Requires-Dist: pyahocorasick
Requires-Dist: simplemma
Provides-Extra: all
Requires-Dist: pymorphy3; extra == 'all'
Requires-Dist: pymorphy3-dicts-ru; extra == 'all'
Requires-Dist: pymorphy3-dicts-uk; extra == 'all'
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: pymorphy3
Requires-Dist: pymorphy3; extra == 'pymorphy3'
Requires-Dist: pymorphy3-dicts-uk; extra == 'pymorphy3'
Provides-Extra: pymorphy3-ruzzian
Requires-Dist: pymorphy3; extra == 'pymorphy3-ruzzian'
Requires-Dist: pymorphy3-dicts-ru; extra == 'pymorphy3-ruzzian'
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Description-Content-Type: text/markdown

# Termflex: Morphology-Aware Term QA

Use Termflex to:

- QA translations in .po, .xlf / .xliff, .csv and .json files against a glossary (in standard TBX and PO formats or free form CSV or JSON).
- Ground LLM output by checking that generated text uses the approved terms. In this case, the source and target languages are the same, the source string is the prompt, and the target string is the LLM response.

**Task:** In localization / machine translation / technical writing / LLM generation, the resulting text can often be semantically correct but ignore approved terms and use synonyms instead. This is an issue for consistency across documents and previously agreed upon project style guide.

**Problem:** Terminology QA checks if the resulting text contains the approved terms pre-defined in a project glossary. However, term checkers are often naive and only do direct string matching, ignoring different word forms. This produces many false positives and false negatives.

**Solution:** Termflex performs term QA even for inflected word forms by using lemma matching instead of plain string matching. Termflex lemmatizes both sides, so it matches on the dictionary form and still reports exactly where the term was found.

## Supported lemmatizers

Works with any language you have a lemmatizer for: termflex ships with
**simplemma** and **pymorphy3** adapters, and you can plug in your own easily. To use another lemmatizer, pass any
callable that maps a word to its lemma:
```python
    checker = termflex.Checker(termflex.Glossary.from_tbx('sample_files/TBX-basic-sample.tbx', source_lang='en', target_lang='fr', lemmatizer=custom_lemmatize))
```
See [Adding your own backend](#Adding-your-own-backend) for other integration options.

## Install

```bash
pip install termflex
```

Optional morphology backends are installed as extras:

```bash
pip install "termflex[pymorphy3]"          # Ukrainian dictionaries
pip install "termflex[pymorphy3-ruzzian]"  # Russian dictionaries
pip install "termflex[all]"                # everything
```

`simplemma` is installed by default. Requires Python 3.10 or newer.

## Quick start (CLI)

Check one file:

```bash
termflex translations/uk.po --glossary glossary.tbx --src en --tgt uk
```

Check a directory. Directories need `--format`, so that nothing is checked by accident:

```bash
termflex locale/uk --format po --glossary glossary.tbx --src en --tgt uk
termflex project/ --format xliff --glossary glossary.tbx --src en --tgt uk
```

CSV and JSON files have no fixed structure, so name the source and target fields:

```bash
# LLM grounding: prompt/response pairs
termflex pairs.csv --glossary required_terms.csv --glossary-source-field term --glossary-target-field term \
                   --src en --tgt en --source-field prompt --target-field response

termflex data/ --format json --glossary glossary.tbx --src en --tgt fr \
    --source-field en --target-field fr
```

### Options

| Option                                                 | Meaning                                                                                                                                                                                       |
|--------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `input`                                                | A file, or a directory (searched recursively).                                                                                                                                                |
| `--format {po,xliff,csv,json}`                         | Which files to collect from a directory. Required for directories. For a single file the format is inferred from its extension.                                                               |
| `--glossary FILE`                                      | The glossary: `.tbx`, `.po`, `.csv` or `.json`, with the format inferred from the extension. `--tbx` still works as an alias.                                                                 |
| `--glossary-format {tbx,po,csv,json}`                  | Set the glossary format when the extension does not say it.                                                                                                                                   |
| `--glossary-source-field`, `--glossary-target-field`   | CSV glossary: the source and target term columns (required). JSON glossary: the term keys, instead of `--glossary-schema`. See [JSON glossaries](#json-glossaries).                           |
| `--glossary-schema JSON_OR_FILE`                       | JSON glossary: where the terms are, as inline JSON or a path to a `.json` file. See [JSON glossaries](#json-glossaries).                                                                      |
| `--src`, `--tgt`                                       | Source and target language codes. They apply to every file checked in the run.                                                                                                                |
| `--source-field`, `--target-field`                     | Column (CSV) or key (JSON) holding the source and target text. Required for those formats.                                                                                                    |
| `--context-field`                                      | Optional column or key to show as context in the report.                                                                                                                                      |
| `--id-field`                                           | Optional id key (.json); reported instead of the record number                                                                                                                                |
| `--schema JSON_OR_FILE`                                | JSON input file: inline JSON or a path to a `.json` file. See [JSON input](#json-input).                                                                                                      |
| `--lemmatizer SPEC`                                    | Lemmatizer for both languages: a registered backend name, or `module:name` (see [Choosing a backend](#choosing-a-backend-from-the-cli)).                                                      |
| `--source-lemmatizer SPEC`, `--target-lemmatizer SPEC` | Same, for one side only. They override `--lemmatizer`.                                                                                                                                        |
| `--plugin`                                             | Import a module (my_pkg.backends) or a .py file (./my_backends.py) whose @register_backend backends become available by name; repeatable. Also read from $TERMFLEX_PLUGINS (separated by ':') |
| `--list-backends`                                      | Print the available backends and exit                                                                                                                                                         |
| `-q`, `--quiet`                                        | Print only the summary, not the report                                                                                                                                                        |
| `-v`, `--verbose`                                      | Also list the terms that were translated correctly                                                                                                                                            |
| `--output-format {text,json}`                          | text (default): report and summary; json: machine-readable results on stdout, summary on stderr                                                                                               |
| `--json FILE`                                          | Also write the full results to a JSON file.                                                                                                                                                   |
| `--workers N`                                          | Worker processes (default: CPU count; `1` runs in-process, which is handy for debugging).                                                                                                     |

Language selection is deliberately left to you. A glossary covers one language pair, so check one language per run, for example `locale/uk`, not the whole `locale/` tree.

### JSON glossaries

A JSON glossary is described in one of two ways (not both):

- **Field flags:** `--glossary-source-field` and `--glossary-target-field`. The file must be a list of objects,
  one per term pair: `[{"en": "folder", "uk": "тека"}, ...]`.
- **`--glossary-schema`:** inline JSON or a path to a `.json` file, either
  - a records schema: `{"root": ..., "source": ..., "target": ...}`, where only `source` and `target` are
    required, or
  - a mapping schema: `{"layout": "mapping"}`, optionally with `"root"`, for `{"folder": "тека", ...}` objects.

A source term with several approved translations is written as several records with the same source.

### Choosing a backend from the CLI

Three flags take a backend spec:

- `--lemmatizer SPEC` sets the lemmatizer for both languages.
- `--source-lemmatizer SPEC` and `--target-lemmatizer SPEC` set it for one language and override `--lemmatizer`.

Without these flags, the glossary's default lemmatizer is used. A spec can be:

| Spec                          | Meaning                                                                               |
|-------------------------------|---------------------------------------------------------------------------------------|
| `pymorphy3`, `simplemma`, ... | The name of a registered backend, including backends from loaded plugins (see below). |
| `my_module:my_function`       | A function `fn(text, lang_code)`, imported from `my_module`.                          |
| `my_module:MyBackend`         | A `Backend` subclass; it is registered under its `name` if it is not registered yet.  |

```bash
termflex uk.po --glossary glossary.tbx --src en --tgt uk --target-lemmatizer pymorphy3
termflex uk.po --glossary glossary.tbx --src en --tgt uk --lemmatizer my_lemmas:lemmatize
```

#### Plugins

A plugin is a module or a `.py` file whose `@register_backend` or `@register_function` backends become available
by name once it is loaded. Load one with `--plugin`, which you can repeat:

```bash
termflex uk.po --glossary glossary.tbx --src en --tgt uk \
    --plugin ./my_backends.py --target-lemmatizer my_backend

termflex uk.po --glossary glossary.tbx --src en --tgt uk \
    --plugin my_pkg.backends --lemmatizer my_backend
```

To load plugins on every run, list them in `TERMFLEX_PLUGINS`, separated by `:`
(`;` on Windows):

```bash
export TERMFLEX_PLUGINS=my_pkg.backends:./my_backends.py
```

Plugins load before anything else runs, so `--list-backends` includes them:

```bash
termflex --plugin ./my_backends.py --list-backends
```

### Output and exit codes

Failures are printed grouped by file, with the source, the target and the glossary terms that are missing:

```
=== sample_files/bulk_test/file2.csv

  line 23
    [MISSING] 'folder' -> ['тека']
      expected: тека
      SRC: No files in this <term>folder</term>
      TGT: У цій папці немає файлів
      SRC lemmas: no file in this <term>folder</term>
      TGT lemmas: у цей папка нема файл
```

The run ends with a summary (total issues, files with issues, most frequently missing terms, and files that could not be read). File-level errors go to stderr.

| Exit code | Meaning                                 |
|-----------|-----------------------------------------|
| `0`       | Clean: no terminology issues.           |
| `1`       | Terminology issues were found.          |
| `2`       | Some files could not be read or parsed. |

## Supported input formats

| Format           | Notes                                                                                                                  |
|------------------|------------------------------------------------------------------------------------------------------------------------|
| `.po`            | Plural forms are paired with the right source string. Fuzzy and obsolete entries are skipped.                          |
| `.xlf`, `.xliff` | XLIFF 1.2 and 2.x. Inline code (`<ph>`, `<bpt>`, `<ept>`, `<it>`) is dropped, units with `translate="no"` are skipped. |
| `.csv`           | Delimiter is auto-detected; UTF-8 with or without a BOM. Name the columns with `--source-field` and `--target-field`.  |
| `.json`          | A list of objects or a `{source: target}` mapping, optionally nested. See [JSON input](#json-input).                   |

### JSON input

JSON input is described in one of two ways (not both):

- **Field flags:** `--source-field` and `--target-field` (required), `--context-field` and `--id-field`
  (optional). The file must be a list of objects.
- **`--schema`:** inline JSON or a path to a `.json` file, either
  - a records schema: `{"root": ..., "source": ..., "target": ..., "context": ..., "id": ...}`,
    where only `source` and `target` are required, or
  - a mapping schema: `{"layout": "mapping"}`, optionally with `"root"`, for `{"source": "target"}` objects.

## Python API

### Check a pair of strings
```python
from termflex import Glossary, Checker

glossary = Glossary.from_json(
    input_json={term: term for term in ["sign in", "workspace", "knowledge base", "shared drive"]},
    schema={'layout': 'mapping'},
    source_lang='en',
    target_lang='en',
)
checker = Checker(glossary=glossary)

prompt = '''Write two sentences for a help article on sharing a document with your team.
Use these terms exactly: "workspace", "sign in", "shared drive".'''

response = '''First, log in to your account and open your workspace.
Then move the document to a shared drive, so it shows up in your teammates' workspaces too.'''

matches = checker.check(
    source_text=prompt,
    target_text=response,
    render=True,
)
for m in matches:
    print(m.source_term, m.status, m.target_hits)
    print(m.target_rendered)   # the target text with the term wrapped in <term>...</term>
```

`check()` returns one result for each glossary term found in the source text. If the same term was found multiple times, all its occurrences are listed under `source_hits`. Each source term in the glossary is allowed to have multiple target variants. The matcher tries to identify all variants in the target text and return them under `target_hits`. If `target_hits` is empty the match is considered `missing`. If the length of `target_hits` is less than the length of `source_hits` the match is considered `partial`, though this doesn't necessarily mean an error.

In case the identified term spans overlap, e.g. "server" and "database server", the longest match will be used.

All source hits return character and lemma spans. Target hits will not return character spans if the backend has no explicit access to the tokenizer and all matching is done using lemmas, because without a tokenizer there is no way to align lemma and token boundaries. A target hit will not return the lemma span if the match was successfully found using plain text and no lemmatization was needed.

The check can be run with the `render=True` flag, in which case the match will also contain rendered source and target texts and lemmas, where all identified hits are wrapped in a <term> tag for easier visual QA.

Before the check is performed, `Checker` masks any localization placeholders (see `DEFAULT_PLACEHOLDERS` in `termflex/checker.py`). You can replace or add to the placeholders inside the checker object:

```python
checker = Checker(glossary=glossary, placeholders=[r"\{[^{}\s]*\}"])
checker.placeholders.append(r"<[^>]+>")
matches = checker.check(...)
```

`Glossary` from file builders also accept custom `normalize` and `tokenizer` callables, which are then passed on to the lemmatizers.
By default, `normalize` comes from `termflex/utils/utils.py:default_normalize`, and `tokenizer` is defined for each supported backend using `termflex/morph_backends/recording_tokenizer.py`.


### Check a single file

To check a single file:
```python
from termflex import check_file

results = check_file('sample_files/example.po', checker)
```

### Check many files
To check many files in parallel, use `process_many`. It takes a **checker factory** rather than a checker: each worker process calls the factory once to build its own `Checker`, because a built checker cannot be sent between processes.

```python
from termflex import Glossary, Checker, process_many

def make_checker():
    return Checker(glossary=Glossary.from_json(
        input_json="sample_files/glossary.json",
        source_lang="en",
        target_lang="uk",
        schema={"layout": "mapping"}
    ))

if __name__ == '__main__':
    results, errors = process_many(
            "sample_files/bulk_test",
            checker_factory=make_checker,
            field_options={"source_col": "source", "target_col": "target"},
            format="csv",
            max_workers=3)
```

### Output options

To convert results into a report:
```python
from termflex import print_report

print_report(results, show_ok=False)
```
When `show_ok` is False, only missing matches will be printed. To see the identified terms in the source and target text, make sure `check_file` or `process_many` was run with `render=True`.

To save the results to a json file:
```python
from termflex import save_json

save_json(results, 'results.json')
```

To get a summary of results and accuracy and the number of partial and missing matches:
```python
from termflex import summarize

summarize(results, errors)
```

## Morphology backends

A backend turns text into lemmas. Termflex ships with `simplemma` (installed by default) and `pymorphy3` (optional extra), and you can pass or register your own callable. Backends are created with `get_backend` and cached, so asking for the same backend and language twice returns the same instance.

To see a mapping to available backends to languages:
```python
from termflex import available_backends

print(available_backends())
```
```text
{'pymorphy3': ['ru', 'uk'], 'simplemma': ['ar', 'ast', 'bg', 'ca', 'cs', 'cy', 'da', 'de', 'el', 'en', 'enm', 'eo', 'es', 'et', 'fa', 'fi', 'fr', 'ga', 'gd', 'gl', 'grc', 'gv', 'hbs', 'he', 'hi', 'hu', 'hy', 'id', 'is', 'it', 'ka', 'la', 'lb', 'lt', 'lv', 'mk', 'ml', 'ms', 'nb', 'nl', 'nn', 'pl', 'pt', 'ro', 'ru', 'se', 'sk', 'sl', 'sq', 'sv', 'sw', 'tl', 'tr', 'uk']}
```

To see a list of backends available for a language:

```python
from termflex import backends_for

print(backends_for('uk'))
```
```text
['pymorphy3', 'simplemma']
```

### Adding your own backend

**The quickest way: a function.** Pass any function `fn(text, lang_code) -> list[str]` (or a single `str`) directly to ``Glossary``. It receives the text and the language code (`"uk"` for `uk_UA`) and returns the lemmas:

```python
from termflex import Glossary

def custom_lemmatize(text: str, lang_code: str) -> list[str]:
    return [word.lower() for word in text.split()]

glossary = Glossary.from_tbx('sample_files/TBX-basic-sample.tbx', source_lang='en', target_lang='fr',
                             lemmatizer=custom_lemmatize)
```

To get the `CallableBackend` object, pass the function to `get_backend`:

```python
from termflex.morph_backends import get_backend

lemmatizer = get_backend(custom_lemmatize, "uk")
```

To make a function available by name, register it with the decorator:

```python
from termflex.morph_backends import register_function, get_backend
from termflex import Glossary

@register_function
def custom_lemmatize(text: str, lang_code: str) -> list[str]:
    ...

lemmatizer = get_backend("custom_lemmatize", "fr")   # an instance of FunctionBackend[custom_lemmatize]
glossary = Glossary.from_tbx('sample_files/TBX-basic-sample.tbx', source_lang='en', target_lang='fr',
                             lemmatizer='custom_lemmatize')
```

To pass the lemmatizer name and a list of supported languages, pass them to the decorator:
```python
from termflex.morph_backends import register_function, get_backend
from termflex import Glossary

@register_function("hunspell-r", languages={"en", "fr"})
def custom_lemmatize(text: str, lang_code: str) -> list[str]:
    ...

lemmatizer = get_backend("hunspell-r", "fr")   # an instance of FunctionBackend[hunspell-r]
print(lemmatizer.supported_languages())
glossary = Glossary.from_tbx('sample_files/TBX-basic-sample.tbx', source_lang='en', target_lang='fr',
                             lemmatizer='hunspell-r')
```

If you pass `tokenizer` to get_backend, it's assumed that the custom function processes a single token per call.

If the tokenizer is not passed, the entire text will be passed to the custom lemmatize function. In this case, the Doc tokens will not contain character spans and the matched lemma spans will not be aligned back to character spans.

The `Doc` class normalizes the text before lemmatization is called using `backend.normalize`. You can pass your own `normalize=` to `get_backend` as well. When normalize and tokenizer are passed to Glossary, they are passed to get_backend and eventually to the callable backend.


```python
from termflex.morph_backends import register_function, get_backend
from termflex import Glossary

def custom_normalize(text: str) -> str:
    ...

def custom_tokenize(text: str) -> list[str]:
    ...

def custom_lemmatize(text: str, lang_code: str) -> str:
    ...

lemmatizer = get_backend(custom_lemmatize, "fr", normalize=custom_normalize, tokenizer=custom_tokenize)
# or pass directly to Glossary:
glossary = Glossary.from_tbx('sample_files/TBX-basic-sample.tbx', source_lang='en', target_lang='fr',
                             lemmatizer=custom_lemmatize, normalize=custom_normalize, tokenizer=custom_tokenize)
```

**A reusable backend: a class.** Subclass `Backend`, implement `lemmatize`, and register it with the decorator:

```python
from termflex.morph_backends import Backend, register_backend
from termflex.doc import Token

@register_backend
class SpacyBackend(Backend):
    name = "spacy-uk"
    languages = frozenset({"uk"})

    def __init__(self, lang, normalize=None):
        super().__init__(lang, normalize)
        import spacy
        self.nlp = spacy.load("uk_core_news_sm")

    def lemmatize(self, norm):
        return [Token(lemma=t.lemma_, start=t.idx, end=t.idx + len(t.text), text=t.text)
                for t in self.nlp(norm) if not t.is_punct and not t.is_space]
```

See `termflex/morph_backends/` for the existing backends as a reference.

**Shipping a backend in a package.** Decorators only run when their module is imported. To have Termflex import your module, list it as an entry point (one per module, pointing at the module, not at a class or function):
```toml
[project.entry-points."termflex.morph_backends"]
mybackend = "my_package.backend"
```

The module then registers its backends with `@register_backend` / `@register_function`. Plugins are loaded lazily, the first time a name is not found or backends are listed. A plugin that fails to import is skipped with a warning. Reinstall the package after changing its entry points.

By default, instances are shared per language (`uk` and `uk_UA` give the same instance). If your backend behaves differently per region or script, override `cache_key` so that those variants are cached separately:

```python
from termflex.morph_backends import register_backend, SimplemmaBackend


@register_backend
class SimplemmaRegionBackend(SimplemmaBackend):
    name = "simplemma-region"

    @classmethod
    def cache_key(cls, lang):
        return str(lang)
```

## Development

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[all,test]"
python -m pytest -q
```

Run pytest as a module (`python -m pytest`), because the process-pool tests need the repository root on the import path.

## License

MIT
